Handling version compatibility is one of the hardest challenges in building software. Until now, conda-build provided helpful tools in terms of the ability to constrain or pin versions in recipes. The limiting thing about this capability was that it entailed editing a lot of recipes.
Conda-build 3 introduces a new scheme for controlling version constraints, which enhances behavior in two ways. First, you can now set versions in an external file and provide lists of versions for conda-build to loop over. Matrix builds are now much simpler and no longer require an external tool, such as conda-build-all. Second, there have been several new jinja2 functions added, which allow recipe authors to express their constraints relative to the versions of packages installed at build time. This dynamic expression greatly cuts down on the need for editing recipes.
Each of these developments have enabled interesting new capabilities for cross-compiling, as well as improving package compatibility by adding more intelligent constraints.
This document is intended as a quick overview of new features in conda-build 3. For more information, see the docs.
These demos use conda-build's python API to render and build recipes. That API currently does not have a docs page, but is pretty self explanatory. See the source at: https://github.com/conda/conda-build/blob/master/conda_build/api.py
This jupyter notebook itself is included in conda-build's tests folder. If you're interested in running this notebook yourself, see the tests/test-recipes/variants folder in a git checkout of the conda-build source. Tests are not included with conda packages of conda-build.
from conda_build import api
import os
from pprint import pprint
First, set up some helper functions that will output recipe contents in a nice-to-read way:
def print_yamls(recipe, **kwargs):
yamls = [api.output_yaml(m[0])
for m in api.render(recipe, verbose=False, permit_unsatisfiable_variants=True, **kwargs)]
for yaml in yamls:
print(yaml)
print('-' * 50)def print_outputs(recipe, **kwargs):
pprint(api.get_output_file_paths(recipe, verbose=False, **kwargs))
Most of the new functionality revolves around much more powerful use of jinja2 templates. The core idea is that there is now a separate configuration file that can be used to insert many different entries into your meta.yaml files.
!cat 01_basic_templating/meta.yaml
package:
name: abc
version: 1.0
requirements:
build:
- something {{ something }}
run:
- something {{ something }}
# The configuration is hierarchical - it can draw from many config files. One place they can live is alongside meta.yaml:
!cat 01_basic_templating/conda_build_config.yaml
something:
- 1.0
- 2.0
Since we have one slot in meta.yaml, and two values for that one slot, we should end up with two output packages:
print_outputs('01_basic_templating/')
Returning non-final recipe for abc-1.0-0; one or more dependencies was unsatisfiable:
Build: something
Host: None
['/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h1332e90_0.tar.bz2',
'/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h9f70ef6_0.tar.bz2']
print_yamls('01_basic_templating/')
package:
name: abc
version: '1.0'
build:
string: '0'
requirements:
build:
- something 1.0
run:
- something 1.0
extra:
final: true
--------------------------------------------------
package:
name: abc
version: '1.0'
build:
string: '0'
requirements:
build:
- something 2.0
run:
- something 2.0
extra:
final: true
--------------------------------------------------
OK, that's fun already. But wait, there's more!
We saw a warning about "finalization." That's conda-build trying to figure out exactly what packages are going to be installed for the build process. This is all determined before the build. Doing so allows us to tell you the actual output filenames before you build anything. Conda-build will still render recipes if some dependencies are unavailable, but you obviously won't be able to actually build that recipe.
!cat 02_python_version/meta.yaml
package:
name: abc
version: 1.0
requirements:
build:
- python
run:
- python
!cat 02_python_version/conda_build_config.yaml
python:
- 2.7
- 3.5
print_yamls('02_python_version/')
package:
name: abc
version: '1.0'
build:
string: py27hef4ac7c_0
requirements:
build:
- readline 6.2 2
- tk 8.5.18 0
- pip 9.0.1 py27_1
- setuptools 27.2.0 py27_0
- openssl 1.0.2l 0
- sqlite 3.13.0 0
- python 2.7.13 0
- wheel 0.29.0 py27_0
- zlib 1.2.8 3
run:
- python >=2.7,<2.8
extra:
final: true
--------------------------------------------------
package:
name: abc
version: '1.0'
build:
string: py35h6785551_0
requirements:
build:
- readline 6.2 2
- setuptools 27.2.0 py35_0
- tk 8.5.18 0
- openssl 1.0.2l 0
- sqlite 3.13.0 0
- python 3.5.3 1
- pip 9.0.1 py35_1
- xz 5.2.2 1
- wheel 0.29.0 py35_0
- zlib 1.2.8 3
run:
- python >=3.5,<3.6
extra:
final: true
--------------------------------------------------
Here you see that we have many more dependencies than we specified, and we have much more detailed pinning. This is a finalized recipe. It represents exactly the state that would be present for building (at least on the current platform).
So, this new way to pass versions is very fun, but there's a lot of code out there that uses the older way of doing things—environment variables and CLI arguments. Those still work. They override any conda_build_config.yaml settings.
# Setting environment variables overrides the conda_build_config.yaml. This preserves older, well-established behavior.
os.environ["CONDA_PY"] = "3.4"
print_yamls('02_python_version/')
del os.environ['CONDA_PY']
package:
name: abc
version: '1.0'
build:
string: py34h31af026_0
requirements: build:
- readline 6.2 2
- python 3.4.5 0
- setuptools 27.2.0 py34_0
- tk 8.5.18 0
- openssl 1.0.2l 0
- sqlite 3.13.0 0
- pip 9.0.1 py34_1
- xz 5.2.2 1
- wheel 0.29.0 py34_0
- zlib 1.2.8 3
run:
- python >=3.4,<3.5
extra:
final: true
--------------------------------------------------
# Passing python as an argument (CLI or to the API) also overrides conda_build_config.yaml
print_yamls('02_python_version/', python="3.6")
package:
name: abc
version: '1.0'
build:
string: py36hd0a5620_0
requirements:
build:
- readline 6.2 2
- wheel 0.29.0 py36_0
- tk 8.5.18 0
- python 3.6.1 2
- openssl 1.0.2l 0
- sqlite 3.13.0 0
- pip 9.0.1 py36_1
- xz 5.2.2 1
- setuptools 27.2.0 py36_0
- zlib 1.2.8 3
run:
- python >=3.6,<3.7
extra:
final: true
--------------------------------------------------
Wait a minute—what is that h7d013e7 gobbledygook in the build/string field?
Conda-build 3 aims to generalize pinning/constraints. Such constraints differentiate a package. For example, in the past, we have had things like py27np111 in filenames. This is the same idea, just generalized. Since we can't readily put every possible constraint into the filename, we have kept the old ones, but added the hash as a general solution.
There's more information about what goes into a hash at: https://conda.io/docs/building/variants.html#differentiating-packages-built-with-different-variants
Let's take a look at how to inspect the hash contents of a built package.
outputs = api.build('02_python_version/', python="3.6",
anaconda_upload=False)
pkg_file = outputs[0]
print(pkg_file)
The following NEW packages will be INSTALLED:
openssl: 1.0.2l-0
pip: 9.0.1-py36_1
python: 3.6.1-2
readline: 6.2-2
setuptools: 27.2.0-py36_0
sqlite: 3.13.0-0 tk: 8.5.18-0
wheel: 0.29.0-py36_0
xz: 5.2.2-1
zlib: 1.2.8-3
source tree in: /Users/msarahan/miniconda3/conda-bld/abc_1498787283909/work
Attempting to finalize metadata for abc
INFO:conda_build.metadata:Attempting to finalize metadata for abc
BUILD START: ['abc-1.0-py36hd0a5620_0.tar.bz2']
Packaging abc
INFO:conda_build.build:Packaging abc
The following NEW packages will be INSTALLED:
openssl: 1.0.2l-0
pip: 9.0.1-py36_1
python: 3.6.1-2
readline: 6.2-2
setuptools: 27.2.0-py36_0
sqlite: 3.13.0-0 tk: 8.5.18-0
wheel: 0.29.0-py36_0
xz: 5.2.2-1
zlib: 1.2.8-3
Packaging abc-1.0-py36hd0a5620_0
INFO:conda_build.build:Packaging abc-1.0-py36hd0a5620_0
number of files: 0
Fixing permissions
Fixing permissions
updating: abc-1.0-py36hd0a5620_0.tar.bz2
Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:
anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
anaconda_upload is not set. Not uploading wheels: []
/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
# Using command line here just to show you that this command exists.
!conda inspect hash-inputs ~/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
Package abc-1.0-py36hd0a5620_0 does not include recipe. Full hash information is not reproducible.
WARNING:conda_build.inspect:Package abc-1.0-py36hd0a5620_0 does not include recipe. Full hash information is not reproducible.
{'abc-1.0-py36hd0a5620_0': {'files': [],
'recipe': {'requirements': {'build': ['openssl '
'1.0.2l 0',
'pip 9.0.1 '
'py36_1',
'python '
'3.6.1 2',
'readline '
'6.2 2',
'setuptools '
'27.2.0 '
'py36_0',
'sqlite '
'3.13.0 0',
'tk 8.5.18 0',
'wheel '
'0.29.0 '
'py36_0',
'xz 5.2.2 1',
'zlib 1.2.8 '
'3'],
'run': ['python '
'>=3.6,<3.7']}}}}
pin_run_as_build is a special extra key in the config file. It is a generalization of the x.x concept that existed for numpy since 2015. There's more information at: https://conda.io/docs/building/variants.html#customizing-compatibility
Each x indicates another level of pinning in the output recipe. Let's take a look at how we can control the relationship of these constraints. Before now you could certainly accomplish pinning, it just took more work. Now you can define your pinning expressions, and then change your target versions in just one config file.
!cat 05_compatible/meta.yaml
package:
name: compatible
version: 1.0
requirements:
build:
- libpng
run:
- {{ pin_compatible('libpng') }}
This is effectively saying "add a runtime libpng constraint that follows conda-build's default behavior, relative to the version of libpng that was used at build time."
pin_compatible is a new helper function available to you in meta.yaml. The default behavior is: exact version match lower bound ("x.x.x.x.x.x.x"), next major version upper bound ("x").
print_yamls('05_compatible/')
package:
name: compatible
version: '1.0'
build:
string: h3d53989_0
requirements:
build:
- libpng 1.6.27 0
- zlib 1.2.8 3
run:
- libpng >=1.6.27,<2
extra:
final: true
--------------------------------------------------
These constraints are completely customizable with pinning expressions:
!cat 06_compatible_custom/meta.yaml
package:
name: compatible
version: 1.0
requirements:
build:
- libpng
run:
- {{ pin_compatible('libpng', max_pin='x.x') }}
print_yamls('06_compatible_custom/')
package:
name: compatible
version: '1.0'
build:
string: ha6c6d66_0
requirements:
build:
- libpng 1.6.27 0
- zlib 1.2.8 3
run:
- libpng >=1.6.27,<1.7
extra:
final: true
--------------------------------------------------
Finally, you can also manually specify version bounds. These supersede any relative constraints.
!cat 07_compatible_custom_lower_upper/meta.yaml
package:
name: compatible
version: 1.0
requirements:
build:
- libpng
run:
- {{ pin_compatible('libpng', min_pin=None, upper_bound='5.0') }}
print_yamls('07_compatible_custom_lower_upper/')
package:
name: compatible
version: '1.0'
build:
string: heb31dda_0
requirements:
build:
- libpng 1.6.27 0
- zlib 1.2.8 3
run:
- libpng <5.0
extra:
final: true
--------------------------------------------------
Much of the development of conda-build 3 has been inspired by improving the compiler toolchain situation. Conda-build 3 adds special support for more dynamic specification of compilers.
!cat 08_compiler/meta.yaml
package:
name: cross
version: 1.0
requirements:
build:
- {{ compiler('c') }}
By replacing any actual compiler with this jinja2 function, we're free to swap in different compilers based on the contents of the conda_build_config.yaml file (or other variant configuration). Rather than saying "I need gcc," we are saying "I need a C compiler."
By doing so, recipes are much more dynamic, and conda-build also helps to keep your recipes in line with respect to runtimes. We're also free to keep compilation and linking flags associated with specific "compiler" packages—allowing us to build against potentially multiple configurations (Release, Debug?). With cross compilers, we could also build for other platforms.
!cat 09_cross/meta.yaml
package:
name: cross
version: 1.0
requirements:
build:
- {{ compiler('c') }}
# But, by adding in a base compiler name, and target platforms, we can make a build matrix
# This is not magic, the compiler packages must already exist. Conda-build is only following a naming scheme.
!cat 09_cross/conda_build_config.yaml
c_compiler:
- gcc
target_platform:
- linux-64
- linux-cos5-64
- linux-aarch64
print_yamls('09_cross/')
Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-64
Host: None
WARNING:conda_build.render:
Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-64
Host: None
Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-cos5-64
Host: None
WARNING:conda_build.render:Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-cos5-64
Host: None
Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-aarch64
Host: None
WARNING:conda_build.render:Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
Build: gcc_linux-aarch64
Host: None
package:
name: cross
version: '1.0'
build:
string: '0'
requirements:
build:
- gcc_linux-64
extra:
final: true
--------------------------------------------------
package:
name: cross
version: '1.0'
build:
string: '0'
requirements:
build:
- gcc_linux-cos5-64
extra:
final: true
--------------------------------------------------
package:
name: cross
version: '1.0'
build:
string: '0'
requirements:
build:
- gcc_linux-aarch64
extra:
final: true
--------------------------------------------------
Finally, it is frequently a problem to remember to add runtime dependencies. Sometimes the recipe author is not entirely familiar with the lower level code and has no idea about runtime dependencies. Other times, it's just a pain to keep versions of runtime dependencies in line. Conda-build 3 introduces a way of storing the required runtime dependencies on the package providing the dependency at build time.
For example, using g++ in a non-static configuration will require that the end-user have a sufficiently new libstdc++ runtime library available at runtime. Many people don't currently include this in their recipes. Sometimes the system libstdc++ is adequate, but often not. By imposing the downstream dependency, we can make sure that people don't forget the runtime dependency.
# First, a package that provides some library.
# When anyone uses this library, they need to include the appropriate runtime.
!cat 10_runtimes/uses_run_exports/meta.yaml
package:
name: package_has_run_exports
version: 1.0
build:
run_exports:
- {{ pin_compatible('bzip2') }}
requirements:
build:
- bzip2
# This is the simple downstream package that uses the library provided in the previous recipe.
!cat 10_runtimes/consumes_exports/meta.yaml
package:
name: package_consuming_run_exports
version: 1.0
requirements:
build:
- package_has_run_exports
# Let's build the former package first.
api.build('10_runtimes/uses_run_exports', anaconda_upload=False)
The following NEW packages will be INSTALLED:
bzip2: 1.0.6-3
source tree in: /Users/msarahan/miniconda3/conda-bld/package_has_run_exports_1498787302719/work
Attempting to finalize metadata for package_has_run_exports
INFO:conda_build.metadata:Attempting to finalize metadata for package_has_run_exports
BUILD START: ['package_has_run_exports-1.0-hcc78ab3_0.tar.bz2']
Packaging package_has_run_exports
INFO:conda_build.build:Packaging package_has_run_exports
The following NEW packages will be INSTALLED:
bzip2: 1.0.6-3
number of files: 0
Fixing permissions
Fixing permissions
updating: package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:
anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
anaconda_upload is not set. Not uploading wheels: []
['/Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2']
print_yamls('10_runtimes/consumes_exports')
package:
name: package_consuming_run_exports
version: '1.0'
build:
string: h8346d2f_0
requirements:
build:
- package_has_run_exports 1.0 hcc78ab3_0
run:
- bzip2 >=1.0.6,<2
extra:
final: true
--------------------------------------------------
In the above recipe, note that bzip2 has been added as a runtime dependency, and is pinned according to conda-build's default pin_compatible scheme. This behavior can be overridden in recipes if necessary, but we hope it will prove useful.
