Continuum Analytics News: Announcing General Availability of conda 4.3

We're excited to announce that conda 4.3 has been released for general availability. The 4.3 release series has several new features and substantial improvements. Below is a summary. 

To get the latest, just run conda update conda.

New Features

• Unlink and Link Packages in a Single Transaction: In the past, conda hasn't always been safe and defensive with its disk-mutating actions. It has gleefully clobbered existing files; mid-operation failures left environments completely broken. In some of the most severe examples, conda can appear to "uninstall itself." With this release, the unlinking and linking of packages for an executed command is done in a single transaction. If a failure occurs for any reason while conda is mutating files on disk, the environment will be returned its previous state. While we've implemented some pre-transaction checks (verifying package integrity for example), it's impossible to anticipate every failure mechanism. In some circumstances, OS file permissions cannot be fully known until an operation is attempted and fails, and conda itself is not without bugs. Moving forward, unforeseeable failures won't be catastrophic.

• Progressive Fetch and Extract Transactions: Like package unlinking and linking, the download and extract phases of package handling have also been given transaction-like behavior. The distinction is that the rollback on error is limited to a single package. Rather than rolling back the download and extract operation for all packages, the single-package rollback prevents the need for having to re-download every package if an error is encountered.

• Generic- and Python-Type Noarch/Universal Packages: Along with conda-build 2.1, a noarch/universal type for Python packages is officially supported. These are much like universal Python wheels. Files in a Python noarch package are linked into a prefix just like any other conda package, with the following additional features:

  1. conda maps the site-packages directory to the correct location for the Python version in the environment,
  2. conda maps the python-scripts directory to either $PREFIX/bin or $PREFIX/Scripts depending on platform,
  3. conda creates the Python entry points specified in the conda-build recipe, and
  4. conda compiles pyc files at install time when prefix write permissions are guaranteed.

  Python noarch packages must be "fully universal." They cannot have OS- or Python version-specific dependencies. They cannot have OS- or Python version-specific "scripts" files. If these features are needed, traditional conda packages must be used.

• Multi-User Package Caches: While the on-disk package cache structure has been preserved, the core logic implementing package cache handling has had a complete overhaul. Writable and read-only package caches are fully supported.

• Python API Module: An oft requested feature is the ability to use conda as a Python library, obviating the need to "shell out" to another Python process. Conda 4.3 includes a conda.cli.python_api module that facilitates this use case. While we maintain the user-facing command-line interface, conda commands can be executed in-process. There is also a conda.exports module to facilitate longer-term usage of conda as a library across conda releases. However, conda's Python code is considered internal and private, subject to change at any time across releases. At the moment, conda will not install itself into environments other than its original install environment.

• Remove All Locks: Locking has never been fully effective in conda, and it often created a false sense of security. In this release, multi-user package cache support has been implemented for improved safety by hard-linking packages in read-only caches to the user's primary user package cache. Still, users are cautioned that undefined behavior can result when conda is running in multiple process and operating on the same package caches and/or environments.

Deprecations/Breaking Changes

• Conda now has the ability to refuse to clobber existing files that are not within the unlink instructions of the transaction. This behavior is configurable via the path_conflict configuration option, which has three possible values: clobber, warn, and prevent. In 4.3, the default value is clobber. This preserves existing behaviors, and it gives package maintainers time to correct current incompatibilities within their package ecosystem. In 4.4, the default will switch to warn, which means these operations continue to clobber, but the warning messages are displayed. In 4.5, the default value will switch to prevent. As we tighten up the path_conflict constraint, a new command line flag --clobber will loosen it back up on an ad hoc basis. Using --clobber overrides the setting for path_conflict to effectively be clobber for that operation.

• Conda signed packages have been removed in 4.3. Vulnerabilities existed, and an illusion of security is worse than not having the feature at all. We will be incorporating The Update Framework (TUF) into conda in a future feature release.

• Conda 4.4 will drop support for older versions of conda-build.

Other Notable Improvements

• A new "trace" log level is added, with output that is extremely verbose. To enable it, use -v -v -v or -vvv as a command-line flag, set a verbose: 3 configuration parameter, or set a CONDA_VERBOSE=3 environment variable.

• The r channel is now part of the default channels.

• Package resolution/solver hints have been improved with better messaging.