Davide Moro: Pip for buildout folks

... or buildout for pip folks.

In this article I'm going to talk about how to manage software (Python) projects with buildout or pip.

What do you mean for project?

A package that contains all the application-specific settings, database configuration, which packages your project will need and where they lives.

Projects should be managed like a software if you want to assure the needed quality:

• under version control
• following a good branching model, see one of my preferred blog posts http://nvie.com/posts/a-successful-git-branching-model/
• with a CHANGES.rst. It is very very important. If you deploy frequently small releases you can easily understand when you introduced a weird bug and fix it, you can see what version is installed on production, etc. You can keep track about what changed in a particular release. For example all software updates (example: kotti_boxes: 0.1.1 -> 0.1.2) with full info about why changed, when, who, etc. Based on this rst file you can also generate as well the project documentation with Sphinx (http://sphinx-doc.org/), etc.
• tagged (zest.releaser can help you. See https://opensourcehacker.com/2012/08/14/high-quality-automated-package-releases-for-python-with-zest-releaser/)
• documentation
• etc

This blog post is not:

• intended to be a complete guide to pip or buildout. If you want to know more about pip or buildout
• talking about how to deploy remotely your projects

Buildout

I've been using buildout for many years and we are still good friends.
Buildout definition (from http://www.buildout.org):

"""
Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later. 
"""

With buildout you can build and share reproducible environments, not only for Python based components.

Before buildout (if I remember well the first time I get started to use buildout was in 2007, probably during the very first Plone Sorrento sprint) it was a real pain sharing a complete and working developing environment pointing to the right version of several repositories, etc. With buildout it was questions of minutes.

From https://pypi.python.org/pypi/mr.developer. Probably with pip there is less fun because there isn't a funny picture that celebrates it?!

Buildout configuration files are modular and extensible (not only on per-section basis). There are a lot of buildout recipes, probably the one I prefer is mr.developer (https://pypi.python.org/pypi/mr.developer). It allowed me to fetch different versions of the repositories depending on the buildout profile in use, for example:

• production -> each developed private egg point to a tag version
• devel -> the same eggs point to the develop/master

You can accomplish this thing creating different configurations for different profiles, like that:

[buildout]

...

[sources]

your_plugin = git git@github.com:username/your_plugin.git

...

I don't like calling ./bin/buildout -c [production|devel].cfgwith the -c syntax because it is too much error prone. I prefer to create a symbolic link to the right buildout profile (called buildout.cfg) and you'll perform the same command both in production or during development always typing:

$ ./bin/buildout

This way you'll avoid nasty errors like launching a wrong profile in producion. So use just the plain ./bin/buildout command and live happy.

With buildout you can show and freeze all the installed versions of your packages providing a versions.cfg file.

Here you can see my preferred buildout recipes:

• https://pypi.python.org/pypi/mr.developer, already discussed before
• https://pypi.python.org/pypi/collective.recipe.cmd, useful for launching commands
• https://pypi.python.org/pypi/z3c.recipe.template, useful for dynamic template generation (like logrotate files, nginx/apache configurations, etc)
• https://pypi.python.org/pypi/zc.recipe.cmmi, compilation on the fly for dependencies hard to install. Sometimes you can provide a very basic optional buildout configuration in your packages if they depends on external software hard to configure (for example you can provide a fully configured solr installation if you assume a specific setup)

Buildout or not buildout, one of the of the most common needs it is the ability to switch from develop to tags depending on you are in development or production mode and reproduce the same software later. I can't figure out to manage software installations without this quality assurance.

More info: http://www.buildout.org

Pip

Let's see how to create reproducible environments with develop or tags dependencies for production environments with pip (https://pip.pypa.io/en/latest/).

Basically you specify your devel requirements on a devel-requirements.txt file (the name doesn't matter) pointing to the develop/master/trunk on your repository.

There is another file that I call production-requirements (the file name doesn't matter) that it is equivalent to the previous one but:

• without devel dependencies you don't want to install in production mode
• tagging your private applications (instead of master -> 0.1.1)

This way it is quite simple seeing which releases are installed in production mode, with no cryptic hash codes.

You can use now the production-requirements.txt as a template for generating an easy to read requirements.txt. You'll use this file when installing in production.

You can create a regular Makefile if you don't want to repeat yourself or make scripts if you prefer:

• compile Sphinx documentation
• provide virtualenv initialization
• launch tests against all developed eggs
• update the final requirements.txt file

For example if you are particular lazy you can create a script that will create your requirements.txt file using the production-requirements.txt like a template.
This is a simple script, it is just an example, that shows how to build your requirements.txt omitting lines with grep, sed, etc:

#!/bin/bash

pip install -r production-requirements.txt
pip freeze -r production-requirements.txt | grep -v mip_project | sed '1,2d'> requirements.txt

When running this script, you should activate another Python environment in order to not pollute the production requirements list with development stuff.

If you want to make your software reusable and as flexible as possible, you can add a regular setup.py module with optional dependencies, that you can activate depending on what you need. For example in devel-mode you might want to activate an entry point called docs (see -e .[docs] in devel-requirements.txt) with optional Sphinx dependencies. Or in production you can install MySQL specific dependencies (-e .[mysql]).

In the examples below I'll also show how to refer to external requirements file (url or a file).

setup.py

You can define optional extra requirements in your setup.py module.

mysql_requires=[
'MySQL-python',
]

docs_requires=[
'Sphinx',
'docutils',
'repoze.sphinx.autointerface',
]
...

setup(
name='mip_project',
version=version,
    ...
extras_require={
'mysql':mysql_requires,
'docs':docs_requires,

        ... 
},

devel-requirements.txt

Optional extra requirement can be activated using the [] syntax (see -e .[docs]).
You can also include external requirement files or urls (see -r) and tell pip how to fetch some concrete dependencies (see -e git+...#egg=your_egg).

-r https://github.com/.../.../blob/VERSION/requirements.txt

# Kotti
Kotti[development,testing]==VERSION

# devel (to no be added in production)
zest.releaser

# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git#egg=kotti_boxes
...

-e .[docs]

production_requirements.txt

The production requirements should point to tags (see @VERSION).

-r https://github.com/Kotti/Kotti/blob/VERSION/requirements.txt
Kotti[development,testing]==VERSION 

# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@0.1.1#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git@0.1.3#egg=kotti_boxes
...

-e .[mysql] 

requirements.txt
The requirements.txt is autogenerated based on the production-requirements.txt model file. All the installed versions are appended in alphabetical at the end of the file, it can be a very long list.
All the tag versions provided in the production-requirements.txt are automatically converted to hash values (@VERSION -> @3c1a191...).

Kotti==1.0.0a4

# Third party's eggs
kotti-newsitem==0.2
kotti-calendar==0.8.2
kotti-link==0.1
kotti-navigation==0.3.1

# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@3c1a1914901cb33fcedc9801764f2749b4e1df5b#egg=kotti_actions-dev
-e git+https://github.com/truelab/kotti_boxes.git@3730705703ef4e523c566c063171478902645658#egg=kotti_boxes-dev
...

## The following requirements were added by pip freeze:
alembic==0.6.7
appdirs==1.4.0
Babel==1.3
Beaker==1.6.4

... 

Final consideration

Use pip to install Python packages from Pypi.

If you’re looking for management of fully integrated cross-platform software stacks, buildout is for you.

With buildout no Python code needed unless you are going to write new recipes (the plugin mechanism provided by buildout to add new functionalities to your software building, see http://buildout.readthedocs.org/en/latest/docs/recipe.html).

Instead with pip you can manage also cross-platform stacks but you loose the flexibility of buildout recipes and inheritable configuration files.

Anyway if you consider buildout too magic or you just need a way to switch from production vs development mode you can use pip as well.

Links

If you need more info have a look at the following urls:

• https://python-packaging-user-guide.readthedocs.org/en/latest/current.html
• http://www.buildout.org 
• https://pip.pypa.io/en/latest/

Other useful links:

• http://www.ianbicking.org/blog/2008/12/using-pip-requirements.html
• https://caremad.io/2013/07/setup-vs-requirement/
• https://devcenter.heroku.com/articles/python-pip
• https://pip.pypa.io/en/latest/user_guide.html

Update 20150629

If you want an example I've created a pip-based project for Kotti CMS (http://kotti.pylonsproject.org):

• https://github.com/davidemoro/kotti_project