BlueDynamics Alliance: Boosting Travis CI: Python and Buildout caching

zc.buildout is Pythons swiss knife to build complex enviroments. MacGywer would love it. Travis CI together with GitHub is a wonderful service for OpenSource projects to do collaborative development hand in hand with Test Driven Development and Continious Integration.

But complex Python builds are taking its time - mainly because of the long list of dependencies and bunch of downloads. It is boring to wait 15minutes for a test

So it was for collective.solr, a package that integrates the excellent Solr open source search platform, (written in Java, from the Apache Lucene project) with the Plone Enterprise CMS. Additional to the complex Plone, it downloads and configures a complete Solr for the test environment.

Since a while Travis CI offers caching on its container based infrastructure.

Using it with buildout is easy once ones knows how.

1. The file .travis.yaml configures Travis CI, open it.
   
2. set language: python if you not already have
   
3. an important setting is sudo = false which switches explicit to container based infrastructure. This is default for projects created at Travis CI before 2015-01-01, but explicit is better than implicit!
4. next the caching is defined. We enable also pip caching. This looks like so
   

   cache:
     pip: true
     directories:
       - $HOME/buildout-cache

5. in order to create our caching directories a before-install step is needed. In this step we install buildout too. Note: there is no need to use the old and busted bootstrap.py any longer (except old and busted Plone builds, since 4.3 at least it will work).
   

   before_install:
     - mkdir -p $HOME/buildout-cache/{eggs,downloads}
     - virtualenv .
     - bin/pip install --upgrade pip setuptools zc.buildout

6. Next we need to tweak file travis.cfg. This is the buildout configuration file used for travis. Under section [buildout] add the lines:

   eggs-directory = /home/travis/buildout-cache/eggs
   download-cache = /home/travis/buildout-cache/downloads

   Note, the $HOME environment variable is not available as buildout variable, so we need to set this fixed to /home/travis - Travis CI can not guarantee that this will stick for all future. So if there is a way to access environment variables before buildout runs the parts please let me know.

The second time Travis CI builds the project it took about 3 minutes instead of 15!
The full files as we use it for collective.solr:

.travis.yaml

language: python
# with next we get on container based infrastructure, this enables caching
sudo: false
python:
  - 2.7
cache:
  pip: true
  directories:
    - $HOME/buildout-cache
env:
  - PLONE_VERSION=4.3.x SOLR_VERSION=4.10.x
  - PLONE_VERSION=5.0.x SOLR_VERSION=4.10.x
before_install:
  - mkdir -p $HOME/buildout-cache/{eggs,downloads}
  - virtualenv .
  - bin/pip install --upgrade pip setuptools zc.buildout
install:
  - sed -ie "s#plone-x.x.x.cfg#plone-$PLONE_VERSION.cfg#" travis.cfg
  - sed -ie "s#solr-x.x.x.cfg#solr-$SOLR_VERSION.cfg#" travis.cfg
  - bin/buildout -N -t 20 -c travis.cfg
script:
  - bin/code-analysis
  - bin/test
after_success:
  - pip install -q coveralls
  - coveralls

travis.cfg

[buildout]
extends =
    base.cfg
    plone-x.x.x.cfg
    solr.cfg
    solr-x.x.x.cfg
    versions.cfg
parts +=
    code-analysis

# caches, see also .travis.yaml
# one should not depend on '/home/travis' but it seems stable in containers.
eggs-directory = /home/travis/buildout-cache/eggs
download-cache = /home/travis/buildout-cache/downloads

[code-analysis]
recipe = plone.recipe.codeanalysis
pre-commit-hook = False
# return-status-codes = True

We may enhance this, so you can always look at its current state at github/collective/collective.solr.

image by"Albert Einstein - Start 1" by DLR German Aerospace Center at Flickr