7. Working with zc.buildout – Expert Python Programming

Chapter 7. Working with zc.buildout

We have seen in the last chapter how to write an application based on several eggs. When distributing such an application, the user gets the package and its dependencies installed in the site-packages directory of Python, and gets some entry points such as command-line utilities.

But for bigger applications than Atomisator, this approach is limited: If you need to deploy some configuration files or write log files, it is not practical to make them live inside the code packages.

The best approach is to integrate them seamlessly in the target system by creating specific installers. On Linux-based systems for instance, the log files should be in /var/log and the configuration files in /etc. But creating such installers requires a lot of system-specific work.

Another approach, a bit similar to what virtualenv provides, is to work on a self-contained directory that has everything needed to run the application and then distribute it. This directory can also contain the required packages, and an installer that takes care of bootstrapping everything on the target system.

zc.buildout (see http://pypi.python.org/pypi/zc.buildout) is a tool that can be used to create such an environment and this chapter presents how to:

  • Organize an application through a descriptive language where all packages needed to run the application can be defined

  • Deploy such applications as a source release

Note

Alternative tools to zc.buildout are Paver and AutomateIt.

See http://www.blueskyonmars.com/projects/paver

and http://automateit.org.

This chapter is organized in three parts:

  • zc.buildout philosophy

  • How to distribute zc.buildout-based applications

  • The application template that creates a zc.buildout application environment using the paster tool

zc.buildout Philosophy

virtualenv is pretty convenient to isolate a Python environment. It works locally, as we saw in the previous chapter, but still requires a lot of manual work at the prompt to set up and maintain a project environment.

zc.buildout offers the same isolation feature, but goes further by providing:

  • A simple description language to define these dependencies in a configuration file

  • A plug-in system that provides entry points to chain a combination of code calls

  • A way to deploy and release the application sources together with their execution environment

The configuration file describes which eggs are needed in the environment, their states (being developed locally, or available at PyPI, or anywhere else), and all other elements needed to build an application.

The plug-in system registers packages and chains them in a sequence it executes.

Last, the whole environment is independent and isolated and can be, therefore, used in the same way as it is to be released and deployed.

zc.buildout has great documentation on its PyPI page ( http://pypi.python.org/pypi/zc.buildout). This section will just summarize the most important elements one needs to know in order to build and work at the application level. The elements are as follows:

  • The configuration file structure

  • The buildout command

  • Recipes

Configuration File Structure

zc.buildout relies on a configuration file that uses a structure compatible with the ConfigParser module. These INI-like files have sections delimited by [headers] with lines that contain name:value or name=value.

Minimum Configuration File

The minimum buildout configuration file contains a [buildout] section and has a variable called parts in it. This variable contains a multi-line value that provides a list of sections:

[buildout]
parts =
    part1
    part2

[part1]
recipe = my.recipe1

[part2]
recipe = my.recipe2

Each section specified in parts has at least one recipe value that provides the name of a package. This package can be any Python package as long as it defines a zc.buildout entry point.

With this file, buildout will play this sequence:

  • It will check if the package my.recipe1 is installed. If it's not installed, it fetches it and installs it locally.

  • It will execute the code pointed to by my.recipe1's entry point.

  • Then, it will do the same thing for part2.

A buildout is, therefore, a plug-in-based script that chains the execution of independent packages called recipes. Building an environment with this tool consists of defining the right sequence of recipes.

[buildout] Section Options

Besides parts, the [buildout] section has several options available. The most important ones are:

  • develop: Multi-line value that lists the eggs to be installed with the python setup.py develop command in the environment. Each of these values is a path to the package folder where setup.py is located.

  • find-links: Multi-line value that provides a list of locations (URL or file) provided to easy_install to find the eggs defined in eggs or in any dependency when installing an egg.

From there, a buildout can list a series of eggs to be installed in the environment. For each value specified in develop, the tool runs the setuptools develop command and fetches PyPI when dependencies are defined.

The web location used to find the package is the same as that used by easy_install is http://pypi.python.org/simple, which is a web page not intended for humans that contains a list of package links that can be browsed automatically.

Last, the find-links option provides a way to point to alternative sources when the packages are available in other places.

Let's take an example:

[buildout]
parts = 

develop = 
    /home/tarek/dev/atomisator.feed

find-links = 
    http://acme.com/packages/index

With this configuration, buildout will install the atomisator.feed package as python setup.py develop would, and use the extra link from http://acme.com/packages/index to find any dependencies when they are not available at PyPI.

This environment can be built using the buildout command.

The buildout Command

The buildout command is installed by zc.buildout with the usual easy_install call and can be used to interpret configuration files:

$ easy_install zc.buildout
...
$ buildout
While:
Initializing.
Error: Couldn't open /Users/tarek/buildout.cfg

An initial call with the init option in an empty directory will create a default buildout.cfg file and a few other elements:

$ cd /tmp
$ mkdir tests
$ cd tests
$ buildout init
Creating '/tmp/tests/buildout.cfg'.
Creating directory '/tmp/tests/bin'.
Creating directory '/tmp/tests/parts'.
Creating directory '/tmp/tests/eggs'.
Creating directory '/tmp/tests/develop-eggs'.
Generated script '/tmp/tests/bin/buildout'.
$ find .
.
./bin
./bin/buildout
./buildout.cfg
./develop-eggs
./eggs/setuptools-0.6c7-py2.5.egg
./eggs/zc.buildout-1.0.0b30-py2.5.egg
./parts
$ more buildout.cfg
[buildout]
parts =


The bin folder contains a local buildout script, in which three other folders are created:

  • parts corresponds to the sections defined in the configuration file. It is a standard place where each called recipe can write elements.

  • develop-eggs will hold information to link the environment to the packages defined in develop.

  • eggs contains eggs used by the environment. It is filled already with zc.buildout and setuptools eggs.

Let's change the cfg file by adding a develop section:

[buildout]
parts =
develop =
/home/tarek/dev/atomisator.feed

The specified folder will be installed by zc.buildout as a develop egg by calling the buildout command again:

$ bin/buildout
Develop: '/home/tarek/dev/atomisator.feed'
$ ls develop-eggs/
atomisator.feed.egg-link
$ more develop-eggs/atomisator.feed.egg-link
/home/tarek/dev/atomisator.feed

The develop-eggs folder now contains a link to the atomisator.feed package located in /home/tarek/dev/atomisator.feed. Of course, any folder containing a package can be tied into the buildout script with the develop option.

Recipes

We have seen that each section specifies a package as a recipe. The zc.recipe.egg one, for instance, is used to specify one or several eggs to install in the buildout. This recipe will pull the package as easy_install would, by calling PyPI, and will eventually look into the links provided in find-links if PyPI does not have it.

For example, if we want to install Nose into the buildout, this can be done by adding a dedicated section into the configuration file and pointing to it in the parts variable of the buildout section:

[buildout]
parts = 
    test
develop = 
    /home/tarek/dev/atomisator.feed

[test]
recipe = zc.recipe.egg
eggs =
    nose

Running the buildout script again will play the test section and pull the Nose egg as easy_install would:

$ bin/buildout
Develop: '/home/tarek/dev/atomisator.feed'
Installing test.
Getting distribution for nose
Got nose 0.10.3.

The nosetest script will be installed into the bin folder, and the Nose egg in the eggs folder.

Let's add a new section in the cfg file called other using zc.recipe.egg again:

[buildout]
parts = 
    test
	 other

develop = 
    /home/tarek/dev/atomisator.feed

[test]
recipe = zc.recipe.egg
eggs =
    nose

[other]
recipe = zc.recipe.egg

eggs =
    elementtree
    PIL
...

This new section defines two new packages. Let's run the buildout script again:

$ bin/buildout
Develop: '/home/tarek/dev/atomisator.feed'
Updating test.
Installing other.
Getting distribution for elementtree
Got elementtree 1.2.7-20070827-preview.
Getting distribution for 'PIL'.
Got PIL 1.1.6.

The sections pointed to in parts are run in the order they are defined. When run again, zc.buildout checks the already installed parts, to see if they need to be updated, and if so installs new ones. From the other section, the eggs folder gets populated with two new eggs.

Recipes are simple Python packages, usually dedicated to this sole role. They are conventionally nested namespaced packages, where the first part is the name of the organization, the second one is the recipe, and the third one the name of the recipe.

The recipe we have used so far is provided by the Zope Corporation (zc), but many recipes are available at PyPI to handle many needs in a buildout environment.

Since frameworks such as Zope or Plone rely on this tool, a quick search on http://pypi.python.org with buildout or recipe in the query will return hundreds of packages that can be used to compose any kind of buildout.

Notable Recipes

Here's a small list of useful recipes found on PyPI :

  • collective.recipe.ant: Builds Ant (Java) projects.

  • iw.recipe.cmd: Executes a command line.

  • iw.recipe.fetcher: Downloads a file pointed by a URL.

  • iw.recipe.pound: Compiles and installs Pound (a load balancer).

  • iw.recipe.squid: Configures and runs Squid (a cache server).

  • z3c.recipe.ldap: Deploys OpenLDAP.

Creating Recipes

A recipe is a simple class with two methods, namely, install and update. They return a list of installed files. Coding a new recipe is, therefore, dead simple and can be done using a template.

The ZopeSkel project, which is used in the Zope community to build new recipes, can be installed to have a new template called recipe among a few others:

$ easy_install ZopeSkel
Searching for ZopeSkel
Best match: ZopeSkel 2.1
...
Finished processing dependencies for ZopeSkel
$ paster create --list-templates
Available templates:
  ...
  recipe:             A recipe project for zc.buildout
  ...

recipe generates a nested namespace package structure with a Recipe class skeleton that has to be completed:

$ paster create -t recipe atomisator.recipe.here
Selected and implied templates:
  ZopeSkel#recipe  A recipe project for zc.buildout
  ...
Enter namespace_package ['plone']: atomisator 
Enter namespace_package2 ['recipe']:       
Enter package ['example']: here
Enter version (Version) ['1.0']: 
Enter description ['']: description is here.
Enter long_description ['']: 
Enter author (Author name) ['']: Tarek
Enter author_email (Author email) ['']: tarek@ziade.org
... 
Creating template recipe
Creating directory ./atomisator.recipe.here
...

$ more atomisator.recipe.here/atomisator/recipe/here/__init__.py 
# -*- coding: utf-8 -*-
"""Recipe here"""

class Recipe(object):
    """zc.buildout recipe"""

    def __init__(self, buildout, name, options):
        self.buildout, self.name, self.options = \
		      buildout, name, options

    def install(self):
        """Installer"""
        # XXX Implement recipe functionality here
        
        # Return files that were created by the recipe. 
        # The buildout will remove all returned files 
        # upon reinstall.
        return tuple()

    def update(self):
        """Updater"""
        pass


Atomisator buildout Environment

The Atomisator project can benefit from zc.buildout by creating a dedicated buildout configuration together with the packages, and defining an environment in it.

The buildout environment can be built in two steps:

  1. Creating a buildout folder structure

  2. Initializing the buildout

buildout Folder Structure

Since buildout allows us to link any folder of the system as a develop package, the application environment can be separated from it. The cleanest layout is to use a folder for the buildout and a folder for the packages being developed.

Let's revisit the Atomisator folder we created in the previous chapter. So far, it contains a bin folder with a local interpreter and a packages folder. Let's add a buildout folder to it:

$ cd Atomisator
$ mkdir buildout

A new buildout environment is then built in the buildout folder:

$ cd buildout
$ buildout init
Creating 'Atomisator/buildout/buildout.cfg'.
Creating directory 'Atomisator/buildout/bin'.
Creating directory 'Atomisator/buildout/parts'.
Creating directory 'Atomisator/buildout/eggs'.
Creating directory 'Atomisator/buildout/develop-eggs'.
Generated script 'Atomisator/buildout/bin/buildout'.

buildout.cfg is changed in order to generate a local nosetest script, and to install the Atomisator eggs as develop eggs:

[buildout]

develop = 
    ../packages/atomisator.main
    ../packages/atomisator.db
    ../packages/atomisator.feed
    ../packages/atomisator.parser

parts = 
    test

[test]
recipe = pbp.recipe.noserunner
eggs = 
    atomisator.main
    atomisator.db
    atomisator.feed
    atomisator.parser 

This configuration file will generate a complete Atomisator environment located in the buildout folder.

In the last chapter, we installed Nose in the same local interpreter where the packages were being developed, thanks to virtualenv. When working in a buildout, having the same feature requires more work: Installing Nose as an egg in the buildout will not make other eggs directly visible to the test runner. To get a similar environment, the pbp.recipe.noserunner is a small recipe that generates a local nosetests runner with a specific environment. All eggs defined in its eggs variable will be added in the test runner execution environment.

The recipe uses the section name for the name of the generated script. So a test script will be available in our case, which can be used to test all atomisator packages:

$ bin/test atomisator
........
-----------------------------------------------------------------
Ran 8 tests in 0.015s
OK

Going Further

Another step could be performed to create and use the atomisator.cfg file in the etc folder, which is in the buildout folder. This would be needed to create a new recipe that reads the values in the buildout.cfg file and generates atomisator.cfg.

A new section would then be created like this:

...
[atomisator-configuration]
recipe = atomisator.recipe.installer

sites = 
    sample1.xml
    sample2.xml

database = sqlite:///${buildout:directory}/var/atomisator.db

title = My Feed
description = The feed
link = the link

file = ${buildout:directory}/var/atomisator.xml
...

The ${buildout:directory} is replaced with the buildout path.

Releasing and Distributing

We have seen in the previous section that a buildout is a standalone folder that is able to include everything needed to run the application. All needed eggs are installed in it, and the console scripts are created in the bin folder.

As a matter of fact, the top Atomisator folder could be archived in an archive as it is, and then unpacked on some other computer that has Python. By running buildout again on this new target, everything would get bootstrapped correctly and the application could run from there.

Distributing the source this way is universal compared to the other packaging systems that every operating system provides, such as apt or RPM. Everything is isolated in a self-contained folder and will work on every system. Therefore, it will not integrate smoothly in the target system and will use its own specific standards. This is fine for many applications, but purists will want it to be installable with the package system used on the target system to ease system maintenance.

If this is required, extra platform-specific integration work is needed. It will not be covered by this book because it is a very wide topic that is out of scope, but the source release that is covered here is the first step toward a target-specific release.

So let's focus on distributing the buildout folder as it is.

However having the packages folder shipped together with the buildout one along with its sub-folders linked as develop eggs is not the best option, since we would like to release tagged versions for each egg. buildout can interpret any configuration file. So the best practice is to create a dedicated configuration file that does not use the develop option together with a set of built eggs for each package we have created.

So releasing a buildout is done in three steps:

  1. Releasing the packages

  2. Creating the release configuration

  3. Building and preparing the release

Releasing the Packages

Each package can be released as eggs using the sdist, bdist, or bdist_egg command. For our application, since there is no code to compile, a source distribution is enough for all platforms.

For each package, a source distribution is built in the same way as we have seen in the last chapter:

$ python setup.py sdist
running sdist
...
Writing atomisator.db-0.1.0/setup.cfg
tar -cf dist/atomisator.db-0.1.0.tar atomisator.db-0.1.0
gzip -f9 dist/atomisator.db-0.1.0.tar
removing 'atomisator.db-0.1.0' (and everything under it)
$ ls dist/
atomisator.db-0.1.0.tar.gz

The result is an archive that is either pushed to PyPI or stored in a folder.

Adding a Release Configuration File

zc.buildout provides an extension mechanism that will let you create configuration files in layers. Using the extends option that specifies another configuration file, a file can inherit all its values and then add new ones, or override some of them.

A new configuration file dedicated to the releases can be created in the following manner to set specific things in it:

  • We need to point to the buildout released packages.

  • We need to get rid of the develop option.

The result is:

[buildout]
extends = buildout.cfg
develop = 
parts = 
    atomisator
    eggs

download-cache = downloads

[atomisator]
recipe = zc.recipe.eggs
eggs = 
    atomisator.main
    atomisator.db
    atomisator.feed
    atomisator.parser

Here, download-cache is a system folder where the buildout stores eggs downloaded from PyPI. The downloads folder is best created inside the buildout folder:

$ mkdir downloads

The eggs part is inherited from buildout.cfg and does not need to be copied in this new file. The atomisator part will pull released eggs from PyPI and store them in downloads.

Building and Releasing the Application

The buildout can then be built using this specific configuration, using the -c option to point to a specific configuration file, together with the -v option to get more details:

$ bin/buildout -c release.cfg -v
Installing 'zc.buildout', 'setuptools'.
...
Installing atomisator.
Installing 'atomisator.db', 'atomisator.feed', 'atomisator.parser', 'atomisator.main'.
...
Picked: setuptools = 0.6c8

When this step is finished, the packages will be downloaded and stored in the downloads folder:

$ ls downloads/dist/
atomisator.feed-0.1.0.tar.gz
atomisator.main-0.1.0.tar.gz
atomisator.db-0.1.0.tar.gz
atomisator.parser-0.1.0.tar.gz

This means that the packages will not get pulled from PyPI on the next run. In other words, the buildout can be built in an offline mode at this point.

The released version is ready to be shipped by distributing the buildout folder in an archived version, for example.

The last thing to do is to add a bootstrap.py file in the folder to automate the installation of zc.buildout, and the creation of the bin/buildout script on the target system in the same way buildout init does:

$ wget http://ziade.org/bootstrap.py

Note

Some tools in the community provide some scripts to prepare those archived versions with extra options, for instance collective.releaser and zc.sourcerelease.

Summary

We have seen in this chapter that zc.buildout:

  • Can be used to build egg-based applications

  • Knows how to gather eggs together to build an isolated environment

  • Chains recipes, which are small Python packages, to build a script for building the environment

  • Can be used to make source distributions of Python applications

To summarize, working with zc.buildout is done by:

  • Creating a buildout with a list of eggs and using it to develop

  • Creating a configuration file dedicated to releases and using it to build a distributable buildout folder

The next chapter will go further with this tool to explain how projects can be managed with it, together with other tools.