5. Writing a Package – Expert Python Programming

Chapter 5. Writing a Package

This chapter focuses on a repeatable process to write and release Python packages. Its intents are:

  • To shorten the time needed to set up everything before starting the real work, in other words the boiler-plate code

  • To provide a standardized way to write packages

  • To ease the use of a test-driven development approach

  • To facilitate the releasing process

It is organized in the following four parts:

  • A common pattern for all packages that describes the similarities between all Python packages, and how distutils and setuptools play a central role

  • How generative programming ( http://en.wikipedia.org/wiki/Generative_programming) can help this through the template-based approach

  • The package template creation, where everything needed to work is set

  • Setting up a development cycle

A Common Pattern for All Packages

We have seen in the last chapter that the easiest way to organize the code of an application is to split it into several packages using eggs. This makes the code simpler, and easier to understand, maintain, and change. It also maximizes the reusability of each package. They act like components.

Applications for a given company can have a set of eggs glued together with a master egg.

Therefore, all packages can be built using egg structures.

This section presents how a namespaced package is organized, released, and distributed to the world through distutils and setuptools.

Writing an egg, as we have seen in the previous chapter, is done by layering the code in a nested folder that provides a common prefix namespace. For instance, for the Acme company, the common namespace can be acme. The result is a namespaced package.

For example, a package whose code relates to SQL can be called acme.sql. The best way to work with such a package is to create an acme.sql folder that contains the acme and then the sql folder:

setup.py, the Script That Controls Everything

The root folder contains a setup.py script, which defines all metadata as described in the distutils module, combined as arguments in a call to the standard setup function. This function was extended by the third-party library setuptools that provides most of the egg infrastructure.

Note

The boundary between distutils and setuptools is getting fuzzy, and they might merge one day.

Therefore, the minimum content for this file is:

from setuptools import setup
setup(name='acme.sql')

name gives the full name of the egg. From there, the script provides several commands that can be listed with the --help-commands option.

$ python setup.py --help-commands   
Standard commands:
  build             build everything needed to install
  ...
  install           install everything from build directory
  sdist             create a source distribution
  register          register the distribution
  bdist             create a built (binary) distribution
Extra commands:
  develop           install package in 'development mode'
  ...
  test              run unit tests after in-place build
  alias             define a shortcut
  bdist_egg         create an "egg" distribution

The most important commands are the ones left in the preceding listing. Standard commands are the built-in commands provided by distutils, whereas Extra commands are the ones created by third-party packages such as setuptools or any other package that defines and registers a new command.

sdist

The sdist command is the simplest command available. It creates a release tree where everything needed to run the package is copied. This tree is then archived in one or many archived files (often, it just creates one tar ball). The archive is basically a copy of the source tree.

This command is the easiest way to distribute a package from the target system independently. It creates a dist folder with the archives in it that can be distributed. To be able to use it, an extra argument has to be passed to setup to provide a version number. If you don't give it a version value, it will use version = 0.0.0:

from setuptools import setup
setup(name='acme.sql', version='0.1.1')

This number is useful to upgrade an installation. Every time a package is released, the number is raised so that the target system knows it has changed.

Let's run the sdist command with this extra argument:

$ python setup.py sdist
running sdist
...
creating dist
tar -cf dist/acme.sql-0.1.1.tar acme.sql-0.1.1
gzip -f9 dist/acme.sql-0.1.1.tar
removing 'acme.sql-0.1.1' (and everything under it)
$ ls dist/
acme.sql-0.1.1.tar.gz

Note

Under Windows, the archive will be a ZIP file.

The version is used to mark the name of the archive, which can be distributed and installed on any system having Python. In the sdist distribution, if the package contains C libraries or extensions, the target system is responsible for compiling them. This is very common for Linux-based systems or Mac OS because they commonly provide a compiler. But it is less usual to have it under Windows. That's why a package should always be distributed with a pre-built distribution as well, when it is intended to run under several platforms.

The MANIFEST.in File

When building a distribution with sdist, distutils browse the package directory looking for files to include in the archive.

distutils will include:

  • All Python source files implied by the py_modules, packages and scripts option

  • All C source files listed in the ext_modules option

  • Files that match the glob pattern test/test*.py

  • README, README.txt, setup.py, and setup.cfg files

Besides, if your package is under Subversion or CVS, sdist will browse folders such as .svn to look for files to include. sdist builds a MANIFEST file that lists all files and includes them into the archive.

Let's say you are not using these version control systems, and need to include more files. Now, you can define a template called MANIFEST.in in the same directory as that of setup.py for the MANIFEST file, where you indicate to sdist which files to include.

This template defines one inclusion or exclusion rule per line, for example:

include HISTORY.txt
include README.txt
include CHANGES.txt
include CONTRIBUTORS.txt
include LICENSE
recursive-include *.txt *.py

The full list of commands is available at http://docs.python.org/dist/ sdist-cmd.html#sdist-cmd.

build and bdist

To be able to distribute a pre-built distribution, distutils provide the build command, which compiles the package in four steps:

  • build_py: Builds pure Python modules by byte-compiling them and copying them into the build folder.

  • build_clib: Builds C libraries, when the package contains any, using Python compiler and creating a static library in the build folder.

  • build_ext: Builds C extensions and puts the result in the build folder like build_clib.

  • build_scripts: Builds the modules that are marked as scripts. It also changes the interpreter path when the first line was set (!#) and fixes the file mode so that it is executable.

Each of these steps is a command that can be called independently. The result of the compilation process is a build folder that contains everything needed for the package to be installed. There's no cross-compiler option yet in the distutils package. This means that the result of the command is always specific to the system it was build on.

Note

Some people have recently proposed patches in the Python tracker to make distutils able to cross-compile the C parts. So this feature might be available in the future.

When some C extensions have to be created, the build process uses the system compiler and the Python header file (Python.h). This include file is available from the time Python was built from the sources. For a packaged distribution, an extra package called python-dev often contains it, and has to be installed as well.

The C compiler used is the system compiler. For Linux-based system or Mac OS X, this would be gcc. For Windows, Microsoft Visual C++ can be used (there's a free command-line version available) and the open-source project MinGW as well. This can be configured in distutils, as explained in Chapter 1.

The build command is used by the bdist command to build a binary distribution. It calls build and all dependent commands, and then creates an archive in the same was as sdist does.

Let's create a binary distribution for acme.sql under Mac OS X:

$ python setup.py bdist
running bdist
running bdist_dumb
running build
...
running install_scripts
tar -cf dist/acme.sql-0.1.1.macosx-10.3-fat.tar .
gzip -f9 acme.sql-0.1.1.macosx-10.3-fat.tar
removing 'build/bdist.macosx-10.3-fat/dumb' (and everything under it)
$ ls dist/
acme.sql-0.1.1.macosx-10.3-fat.tar.gz    acme.sql-0.1.1.tar.gz

Notice that the newly created archive's name contains the name of the system and the distribution it was built under (Mac OS X 10.3).

The same command called under Windows will create a specific distribution archive:

C:\acme.sql> python.exe setup.py bdist
...
C:\acme.sql> dir dist
25/02/2008  08:18    <DIR>          .
25/02/2008  08:18    <DIR>          ..
25/02/2008  08:24            16 055 acme.sql-0.1.win32.zip
               1 File(s)         16 055 bytes
               2 Dir(s)  22 239 752 192 bytes free

If a package contains C code, apart from a source distribution, it's important to release as many different binary distributions as possible. At the very least, a Windows binary distribution is important for those who don't have a C compiler installed.

A binary release contains a tree that can be copied directly into the Python tree. It mainly contains a folder that is copied into Python's site-packages folder.

bdist_egg

The bdist_egg command is an extra command provided by setuptools. It basically creates a binary distribution like bdist, but with a tree comparable to the one found in the source distribution. In other words, the archive can be downloaded, uncompressed, and used as it is by adding the folder to the Python search path (sys.path).

These days, this distribution mode should be used instead of the bdist-generated one.

install

The install command installs the package into Python. It will try to build the package if no previous build was made and then inject the result into the Python tree. When a source distribution is provided, it can be uncompressed in a temporary folder and then installed with this command. The install command will also install dependencies that are defined in the install_requires metadata.

This is done by looking at the packages in the Python Package Index (PyPI). For instance, to install pysqlite and SQLAlchemy together with acme.sql, the setup call can be changed to:

from setuptools import setup
setup(name='acme.sql', version='0.1.1',
      install_requires=['pysqlite', 'SQLAlchemy'])

When we run the command, both dependencies will be installed.

How to Uninstall a Package

The command to uninstall a previously installed package is missing in setup.py. This feature was proposed earlier too. This is not trivial at all because an installer might change files that are used by other elements of the system.

The best way would be to create a snapshot of all elements that are being changed, and a record of all files and directories created.

A record option exists in install to record all files that have been created in a text file:

$ python setup.py install --record installation.txt
running install
...
writing list of installed files to 'installation.txt'

This will not create any backup on any existing file, so removing the file mentioned might break the system. There are platform-specific solutions to deal with this. For example, distutils allow you to distribute the package as an RPM package. But there's no universal way to handle it as yet.

The simplest way to remove a package at this time is to erase the files created, and then remove any reference in the easy-install.pth file that is located in the site-packages folder.

develop

setuptools added a useful command to work with the package. The develop command builds and installs the package in place, and then adds a simple link into the Python site-packages folder. This allows the user to work with a local copy of the code, even though it's available within Python's site-packages folder. We will see in the next chapter that this is a great feature when building an egg-based application. All packages that are being created are linked with the develop command to the interpreter.

When a package is installed this way, it can be removed specifically with the -u option, unlike the regular install:

$ sudo python setup.py develop
running develop
...
Adding iw.recipe.fss 0.1.3dev-r7606 to easy-install.pth file
Installed /Users/repos/ingeniweb.sourceforge.net/iw.recipe.fss/trunk
Processing dependencies ...
$ sudo python setup.py develop -u
running develop
Removing
...
Removing iw.recipe.fss 0.1.3dev-r7606 from easy-install.pth file

Notice that a package installed with develop will always prevail over other versions of the same package installed.

test

Another useful command is test. It provides a way to run all tests contained in the package. It scans the folder and aggregates the test suites it finds. The test runner tries to collect tests in the package but is quite limited. A good practice is to hook an extended test runner such as zope.testing or Nose that provides more options.

To hook Nose transparently to the test command, the test_suite metadata can be set to 'nose.collector' and Nose added in the test_requires list:

setup(
...
test_suite='nose.collector',
test_requires=['Nose'],
...
)

Note

Chapter 11 presents a few test runners, and explains how to use Nose.

register and upload

To distribute a package to the world, two commands are available:

  • register: This will upload all metadata to a server.

  • upload: This will upload to the server all archives previously built in the dist folder.

The main PyPI server, previously named the Cheeseshop, is located at http://pypi.python.org/pypi and contains over 3000 packages from the community. It is a default server used by the distutils package, and an initial call to the register command will generate a .pypirc file in your home directory.

Since the PyPI server authenticates people, when changes are made to a package, you will be asked to create a user over there. This can also be done at the prompt:

$ python setup.py register
running register
...
We need to know who you are, so please choose either:
 1. use your existing login,
 2. register as a new user,
 3. have the server generate a new password for you (and email it to you), or
 4. quit
Your selection [default 1]:

Now, a .pypirc file will appear in your home directory containing the user and password you have entered. These will be used every time register or upload is called:

[server-index]
username: tarek
password: secret

Note

There is a bug on Windows with Python 2.4 and 2.5. The home directory is not found by distutils unless a HOME environment variable is added. But, this has been fixed in 2.6. To add it, use the technique described in Chapter 1 where we modified the PATH variable. Then add a HOME variable for your user that points to the directory returned by os.path.expanduser('~').

When the download_url metadata or the url is specified, and is a valid URL, the PyPI server will make it available to the users on the project web page as well.

Using the upload command will make the archive directly available at PyPI, so the download_url can be omitted:

Distutils defines a Trove categorization (see PEP 301: http://www.python.org/dev/peps/pep-0301/#distutils-trove-classification) to classify the packages, such as the one defined at Sourceforge. The trove is a static list that can be found at http://pypi.python.org/pypi?%3Aaction=list_classifiers, and that is augmented from time to time with a new entry.

Each line is composed of levels separated by "::":

...
Topic :: Terminals
Topic :: Terminals :: Serial
Topic :: Terminals :: Telnet
Topic :: Terminals :: Terminal Emulators/X Terminals
Topic :: Text Editors Topic :: Text Editors :: Documentation
Topic :: Text Editors :: Emacs
...

A package can be classified in several categories, which can be listed in the classifiers meta-data. A GPL package that deals with low-level Python code (for instance) can use:

Programming Language :: Python
Topic :: Software Development :: Libraries :: Python Modules
License :: OSI Approved :: GNU General Public License (GPL)

Python 2.6 .pypirc Format

The .pypirc file has evolved under Python 2.6, so several users and their passwords can be managed along with several PyPI-like servers. A Python 2.6 configuration file will look somewhat like this:

[distutils]
index-servers =
    pypi
    alternative-server
    alternative-account-on-pypi

[pypi]
username:tarek
password:secret

[alternative-server]
username:tarek
password:secret
repository:http://example.com/pypi

The register and upload commands can pick a server with the help of the -r option, using the repository full URL or the section name:

# upload to http://example.com/pypi
$ python setup.py sdist upload -r alternative-server
# registers with default account (tarek at pypi)
$ python setup.py register
# registers to http://example.com
$ python setup.py register -r http://example.com/pypi

This feature allows interaction with servers other than PyPI. When dealing with a lot of packages that are not to be published at PyPI, a good practice is to run your own PyPI-like server. The Plone Software Center (see http://plone.org/products/plonesoftwarecenter) can be used, for example, to deploy a web server that can interact with distutils upload and register commands.

Creating a New Command

distutils allows you to create new commands, as described in http://docs.python.org/dist/node84.html. A new command can be registered with an entry point, which was introduced by setuptools as a simple way to define packages as plug-ins.

An entry point is a named link to a class or a function that is made available through some APIs in setuptools. Any application can scan for all registered packages and use the linked code as a plug-in.

To link the new command, the entry_points metadata can be used in the setup call:

setup(name="my.command",
          entry_points="""
             [distutils.commands]
             my_command  = my.command.module.Class
          """)

All named links are gathered in named sections. When distutils is loaded, it scans for links that were registered under distutils.commands.

This mechanism is used by numerous Python applications that provide extensibility.

setup.py Usage Summary

There are three main actions to take with setup.py:

  • Build a package.

  • Install it, possibly in develop mode.

  • Register and upload it to PyPI.

Since all the commands can be combined in the same call, some typical usage patterns are:

# register the package with PyPI, creates a source and
# an egg distribution, then upload them
$ python setup.py register sdist bdist_egg upload
# installs it in-place, for development purpose
$ python setup.py develop
# installs it
$ python setup.py install

The alias Command

To make the command line work easily, a new command has been introduced by setuptools called alias. In a file called setup.cfg, it creates an alias for a given combination of commands. For instance, a release command can be created to perform all actions needed to upload a source and a binary distribution to PyPI:

$ python setup.py alias release register sdist bdist_egg upload
running alias
Writing setup.cfg
$ python setup.py release
...

Other Important Metadata

Besides the name and the version of the package being distributed, the most important arguments setup can receive are:

  • description: A few sentences to describe the package

  • long_description: A full description that can be in reStructuredText

  • keywords: A list of keywords that define the package

  • author: The author's name or organization

  • author_email: The contact email address

  • url: The URL of the project

  • license: The license (GPL, LGPL, and so on)

  • packages: A list of all names in the package; setuptools provides a small function called find_packages that calculates this

  • namespace_packages: A list of namespaced packages

A completed setup.py file for acme.sql would be:

import os
from setuptools import setup, find_packages

version = '0.1.0'

README = os.path.join(os.path.dirname(__file__), 'README.txt')
long_description = open(README).read() + '\n\n'

setup(name='acme.sql',
      version=version,
      description=("A package that deals with SQL, "
                   "from ACME inc"),
      long_description=long_description,
      classifiers=[
        "Programming Language :: Python",
        ("Topic :: Software Development :: Libraries ::
         "Python Modules"),
        ],
      keywords='acme sql',
      author='Tarek',
      author_email='tarek@ziade.org',
      url='http://ziade.org',
      license='GPL',
      packages=find_packages(),
      namespace_packages=['acme'],
      install_requires=['pysqlite','SQLAchemy']
      )

Note

The two comprehensive guides to keep under your pillow are:

The distutils guide at http://docs.python.org/dist/ dist.html

The setuptools guide at http://peak.telecommunity.com/DevCenter/setuptools

The Template-Based Approach

The boiler-plate code in acme.sql is composed of a tree of folders that create the namespace of a few files in the root folder. To make all packages follow the same structure, a generic code template can be extracted and provided through a code-generation tool. This approach, called generative programming, is very useful at the organization level. It standardizes the way the code is written and makes developers more productive, as they focus on the code they really need to create. This approach is also a good opportunity to prepare a few things in the package such as complex test fixtures that are common to several packages.

There are numerous generative tools available in the community, but the most used is probably Python Paste ( http://pythonpaste.org).

Python Paste

The Python Paste project was partly responsible for the success of frameworks such as Pylons ( http://pylonshq.com). Developers are driven by an extensive suite of templates that lets them create applications' skeletons within minutes.

From the official tutorial, this is a three-liner to create a web application and run it:

$ paster create -t pylons helloworld
$ cd helloworld
$ paster serve --reload development.ini

The Plone and Zope communities followed this philosophy, and now provide Python Paste templates to generated skeletons as well. ZopeSkel ( http://pypi.python.org/pypi/ZopeSkel) is one of them.

Python Paste contains several tools, and the template engine we are interested in is PasteScript. It can be installed with easy_install. It will get all dependencies from the Paste project:

$ easy_install PasteScript
Searching for PasteScript
Reading http://pypi.python.org/simple/PasteScript/
Reading http://pythonpaste.org/script/
Best match: PasteScript 1.6.2
Downloading
...
Processing dependencies for PasteScript
Searching for PasteDeploy
...
Searching for Paste>=1.3
...
Finished processing dependencies for PasteScript

The paster command will be available with a few default templates than can be listed with the -list-templates option of the create command:

$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  paste_deploy:   A web application deployed through paste.deploy

The basic_package is almost what acme.sql would have needed to build a namespaced package with a setup.py file. When run, the command line asks a few questions and the corresponding answers will be used to fill the templates:

$ paster create -t basic_package mypackage
Selected and implied templates:
  PasteScript#basic_package  A basic setuptools-enabled package

...
Enter version (Version (like 0.1)) ['']: 0.1
Enter description ['']: My package
Enter long_description ['']: this is the package
Enter keywords ['']: package is mine
Enter author (Author name) ['']: Tarek
Enter author_email (Author email) ['']: tarek@ziade.org
Enter url (URL of homepage) ['']: http://ziade.org
Enter license_name (License name) ['']: GPL
Enter zip_safe [False]:
Creating template basic_package
...

The resulting structure is a valid, setuptools-compliant, one-level structure:

$ find mypackage
mypackage
mypackage/mypackage
mypackage/mypackage/__init__.py
mypackage/setup.cfg
mypackage/setup.py

Creating Templates

Python Paste, let's call it the paster, can work with the Cheetah template engine for instance ( http://cheetahtemplate.org), and feed it with the user input.

To create a new template for the paster, three elements have to be provided:

  • A class derived from paste.script.templates.Template

  • The structure to be created that contains folder and files (Cheetah templates or static files)

  • A setuptools entry point to paste.paster_create_template, to register the class

Creating the Package Template

Let's create the template that would have been used for acme.sql.

Note

All templates created in this book, including the package are gathered in pbp.skels that is available for your convenience at PyPI. So if you don't want to create your own from scratch, install it:

$ easy_install pbp.skels

This section has step-by-step instructions explaining how pbp.skels was created.

To create the package template, the first thing to do is to create a structure for this new package:

$ mkdir -p pbp.skels/pbp/skels
$ find pbp.skels
pbp.skels
pbp.skels/pbp
pbp.skels/pbp/skels

Then, an __init__.py file with the following code is created in the pbp folder. It tells distutils to make it a namespaced package:

try:
    __import__('pkg_resources').declare_namespace(__name__)
except ImportError:
    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

Next, create a setup.py file in the root folder (path_to_pbp_package/pbp.skels/__init__.py) with the right metadata. The correct code for this is shown here:

from setuptools import setup, find_packages

version = '0.1.0'
classifiers = [
    "Programming Language :: Python",
    ("Topic :: Software Development :: "
     "Libraries :: Python Modules")]

setup(name='pbp.skels',
      version=version,
      description=("PasteScript templates for the Python "
                   "Best Practice Book."),
      classifiers=classifiers,
      keywords='paste templates',
      author='Tarek Ziade',
      author_email='tarek@ziade.org',
      url='http://atomisator.ziade.org',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['pbp'],
      include_package_data=True,
      install_requires=['setuptools',
                        'PasteScript'],
      entry_points="""
      # -*- Entry points: -*-
      [paste.paster_create_template]
      pbp_package = pbp.skels.package:Package
      """)

The entry point adds a new template that will be available in the paster.

The next step is to write the Package class in the pbp/skels folder, in a module called package:

from paste.script.templates import var
from paste.script.templates import Template

class Package(Template):
    """Package template"""
    _template_dir = 'tmpl/package'
    summary = "A namespaced package with a test environment"
    use_cheetah = True

    vars = [
        var('namespace_package', 'Namespace package',
            default='pbp'),
        var('package', 'The package contained',  
            default='example'),
        var('version', 'Version', default='0.1.0'),
        var('description',
            'One-line description of the package'),
        var('author', 'Author name'),
        var('author_email', 'Author email'),
        var('keywords', 'Space-separated keywords/tags'),
        var('url', 'URL of homepage'),
        var('license_name', 'License name', default='GPL')
        ]

    def check_vars(self, vars, command):
        if not command.options.no_interactive and \
           not hasattr(command, '_deleted_once'):
            del vars['package']
            command._deleted_once = True
        return Template.check_vars(self, vars, command)

This class defines:

  • The folder containing the template structure (_template_dir)

  • A summary of the template that will appear in the paster

  • A flag to indicate if Cheetah is used in the template structure

  • A list of variables, where each variable is composed of a name, a label, and a default value (if needed), which is used by the paster to ask the user at the prompt to enter his or her values

  • A check_vars method that makes sure the package variable will be requested at the prompt

The last thing to do is to create the tmpl/package directory content by copying the one created for acme.sql. All files that contain values to be changed, such as the namespace, have to be suffixed by _tmpl. The values are replaced by ${variable}, where variable is the name of the variable listed in the Package class.

The setup.py file (for instance) becomes setup.py_tmpl and contains:

from setuptools import setup, find_packages
import os

version = ${repr($version) or "0.0"}
long_description = open("README.txt").read()
classifiers = [
    "Programming Language :: Python",
    ("Topic :: Software Development :: "
     "Libraries :: Python Modules")]

setup(name=${repr($project)},
      version=version,
      description=${repr($description) or $empty},
      long_description=long_description,
      classifiers=classifiers,
      keywords=${repr($keywords) or $empty},
      author=${repr($author) or $empty},
      author_email=${repr($author_email) or $empty},
      url=${repr($url) or $empty},
      license=${repr($license_name) or $empty},
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=[${repr($namespace_package)}],
      include_package_data=True,
      install_requires=[
          'setuptools',
          # -*- Extra requirements: -*-
      ],
      entry_points="""
      # -*- Entry points: -*-
      """,
      )

The repr function will tell Cheetah to add quotes around the string values.

You can use the same technique for all files located in acme.sql to make a template. For instance, the README.txt file is copied to README.txt_tmpl. Then all references to acme.sql are replaced by values defined in the Package class in the vars list.

For instance, getting the full package name is done by:

${namespace_package}.${package}

Last, to use a variable value for a folder name it has to be named with a "+" prefix and suffix. For instance, the namespaced package folder will be called +namespace_package+ and the package folder +package+.

The final structure of pbp.skeles, after the acme.sql has been generalized, will look like this:

$ cd pbp.skels
$ find .
setup.py
pbp
pbp/__init__.py
pbp/skels
pbp/skels/__init__.py
pbp/skels/package.py
pbp/skels/tmpl
pbp/skels/tmpl/package
pbp/skels/tmpl/package/README.txt_tmpl
pbp/skels/tmpl/package/setup.py_tmpl
pbp/skels/tmpl/package/+namespace_package+
pbp/skels/tmpl/package/+namespace_package+/__init__.py_tmpl
pbp/skels/tmpl/package/+namespace_package+/+package+
pbp/skels/tmpl/package/+namespace_package+/+package+/__init__.py
---

From there, the package can be symlinked to Python's site-packages directory with a develop command, and made available to the paster:

$ python setup.py develop
...
Finished processing dependencies for pbp.skels==0.1.0dev

After the develop command is run, you should find the template listed in paster :

$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  pbp_package:    A namespaced package with a test environment
  paste_deploy:   A web application ... paste.deploy
$ paster create -t pbp_package trying.it
Selected and implied templates:
  pbp.skels#package  A namespaced package with a test environment

Variables:
  egg:      trying.it
  package:  tryingit
  project:  trying.it
Enter namespace_package (Namespace package) ['pbp']: trying
Enter package (The package contained) ['example']: it
...
Creating template package
...

The generated tree will then contain the structure ready to work with right away.

Development Cycle

The development cycle of a package is composed of iterations, where the code is moved forward from an initial state to a new state. This phase lasts mostly for a few weeks and ends with a release. This does not happen in small packages that are very simple to work with, but can be found in all packages that have enough modules to make it worthwhile.

At the end of the iteration, a release is created with the commands we have previously seen. The package moves at this moment from a development state to a releasable state, and the delivered code can be seen as an official release.

Then a new cycle starts with an incremented version for the package.

What Version Numbers Should be Used?

There are no fixed conventions for incrementing a package's version number, and when developers feel the software has grown a lot, they often jump to a higher number that does not follow the previous series.

Most software usually start with a very small value and uses two or three digits. Sometimes an alphabet letter is appended to it when they are trying to finalize a version. rc suffixes are also used to mark a release candidate. That is a version in test phase where some fixes might be done:

  • 0.1, 0.2, 0.3

  • 0.1.0, 0.1.1, 0.1.2a, 0.1.2b

  • 0.1, 0.2rc1, 0.2rc2

You should decide of your own convention as long as the versions stay consistent all the way. In companies, there are usually standards followed by all applications; whereas open-source applications have their own conventions.

The only rule that should be applied is to make sure that the number of digits is always the same, and avoid the "-" sign in the version, because it is used as a separator by many tools to extract a version number from a package name.

For instance, these should be avoided:

  • 0.1, 0.1.1-alpha, 0.1.1-b, 0.2

  • 0.1, 0.1-a, 0.1-b

Nightly Builds

If the package is still releasable anytime during the iteration, development releases can be made. Those are also called nightly builds. This continuous releasing process allows developers to get live feedback on their work, and save beta users some work. They don't need to get the code from a version repository, for instance, and can install the development release like a regular one.

To differentiate a development release from a regular release, the user has to append the dev suffix to the version number. For instance, the 0.1.2 version that is being developed and not yet released, will be known as the 0.1.2dev release.

distutils provide a way to mark this, by adding in a setup.cfg file a section that informs the build command about the development state:

[egg_info]
tag_build = dev

This will automatically add the dev prefix added to the version:

$ python setup.py bdist_egg
running bdist_egg
running egg_info
...
creating 'dist/iw.selenium-0.1.0dev-py2.4.egg'

Another useful tag can be the revision number when the package is living in Subversion repository. It can be appended with the tag_svn_revision flag:

[egg_info]
tag_build = dev
tag_svn_revision = true

The revision number will appear in the version as well in that case.

$ python setup.py bdist_egg
running bdist_egg
running egg_info
...
creating 'dist/iw.selenium-0.1.0dev_r38360-py2.4.egg'

The simplest way is to always keep this file in the trunk and remove it right before making a regular release. In other words, a releasing process with Subversion can be:

  • Make a tag copy of the trunk.

  • Check out the tag branch.

  • Remove the setup.cfg (or the egg_info-specific section) in this branch and commit the change.

  • Build the release from there.

  • Raise the version number in the trunk.

This looks as follows:

$ svn cp http://example.com/my.package/trunk http://example.com/my.package/tags/0.1
$ svn co http://example.com/my.package/tags/0.1 0.1
$ cd 0.1
$ svn rm setup.cfg
$ svn ci -m "removing the dev flag""
$ python setup.py register sdist bdist_egg upload

Note

Chapter 8 explains what Version Control Systems, such as Subversion, are and how they work.

Summary

In this chapter we have seen:

  • How a namespaced package is created

  • The central role of setup.py, and how to use it to build and release the package

  • The template-based approach to generated package skeletons

  • How The Paster works and how to create a package skeleton

  • How to release the package and provide nightly builds

The next chapter will focus on the same topics, but at the application level.