9. Managing Life Cycle – Expert Python Programming

Chapter 9. Managing Life Cycle

Managing a software development is hard. Often, projects are delivered late; and in some cases, they are even dropped. Modern software management has created methods to reduce risks. And the most common approach that has proven its efficiency is using an iterative development approach. Many methodologies exist that use an iterative approach. They are commonly named agile methodologies.

This chapter will not provide a complete software management guide, as this would require an entire book. (You might want to read Agile and Iterative Development: A Manager's Guide from Addison-Wesley.)

It will rather give some tips and a summary on how to manage a software life cycle based on iterations, and how this can be done with a few tools.

Different Approaches

Before presenting how an iterative life cycle can be set, let's describe a few development models that exist in the software industry.

Waterfall Development Model

The waterfall development model treats software as a whole, where each phase is started only when the previous phase is over. In other words, the work is organized into a sequence of phases, which can be:

  • Analyze the needs.

  • Design the global architecture and how the software is organized in pieces.

  • Design each piece.

  • Code each piece with a TDD (Test-Driven Development) approach. (Some people don't use TDD in this method.)

  • Reunite pieces and do global system tests.

  • Deploy.

Therefore, the work organization looks like a waterfall, as shown in the illustration:

This model is used in many large companies, where each step is carefully done and reviewed before the next one is started. This means that after the designing is over, developers just have to implement that design. The final step is where all pieces are gathered and tested together.

This model is quite rigid, since it is almost impossible to think about all aspects of software before it is globally tested. In reality, the final step often reveals some inconsistency, or missing pieces, or even performance issues due to a design flaw.

Such a model is probably easier to apply with a really well-known target environment, and with an experienced team. But for most software, this is impossible.

Spiral Development Model

The spiral model is based on feedback over prototypes. A first version of the program is created based on the initial requirements, without any polishing. This prototype is then refined depending on the feedback it receives. Its weaknesses and strengths are pinpointed so that it can be refactored.

After a few cycles, when everyone agrees that the prototype meets the needs, it is polished and then delivered.

This model reduces the risk a lot as developers can start to code earlier on the software, without having to deal with a stone-set design, like that which the Waterfall model induces. Managers have a better overview of the time needed to finish the project after the first cycle is over.

Iterative Development Model

The Iterative model is similar to the Spiral model, but doesn't look at the application as a whole. Instead, it focuses on delivering some new features to the system, and reworking the existing ones through feedback.

This means that unlike the Spiral and Waterfall models, analysis, design, and coding occur during the whole project since they only concern a subset of functionalities on each cycle.

Therefore, a project is cut into several iterations that can be seen as independent projects, where the software is designed, built, and delivered. Every involved party focuses on the functionalities of the current iteration through cross-disciplined work.

Providing feedback at each iteration helps a lot in making all involved people work smoothly together. Each functionality can evolve a lot before it sticks to reality, and often differs from the initial thoughts people had. Everyone gets some food for thought, and can correct the analysis and design on the next iteration.

Since this gradual approach concerns smaller parts of the system, it raises the frequency of feedback. An iteration often lasts from one to four weeks, and a software requires several iterations before it can be delivered. Since the whole software grows after each iteration, it is built incrementally.

Many methodologies are based on iterative approaches, such as Scrum or XP. And this is a common base to all agile methods.

However, an iterative approach is not an excuse to be lazy about design. It has a cost. In order to maintain flexibility, agile programmers heavily emphasize testing. By writing tests, they ensure that they do not break their existing code when changing it.

A rule of thumb in such an iterative approach is to define a fixed length for each iteration, and to make something special at its end.

This chapter will propose a generic model that is a common base to many open-source projects, and will describe it through a set of tools that can be used for this purpose.

Defining a Life Cycle

Defining a life cycle for a project consists of planning regular releases and trying to keep up the pace. It is often better to postpone some elements that are not ready by the end of a cycle, and keep up with the promised calendar. This is called the train approach: When the train leaves, get in or stay there. However, some projects move the release date rather than removing some features. But this approach often seems less predictable.

After the project has started, through an initial initiation phase, a global planning is defined. Basically, people join at a kick-off meeting and define together when the software should be delivered, and then cut the remaining time into iterations.

Depending on the nature of the software, these iterations can last from a week to four or five weeks. Each iteration should have the same length in order to keep the same rhythm throughout the project, as displayed in the following figure:

The iteration length might vary when the project reaches a stable state: It can last longer.

An iteration is a small independent project that can be composed of four phases:

  • The Planning phase, where the work to be done is defined

  • The Analysis, Design, and Test-Driven Development phase, where the work is done

  • A Cleaning phase, where the work done is globally tested and debugged

  • The Release phase, where the work is delivered

For an iteration that lasts for two weeks, which is ten working days, the duration for each phase can be:

  • One day for planning

  • Seven days for development

  • A few days for global cleaning

  • One day for release

Planning

The planning phase of an iteration consists of defining a series of tasks that have to be done. A task is an atomic operation over the code that can be done by a developer. Each task's duration is estimated to make sure the amount of work is realistic, given the remaining time and the number of developers involved.

This planning is done by the manager, with developers' feedback. Estimates must be validated to be accurate by the person who does the work. By the end of this step, the iteration should be clearly defined through a list of tasks to be achieved.

Development

The development phase consists of tasks for each developer to work on. This is done by:

  • Accepting the task to be done, so everyone knows it is being processed

  • Reviewing and correcting the estimated time, so the manager knows if it was not estimated correctly; this reviewing is also important to know if the workload for the iteration is realistic

  • Coding

  • Closing the task when it is done

  • Taking another task and doing the same work on it

Global Debug

The global debug phase closes iteration. The whole software is tested and the remaining tasks are worked out.

The most efficient way to perform this final step is to gather all developers and managers in a special event.

The remaining tasks that could not be closed are postponed to the next release, and some showstoppers are often found during this final stage.

Release

The releasing phase consists of:

  • Tagging the code and creating the release, as explained in the previous chapter

  • Launching a new iteration

Setting Up a Tracking System

The planning phases define tasks to be done within an iteration. To keep track of these tasks, a tracking system can be used.

Such an application is used to keep relevant information for each task, such as:

  • The person in charge

  • The nature of the change: new feature, debug, refactoring, and so on

  • The due date

  • The iteration concerned

  • The status: open, fixed, and so on

  • The estimated charge

Each task can be put in an iteration, and the software needs to provide a global display of this iteration. During the development phase of an iteration, the status of each task is updated by developers, so that the global status can be followed.

Trac

Trac is a good candidate for such software. This wiki-based web application provides a complete issue-tracking system together with a lot of useful features:

  • A user management system

  • A fully editable wiki-based interface

  • A plug-in system that allows adding new features to the software

Trac interacts smoothly with most version control systems through plug-ins that wire the repository with the web interface. From there, the repository tree can be browsed through the Web and a live timeline displays the commit, together with readable diff reports.

This tool is often used in the open-source community as a project's website, and provides all needed features for developers to work with the code. Here are some examples of software that uses Trac:

Note

Trac maintains a Who Uses Trac? page here: http://trac.edgewall.org/wiki/TracUsers. If your project uses it, you can add it there.

The minimalistic approach of its interface makes it simple to understand and use.

Installation

The pbp.recipe.trac recipe available at PyPI provides a fast way to set up a Trac instance together with the Mercurial server we have installed.

A new section trac can be added in the buildout.cfg file that we created in the previous chapter for Buildbot:

[buildout]
parts =
    buildmaster
    linux
    trac

[buildmaster]
...

[buildslave]
...

[trac]
recipe = pbp.recipe.trac

project-name = Atomisator
project-url = ${buildmaster:project-url}
repos-type = hg
repos-path = /home/mercurial/atomisator/repositories/unstable
buildbot-url = http://buildbot-atomisator.ziade.org/

components =
    atomisator.db       tarek
    atomisator.feed     tarek
    atomisator.main     tarek
    atomisator.parser   tarek
    pbp.recipe.trac     tarek

header-logo = atomisator.png

The new section defines:

  • A project name that will be used as a title

  • A project url that will be used in the Trac instance as the home page

  • A repository type, hg in our case, that will be used to install the correct plug-in

  • A repository path that points the repository, which has to be on the same server

  • A Buildbot url that will be linked to a navigation button on the Trac navigation bar

  • A component list with a component name and an owner, which will be used in the issue tracker

  • A header-logo, which points to an image that will be used to replace the Trac logo in the header

The logo in our example is put in the buildout folder.

Let's run the buildout instance dedicated to Buildbot and Trac again:

$ bin/buildout -v
...
Project environment for 'Atomisator' created.
...
Try running the Trac standalone web server `tracd`:
tracd --port 8000 /home/mercurial/atomisator/buildbot/parts/trac
...
Creating new milestone 'future'
Creating new component 'atomisator.db'
Creating new component 'atomisator.feed'
Creating new component 'atomisator.main'
Creating new component 'atomisator.parser'

A Trac environment is added in parts/trac, and two new scripts are added into the bin directory:

  • tracd: A standalone web server that can be used to run the Trac instance

  • trac-admin: A command-line shell that can be used to manage the instance

Try to run the tracd script from the buildout folder:

$ bin/tracd --port 8000 parts/trac
Server starting in PID 24971.
Serving on 0.0.0.0:8000 view at http://127.0.0.1:8000/

The Trac instance should be reachable in a browser at http://127.0.0.1:8000/trac, as shown in the figure that follows:

Notice that the Buildbot button allows the user to visit the Buildbot web page.

Apache Settings

Like Buildbot, Trac can be hooked to Apache through several handlers. The easiest way is to use the mod_python handler, which can be wired to the front end Trac provides.

mod_python can be installed on Debian Linux with:

$ sudo apt-get install libapache2-mod-python

For other platforms, refer to the project page: http://www.modpython.org.

From there, a new host can be added in the Apache configuration:

<VirtualHost *:80>
  ServerName atomisator.ziade.org

  <Location />
    SetHandler mod_python
    PythonHandler trac.web.modpython_frontend
    PythonOption TracEnv /home/mercurial/atomisator/buildbot/parts/trac
    PythonOption TracUriRoot /
    PythonPath "sys.path + ['/home/mercurial/atomisator/buildbot/parts/trac', '/home/mercurial/atomisator/buildbot/eggs']"
  </Location>

  <Location "/login">
    AuthType Basic
    AuthName "Trac login"
    AuthUserFile /home/mercurial/atomisator/passwords
    Require valid-user
  </Location>

</VirtualHost>

A few remarks on this configuration:

  • The PythonOption defines a TracEnv value, so the Trac system knows where the instance is located.

  • The PythonPath option points to the local buildout directories needed by the script to access Trac modules.

  • The /login section's settings hook the passwords file we previously created for Mercurial, so users can log into the system with the same username.

Permission Settings

To work with the issue management system, we need to define a few groups:

  • manager: A person who is able to fully manage Trac

  • developer: A person who is able to modify the tickets and change the wiki's pages

  • authenticated: A person who is able to create a ticket

  • anonymous: A person who is able to view everything

These four roles are already set and available in Trac. This was either a default permission setting, or a setting automatically done by pbp.recipe.trac when the buildout was run.

The work that's left is to add some people in each group. Trac provides a command-line utility that can be used to set up a few elements in the instance:

$ bin/trac-admin parts/trac/
Welcome to trac-admin 0.11b2
Interactive Trac administration console.
Copyright (c) 2003-2007 Edgewall Software

Type:  '?' or 'help' for help on commands.
        
Trac [parts/trac]>

From there you can add a user to a group. Let's define tarek as a manager, and bill and bob as developers:

Trac [parts/trac]> permission add tarek manager
Trac [parts/trac]> permission add bob developer
Trac [parts/trac]> permission add bill developer

This will allow tarek to manage the project through the Web, and bill and bob to deal with the tickets and the wiki pages.

Project Life Cycle with Trac

These settings make Trac really easy to use when dealing with the life cycle, through the web interface and with the command-line tool.

Planning

The planning is done in Trac by creating a new milestone that corresponds to the iteration, and deciding when it should be delivered. The manager adds it either through the web interface or through trac-admin command line. The latter is more convenient, but it means you will have to connect to the server to perform those tasks.

Let's use the command line to add it:

Trac [parts/trac] > milestone add atomisator-0.1.0
Trac [parts/trac]> milestone due atomisator-0.1.0 2008-08-01

The same operation can be performed in the admin section of the web interface at /admin/ticket/milestones.

This milestone will then appear in the roadmap section. From there, tickets can be added to the milestone through the web interface by hitting the New Ticket button.

Let's create a ticket that defines a task in atomisator.db, defining a mapper to store the feed entries, as shown in this figure:

Besides the summary and description, the important information to be provided is:

  • Assign to: The name of the developer (let's assign it to Bob )

  • Type: Task, since this is a new feature

  • Component: atomisator.db

  • Milestone: atomisator-0.1.0

  • Estimated hours: 8

The ticket will then appear in the milestone.

Bugs and enhancements are entered in the same way.

Note

The estimated hours are part of a Trac plug-in called TimeAndEstimation, automatically installed by pbp.recipe.trac.

See http://trac-hacks.org/wiki/TimingAndEstimationPlugin.

Development

Each developer can view his or her tasks through the reports available under the /report section, accessible through the View Tickets button. The My Tickets report will display a list of tickets for the current user.

When Bob starts to work on the ticket we have entered previously, he performs the accept action on it.

When the task is over, he fills the Total Hours field and performs the resolve action.

Note

This lightweight time management is not replacing a real management planning system, but will provide a useful indication on how the time was spent on tasks.

Tracking time is an important part of improving estimates.

Cleaning

Often, when the team ends an iteration, some tasks are remaining. The cleaning phase is a good opportunity to close a maximum number of small issues. Some teams organize bug sprints, and work on bugs in a day or two.

At the end of the phase, the remaining tasks are postponed to the future milestone, unless they are showstoppers. In that case they must be fixed, hopefully before the end of the phase.

All these operations are done in the web interface by editing each task.

Release

Releasing consists of:

  • Tagging the code

  • Pulling the changes from the unstable repository to the stable repository

  • Creating the release repository

  • Preparing and shipping a release

  • Closing the milestone by setting its state to completed

  • Creating a new milestone and starting a new cycle: some teams create up to three future milestones, to be able to push tickets to future milestones with priorities

The tagging-and-pulling work occurs on the Mercurial side, as explained in the previous chapter, with the following set of commands:

$ cd /home/mercurial/atomisator/repositories/unstable
$ hg tag -f -m "tag for 0.1.0 release" 0.1.0
$ cd ../stable
$ hg pull ../unstable
$ hg clone

Last, the milestone is closed by completing its due date:

Trac [parts/trac]> milestone completed atomisator-0.1.0 2008-08-01

Trac can be used as the central place of information, and the created releases can be added and announced on the wiki pages.

Summary

In this chapter we have:

  • Described various ways to manage a software's development life cycle

  • Explained why an iterative approach is great

  • Described how to organize a project in iterations

  • Explained how to install and use Trac

The next chapter will focus on how to document your software.