The article won’t attempt to reproduce doit documentation, but will explain how it could be used to solve a specific problem in a practical way. For a complete introduction of doit, and a description of all its features, please refer to the project documentation. Debian packaging or bazaar knowledge isn’t required to follow the discussion, but it would be helpful.

Background

When working on a project’s source code, a developer usually needs to perform different repetitive administrative tasks that are required to compile, test and distribute the source code. In general, those tasks are pretty similar from project to project; although, the details may greatly vary depending on the application type, target platform, software development cycle, etc. As a consequence, the implementation of custom scripts that automate them is needed as a part of the maintenance of the source code.

Given that, this is a very common problem, many task automation tools have been created, make is one of the most well-known among them and is used as a reference to compare with other similar tools. As the reader probably knows, make provides an imperative way to automate small tasks by defining in a file (a makefile) a series of rules that have a target file, multiple dependency files and a set of commands. To reach a given target, make must ensure that target file isn’t outdated and that all the dependency files are present before running the commands that will generate the target file. During this process, the evaluation of other rules might be needed to fulfill the required dependencies.

Although this approach may look simple, it has been really successful in many projects for years. However, since it tries to solve a general problem, it doesn’t perfectly fit in every situation. This fact has led to the creation of similar tools that attempt to address some of the drawbacks of make.:

• The makefile format forces the developer to learn a new mini-language.
• Rules are statically defined.
• Just one target file per rule is allowed.

With the advent of dynamic programming languages, a new generation of make-like tools that solved those issues were designed. For example, rake did a really good job in providing a familiar environment for ruby developers who wanted to use an advanced task automation tool without having to learn something new other than an API.

With regard to python developers, many of these tools are currently available for them with different goals in their designs. One that I find particularly interesting is doit because it doesn’t have any of the make problems listed above. In particular:

• It’s really simple to use because it uses python itself as the language to write the configuration statements needed.
• Tasks, the equivalent to make‘s rules, may have as many targets as needed, which makes things simpler when the execution of a command entails the creation of multiple files.
• Task themselves aren’t defined in the configuration, but task generators. This is really flexible when dependencies and/or targets depend on variables that need to be evaluated at run time.

The problem

Let’s imagine that we are working on checkbox-editor, a simple python project hosted in Launchpad that provides an easy GTK interface to write test cases for checkbox. The way the application is delivered to users is by means of .deb packages for the latest Ubuntu distributions in a Personal Package Archive or PPA, so we’d like to be able to:

• Package the application at any time.
• Install the package locally for testing.
• Upload the package automatically to a PPA.

Fortunately, the project’s trunk branch already has the configuration files needed to generate a .deb package using the usual set of tools, so we’re going to focus on the process of writing the file needed to generate and upload the desired packages. Of course, since we don’t like to waste our time, we only want to generate the files needed for packaging when necessary; that’s is to say, we’re going to follow make‘s approach of generating target files only when they aren’t up-to-date.

Tasks

In this section, a file that contains the tasks generators, which are required to automate the package generation using doit, will be created step by step. The same way as a makefile is created with all the rules for make; in doit, the default file name with the task generators is dodo.py. Of course, another file name can be used by passing an argument to doit, but we’ll stick to the usual name in this example.

In the code snippets that will be displayed in the following sections, some global variables will be used mainly to get the name of some files. For now, just assume that they’re available in the task generators methods. The code that calculates those variables value will be shown at the end of the article.

Identification

There are two different classes of packages: source and binary ones.

Binary packages are the ones that are compiled for an specific architecture, and that are expected to be installed directly into the destination hardware without any problem. These are the type of packages that we need to generate to accomplish the goal of installing a package locally for testing purposes. Hence, two of the tasks that we need to automate are the generation of the binary package and it’s installation.

Source packages are useful to distribute the source code of an application in a platform independent way, so that anyone can take a look at the code, fix it or compile it for another architecture if needed. This is also the package that must be uploaded to a Launchpad PPA, since it will take care to compile it for different architectures and publish the binary packages for them. Consequently, two more tasks that should be automated are the generation of a source package and the upload to the Launchpad PPA.

Before creating any package is generated, we also need to generate a copy of the source code with the latest changes. This is not absolutely needed; but it’s advised since the package generation process creates some temporary files. The diagram of the tasks that have just been identified is the following:

Tasks that should be automated

Code

The first task before any package generation is copying the source code to a new directory (for example, pkg), to keep the development directory clean from the temporary files created during the packaging process.

The code that implements this task is as follows:

 1 def task_code():
 2 """
 3 Create package directory and copy source files
 4 (bzr branch not used to take into account uncommited changes)
 5 """
 6 def copy_file(from_file, to_file):
 7 dir = os.path.dirname(to_file)
 8 if dir and not os.path.isdir(dir):
 9 os.makedirs(dir)
 10 print from_file, '=>', to_file
 11 shutil.copyfile(from_file, to_file)
 12 shutil.copystat(from_file, to_file)
 13 return True
 14 
 15 yield {'name': 'clean',
 16 'actions': None,
 17 'clean': ['rm -rf pkg']}
 18 
 19 for target, dependency in zip(PKG_FILES, SRC_FILES):
 20 yield {'name': dependency,
 21 'actions': [(copy_file, (dependency, target))],
 22 'dependencies': [dependency],
 23 'targets': [target]}