|Read more about this book|
(For more resources on Python, see here.)
So let’s get on with it!
Tests tell you when the code you’re testing doesn’t work the way you thought it would, but they don’t tell you a thing about the code that you’re not testing. They don’t even tell you that the code you’re not testing isn’t being tested.
Code coverage is a technique, which can be used to address that shortcoming. A code coverage tool watches while your tests are running, and keeps track of which lines of code are (and aren’t) executed. After the tests have run, the tool will give you a report describing how well your tests cover the whole body of code.
It’s desirable to have the coverage approach 100%, as you probably figured out already. Be careful not to focus on the coverage number too intensely though, it can be a bit misleading. Even if your tests execute every line of code in the program, they can easily not test everything that needs to be tested. That means you can’t take 100% coverage as certain proof that your tests are complete. On the other hand, there are times when some code really, truly doesn’t need to be covered by the tests—some debugging support code, for example—and so less than 100% coverage can be completely acceptable.
Code coverage is a tool to give you insight into what your tests are doing, and what they may be overlooking. It’s not the definition of a good test suite.
We’re going to be working with a module called coverage.py, which is—unsurprisingly—a code coverage tool for Python.
Since coverage.py isn’t built in to Python, we’ll need to download and install it. You can download the latest version from the Python Package Index at http://pypi.python.org/pypi/coverage. As before, users of Python 2.6 or later can install the package by unpacking the archive, changing to the directory, and typing:
$ python setup.py install --user
Users of older versions of Python need write permission to the system-wide site-packages directory, which is part of the Python installation. Anybody who has such permission can install coverage by typing:
$ python setup.py install
We’ll walk through the steps of using coverage.py here, but if you want more information you can find it on the coverage.py home page at http://nedbatchelder.com/code/coverage/.
Time for action – using coverage.py
We’ll create a little toy code module with tests, and then apply coverage.py to find out how much of the code the tests actually use.
- Place the following test code into test_toy.py. There are several problems with these tests, which we’ll discuss later, but they ought to run.
from unittest import TestCase
self.assertEqual(toy.global_function(2**13), 2**13 + 1)
example = toy.example_class(5)
example = toy.example_class(7)
self.assertEqual(repr(example), '<example param="7">')
- Put the following code into toy.py. Notice the if __name__ == ‘__main__’ clause at the bottom. We haven’t dealt with one of those in a while, so I’ll remind you that the code inside that block runs doctest if we were to run the module with python toy.py.
return x + 1
def __init__(self, param):
self.param = param
return self.param * 2
return '<example param="%s">' % self.param
if __name__ == '__main__':
- Go ahead and run Nose. It should find them, run them, and report that all is well. The problem is, some of the code isn’t ever tested.
- Let’s run it again, only this time we’ll tell Nose to use coverage.py to measure coverage while it’s running the tests.
$ nosetests --with-coverage --cover-erase
What just happened?
In step 1, we have a couple of TestCase classes with some very basic tests in them. These tests wouldn’t be much use in a real world situation, but all we need them for is to illustrate how the code coverage tool works.
In step 2, we have the code that satisfies the tests from step 1. Like the tests themselves, this code wouldn’t be much use, but it serves as an illustration.
In step 4, we passed –with-coverage and –cover-erase as command line parameters when we ran Nose. What did they do? Well, –with-coverage is pretty straightforward: it told Nose to look for coverage.py and to use it while the tests execute. That’s just what we wanted. The second parameter, –cover-erase, tells Nose to forget about any coverage information that was acquired during previous runs. By default, coverage information is aggregated across all of the uses of coverage.py. This allows you to run a set of tests using different testing frameworks or mechanisms, and then check the cumulative coverage. You still want to erase the data from previous test runs at the beginning of that process, though, and the –cover-erase command line is how you tell Nose to tell coverage.py that you’re starting anew.
What the coverage report tells us is that 9/12 (in other words, 75%) of the executable statements in the toy module were executed during our tests, and that the missing lines were line 16 and a lines 19 through 20. Looking back at our code, we see that line 16 is the __repr__ method. We really should have tested that, so the coverage check has revealed a hole in our tests that we should fix. Lines 19 and 20 are just code to run doctest, though. They’re not something that we ought to be using under normal circumstances, so we can just ignore that coverage hole.
Code coverage can’t detect problems with the tests themselves, in most cases. In the above test code, the test for the timestwo method violates the isolation of units and invokes two different methods of example_class. Since one of the methods is the constructor, this may be acceptable, but the coverage checker isn’t in a position to even see that there might be a problem. All it saw was more lines of code being covered. That’s not a problem— it’s how a coverage checker ought to work— but it’s something to keep in mind. Coverage is useful, but high coverage doesn’t equal good tests.
Version control hooks
Most version control systems have the ability to run a program that you’ve written in response to various events, as a way of customizing the version control system’s behavior. These programs are commonly called hooks.
Version control systems are programs for keeping track of changes to a source code tree, even when those changes are made by different people. In a sense, they provide an universal undo history and change log for the whole project, going all the way back to the moment you started using the version control system. They also make it much easier to combine work done by different people into a single, unified entity, and to keep track of different editions of the same project.
You can do all kinds of things by installing the right hook programs, but we’ll only focus on one use. We can make the version control program automatically run our tests, when we commit a new version of the code to the version control repository.
This is a fairly nifty trick, because it makes it difficult for test-breaking bugs to get into the repository unnoticed. Somewhat like code coverage, though there’s potential for trouble if it becomes a matter of policy rather than simply being a tool to make your life easier.
In most systems, you can write the hooks such that it’s impossible to commit code that breaks tests. That may sound like a good idea at first, but it’s really not. One reason for this is that one of the major purposes of a version control system is communication between developers, and interfering with that tends to be unproductive in the long run. Another reason is that it prevents anybody from committing partial solutions to problems, which means that things tend to get dumped into the repository in big chunks. Big commits are a problem because they make it hard to keep track of what changed, which adds to the confusion. There are better ways to make sure you always have a working codebase socked away somewhere, such as version control branches.
Bazaar is a distributed version control system, which means that it is capable of operating without a central server or master copy of the source code. One consequence of the distributed nature of Bazaar is that each user has their own set of hooks, which can be added, modified, or removed without involving anyone else. Bazaar is available on the Internet at http://bazaar-vcs.org/.
If you don’t have Bazaar already installed, and don’t plan on using it, you can skip this section.
Time for action – installing Nose as a Bazaar post-commit hook
- Bazaar hooks go in your plugins directory. On Unix-like systems, that’s ~/.bazaar/plugins/, while on Windows it’s C:Documents and Settings<username>Application DataBazaar<version>plugins. In either case, you may have to create the plugins subdirectory, if it doesn’t already exist.
- Place the following code into a file called run_nose.py in the plugins directory. Bazaar hooks are written in Python:
from bzrlib import branch
from os.path import join, sep
from os import chdir
from subprocess import call
def run_nose(local, master, old_num, old_id, new_num, new_id):
base = local.base
base = master.base
if not base.startswith('file://'):
'Runs Nose after each
- Make a new directory in your working files, and put the following code into it in a file called test_simple.py. These simple (and silly) tests are just to give Nose something to do, so that we can see that the hook is working.
from unittest import TestCase
- Still in the same directory as test_simple.py, run the following commands to create a new repository and commit the tests to it. The output you see might differ in details, but it should be quite similar overall.
$ bzr init
$ bzr add
$ bzr commit
- Notice that there’s a Nose test report after the commit notification. From now on, any time you commit to a Bazaar repository, Nose will search for and run whatever tests it can find within that repository.
What just happened?
Bazaar hooks are written in Python, so we’ve written our hook as a function called run_nose. Our run_nose function checks to make sure that the repository which we’re working on is local, and then it changes directories into the repository and runs nose. We registered run_nose as a hook by calling branch.Branch.hooks.install_named_hook.