Grok offers some tools for testing, and in fact, a project created by grokproject (as the one we have been extending) includes a functional test suite. In this article, we are going to discuss testing a bit and then write some tests for the functionality that our application has so far.

Testing helps us avoid bugs, but it does not eliminate them completely, of course. There are times when we will have to dive into the code to find out what’s going wrong. A good set of debugging aids becomes very valuable in this situation. We’ll see that there are several ways of debugging a Grok application and also try out a couple of them.

Testing

It’s important to understand that testing should not be treated as an afterthought.

As mentioned earlier, agile methodologies place a lot of emphasis on testing. In fact, there’s even a methodology called Test Driven Development (TDD), which not only encourages writing tests for our code, but also writing tests before any other line of code.

There are various kinds of testing, but here we’ll briefly describe only two:

• Unit testing
• Integration or functional tests

Unit testing

The idea of unit testing is to break a program into its constituent parts and test each one of them in isolation. Every method or function call can be tested separately to make sure that it returns the expected results and handles all of the possible inputs correctly.

An application which has unit tests that cover the majority of its lines of code, allows its developers to constantly run the tests after a change, and makes sure that modifications to the code do not break the existing functionality.

Functional tests

Functional tests are concerned with how the application behaves as a whole. In a web application, this means how it responds to a browser request and whether it returns the expected HTML for a given call.

Ideally, the customer himself has a hand in defining these tests, usually through explicit functionality requirements or acceptance criteria. The more formal the requirements from the customer are, the easier it is to define appropriate functional tests.

Testing in Grok

Grok highly encourages the use of both kinds of tests, and in fact, includes a powerful testing tool that is automatically configured with every project. In the Zope world—from where Grok originated—a lot of value is placed in a kind of tests known as “doctests”, so Grok comes with a sample test suite of this kind.

Doctests

A doctest is a test that’s written as a text file, with lines of code mixed with explanations of what the code is doing. The code is written in a way that simulates a Python interpreter session. As tests exercise large portions of the code (ideally 100%), they usually offer a good way of finding out of what an application does and how. So, if an application has no written documentation, its tests would be the next obvious way of finding out what it does. Doctests take this idea further by allowing the developer to explain in the text file exactly what each test is doing.

Doctests are especially useful for functional testing, because it makes more sense to document the high-level operations of a program. Unit tests, on the other hand, are expected to evaluate the program bit by bit and it can be cumbersome to write a text explanation for every little piece of code.

A possible drawback of doctests is that they can make the developer think that he needs no other documentation for his project. In almost all of the cases, this is not true. Documenting an application or package makes it immediately more accessible and useful, so it is strongly recommended that doctests should not be used as a replacement for good documentation. We’ll show an example of using doctests in the Looking at the test code section of this article.

Default test setup for Grok projects

As mentioned above, Grok projects that are started with the grokproject tool already include a simple functional test suite by default. Let’s examine it in detail.

Test configuration

The default test configuration looks for packages or modules that have the word ‘tests’ in their name and tries to run the tests inside. For functional tests, any files ending with .txt or .rst are considered.

For functional tests that need to simulate a browser, a special configuration is needed to tell Grok which packages to initialize in addition to the Grok infrastructure (usually the ones that are being worked on). The ftesting.zcml file in the package directory has this configuration. This also includes a couple of user definitions that are used by certain tests to examine functionality specific to a certain role, such as manager.

Test files

Besides the already mentioned ftesting.zcml file, in the same directory, there is a tests.py file added by grokproject, which basically loads the ZCML declarations and registers all of the tests in the package.

The actual tests that are included with the default project files are contained in the app.txt file. These are doctests that do a functional test run by loading the entire Grok environment and imitating a browser. We’ll take a look at the contents of the file soon, but first let’s run the tests.

Running the tests

As part of the project’s build process, a script named test is included in the bin directory when you create a new project. This is the test runner and calling it without arguments, finds and executes all of the tests in the packages that are included in the configuration.

We haven’t added a single test so far, so if we type bin/test in our project directory, we’ll see more or less the same thing that doing that on a new project would show:

$ bin/test
Running tests at level 1 
Running todo.FunctionalLayer tests: 
 Set up 
 in 12.319 seconds. 
 Running: 
...2009-09-30 15:00:47,490 INFO sqlalchemy.engine.base.
Engine.0x...782c PRAGMA table_info("users") 
2009-09-30 15:00:47,490 INFO sqlalchemy.engine.base.Engine.0x...782c () 

 Ran 3 tests with 0 failures and 0 errors in 0.465 seconds. 
Tearing down left over layers: 
 Tear down todo.FunctionalLayer ... not supported 

The only difference between our output to that of a newly created Grok package is in the sqlalchemy lines. Of course, the most important part of the output is the “penultimate” line, which shows the number of tests that were run and whether there were any failures or errors. A failure means that some test didn’t pass, which means that the code is not doing what it’s supposed to do and needs to be checked. An error signifies that the code crashed unexpectedly at some point, and the test couldn’t even be executed, so it’s necessary to find the error and correct it before worrying about the tests.

The test runner

The test runner program looks for modules that contain tests. The test can be of three different types: Python tests, simple doctests, and full functionality doctests. To let the test runner know, which test file includes which kind of tests, a comment similar to the following is placed at the top of the file:

Do a Python test on the app.

:unittest:

In this case, the Python unit test layer will be used to run the tests. The other value that we are going to use is “doctest” when we learn how to write doctests.

The test runner then finds all of the test modules and runs them in the corresponding layer. Although unit tests are considered very important in regular development, we may find functional tests more necessary for a Grok web application, as we will usually be testing views and forms, which require the full Zope/Grok stack to be loaded to work. That’s the reason why we find only functional doctests in the default setup.

Test layers

A test layer is a specific test setup which is used to differentiate the tests that are executed. By default, there is a test layer for each of the three types of tests handled by the test runner. It’s possible to run a test layer without running the others and also to name new test layers to be able to cluster together tests that require a specific setup.

Invoking the test runner

As shown above, running bin/test will start the test runner with the default options. It’s also possible to specify a number of options, and the most important ones are summarized below. In the following table, command-line options are shown to the left. Most options can be expressed with a short form (one dash) or a long form (two dashes). Arguments for the option in question are shown in uppercase.

-s PACKAGE, –package=PACKAGE, –dir=PACKAGE	Search the given package’s directories for tests. This can be specified more than once, to run tests in multiple parts of the source tree. For example, when refactoring interfaces, you don’t want to see the way you have broken setups for tests in other packages. You just want to run the interface tests. Packages are supplied as dotted names. For compatibility with the old test runner, forward and backward slashes in package names are converted to dots. (In the special case of packages, which are spread over multiple directories, only directories within the test search path are searched.)
-m MODULE, –module=MODULE	Specify a test-module filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which test modules are searched for tests. The regular expressions are checked against dotted module names. In an extension of Python regexp notation, a leading “!” is stripped and causes the sense of the remaining regexp to be negated (so “!bc” matches any string that does not match “bc”, and vice versa). The option can specy multiple test-module filters. Test modules matching any of the test filters are searched. If no test-module filter is specified, then all of the test modules are used.
-t TEST, –test=TEST	Specify a test filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which tests are run. In an extension of Python regexp notation, a leading “!” is stripped and causes the sense of the remaining regexp to be negated (so “!bc” matches any string that does not match “bc”, and vice versa). The option can specify multiple test filters. Tests matching any of the test filters are included. If no test filter is specified, then all of the tests are executed.
–layer=LAYER	Specify a test layer to run. The option can be given multiple times to specify more than one layer. If not specified, all of the layers are executed. It is common for the running script to provide default values for this option. Layers are specified regular expressions that are used in search mode, for dotted names of objects that define a layer. In an extension of Python regexp notation, a leading “!” is stripped and causes the sense of the remaining regexp to be negated (so “!bc” matches any string that does not match “bc”, and vice versa). The layer named ‘unit’ is reserved for unit tests, however, take note of the -unit and non-unit options.
-u, –unit	Executes only unit tests, ignoring any layer options.
-f, –non-unit	Executes tests other than unit tests.
-v, –verbose	Makes output more verbose. Increment the verbosity level.
-q, –quiet	Makes the output minimal by overriding any verbosity options.

Looking at the test code

Let’s take a look at the three default test files of a Grok project, to see what each one does.

ftesting.zcml

As we explained earlier, ftesting.zcml is a configuration file for the test runner. Its main objective is to help us set up the test instance with users, so that we can test different roles according to our needs.

<configure 
  
 i18n_domain="todo" 
 package="todo" 
 > 

 <include package="todo" /> 
 <include package="todo_plus" /> 

 <!-- Typical functional testing security setup --> 
 <securityPolicy 
 component="zope.securitypolicy.zopepolicy.ZopeSecurityPolicy" 
 /> 

 <unauthenticatedPrincipal 
 id="zope.anybody" 
 title="Unauthenticated User" 
 /> 
 <grant 
 permission="zope.View" 
 principal="zope.anybody" 
 /> 

 <principal 
 id="zope.mgr" 
 title="Manager" 
 login="mgr" 
 password="mgrpw" 
 /> 

 <role id="zope.Manager" title="Site Manager" /> 
 <grantAll role="zope.Manager" /> 
 <grant role="zope.Manager" principal="zope.mgr" /> 

As shown in the preceding code, the configuration simply includes a security policy, complete with users and roles and the packages that should be loaded by the instance, in addition to the regular Grok infrastructure. If we run any tests that require an authenticated user to work, we’ll use these special users.

The includes at the top of the file just make sure that all of the Zope Component Architecture setup needed by our application is performed prior to running the tests.

tests.py

The default test module is very simple. It defines the functional layer and registers the tests for our package:

import os.path 
import z3c.testsetup 
import todo 
from zope.app.testing.functional import ZCMLLayer 

ftesting_zcml = os.path.join( 
 os.path.dirname(todo.__file__), 'ftesting.zcml') 
FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__, 'FunctionalLayer', allow_teardown=True) 

test_suite = z3c.testsetup.register_all_tests('todo')

After the imports, the first line gets the path for the ftesting.zcml file, which then is passed to the layer definition method ZCMLLayer. The final line in the module tells the test runner to find and register all of the tests in the package.

This will be enough for our testing needs in this article, but if we needed to create another non-Grok package for our application, we would need to add a line like the last one to it, so that all of its tests are found by the test runner. This is pretty much boilerplate code, as only the package name has to be changed.