Knowing when your build breaks

First, let us define a continuous integration, or CI server. A CI server is an application that is used to monitor your project’s build and ensure the health of your source repository.

In a software development setup, a project includes a number of developers. If it is a big project, expect the team to be larger as well. Apart from this, a team can be physically distributed in different locations. An open source project is a good example of this. Developers in open source communities are often located in different parts of the world. All these developers commit or check their code into a central source repository. Sometimes though, people only tend to build the component which they have made changes to (especially when building the entire project takes a while!). Moreover, when that component builds successfully, they commit their changes without checking whether there are dependent components that had been affected by these changes. Now, what if it turned out that one dependent component’s build failed because of those changes? No one would know until someone from the team updated his or her local copy and ran a full build. How can everyone else proceed if they were then unable to build the project?

If the developer who discovered the broken build were a Good Samaritan, he would try to fix the build on his own (which would probably take a while). Tracking down and identifying which changes caused the build to break is a tedious job. Either that or the developer would notify his team mates about the broken build so that whoever caused the build to fail would know and would be able to fix the build. However, what if the person who caused the build to break has just left for a vacation? His teammates could either track him down and bug him while he is on vacation, or they could fix it themselves. Either of which would eat up a lot of development time!

If only they had their project source in a CI server:

• They would know that they broke some component in the project when the build failed—whether it was a compilation failure or a test failure, because a CI server ensures that the entire source code builds and all the tests pass.
• They would know that the build is broken because the CI server would notify them when it encountered the build failure.
• They would know who caused the build to break, and they can contact that person and help them to fix it.
• They would know which changes caused the build to break as this information is available in the CI server.
• They would know when there is a problem in their build and deployment infrastructure. For example, when their source repository is down or their deployment repository is not accessible. We did not see these issues in the scenario above but they do happen.

It is obvious from our definition and list above what the importance of a CI server is. However, we will see in the following sections that it’s useful for more than just that.

Setting up Continuum

Continuum is an open source continuous integration server that has support for different types of projects such as Maven 2, Maven 1, Ant, and shell projects. Some of its features include build automation, release management, source control management, and build statistics.

Archiva and Continuum used to be sub-projects of Maven. Continuum officially became a top-level project at the ASF in February 2008, just a couple of months before Archiva.

Setting up a source repository

We start off by setting up our source repository where we will be adding the example application that we have been utilizing. The source repository server that we will be using is Subversion or SVN for short. More details about Subversion can be found at http://subversion.tigris.org/. First, let’s download and install Subversion as specified from their site http://subversion.tigris.org/getting.html#binary-packages. We can then set up our local SVN repository by executing the following command:

$ svnadmin create /path/to/data/svn

Make sure that you replace /path/to from the above command with the actual directory path where you will put your SVN data. This applies to all the commands with the same convention in this article.

Before we add our example application centrepoint and its parent POM effectivemaven-parent in the SVN repository, let’s first create the appropriate directories in the source repository by executing the following commands:

$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/trunk
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/
 branches
$ svn mkdir file://localhost/path/to/data/svn/effectivemaven-parent/tags
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/trunk
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/branches
$ svn mkdir file://localhost/path/to/data/svn/centrepoint/tags

The trunk/ directory is where the latest or current development happens. To use software lingo, this is the bleeding edge. The branches/ directory contains development code of certain versions of the project for maintenance purposes. It may also contain temporary copies of the source code in trunk/ specifically for developing major features or fixing critical bugs that may possibly break the code (they eventually get merged back to trunk/ once the changes made are stable). The tags/ directory on the other hand, contains all the released or tagged versions. A tagged version should no longer be modified as that version has already been frozen.

It is a convention to structure your SVN repository as such because it makes it easier to identify or to know where to look for specific sources. Maven and Continuum both use this structure in determining the default tag URL when releasing a project.

Now, let’s import our example projects centrepoint and its parent project, effectivemaven-parent, to our SVN repository. In the sample code for this article, cd to the effectivemaven-parent first then execute the following command:

$ svn import . file://localhost/path/to/data/svn/effectivemaven-parent/
trunk

Do the same with centrepoint. cd to the centrepoint directory and execute:

$ svn import . file://localhost/path/to/data/svn/centrepoint/trunk

Before importing new projects in a source repository, make sure that there are no unnecessary files (like the Maven generated target/ directory or IDE descriptor files) present. Execute mvn clean to cleanup target/directories or mvn eclipse:clean and mvn idea:clean to remove Eclipse and IntelliJ IDEA descriptor files before adding the project in source control.

Installing prerequisites

We know that Continuum supports four different types of projects: Maven 1, Maven 2, Ant, and Shell projects. In order for Continuum to build them, of course we need to have the necessary tools installed. Not all of them are required to be installed though, it really depends on the projects you will be adding to and building in Continuum. It does not make sense to have Ant installed if all of your projects are in Maven 2.

Maven 1 can be downloaded at http://maven.apache.org/maven-1.x/start/download.html.

To install Ant, we can follow the steps specified at the following web page http://ant.apache.org/manual/index.html.

Apart from the build tools above, we will also need a mail server. If you do not have a mail server installed in your machine, you can use Apache James (http://james.apache.org). Following steps 1 to 4 in http://wiki.apache.org/james/JamesQuickstart should be enough to get you started.

Installing and configuring Continuum

Now that we have everything ready, we can proceed with the installation and setup of Continuum.

Continuum can be downloaded from http://continuum.apache.org/download.html either as a standalone bundle or as a WAR file (just like Archiva). In our case, we will be using the standalone bundle.

To use the WAR file and deploy it to an application server, just follow the steps specified in http://continuum.apache.org/docs/1.3.3/installation/installation.html.

The latest version of Continuum is 1.3.3. Just download the bundle and unpack it.

I have identified below the directories and configuration files that we need to take note of in Continuum:

• conf/continuum.xml: This contains Continuum-specific configuration such as the location of the build and release output directories, working copy directory, and the base URL for accessing the webapp. The base URL is used or emails and such, which provides a URL back to the application.
• conf/jetty.xml : This is a Jetty-specific configuration. This is where the Continuum and Redback databases and mail resources are configured.
• data/: This is the default location of the Continuum and Redback databases as seen in the jetty.xml file. This is also the default base location of the working directory, build output directory, and release output directory for projects being built in Continuum.
• logs/: This is the default location of the Jetty and Continuum log files. The location of the log files can be set in the jetty-logging.xml for Jetty and apps/continuum/WEB-INF/classes/log4j.xml for Continuum-specific logs. Once we start up Continuum, we will be seeing different types of log files created in this directory. These are composed of the Continuum rolling log file, audit logs, Jetty logs and wrapper log.

We can see in the bin/ directory that there are multiple Java wrappers for starting up Continuum. The following OS platforms are supported: linux-x86-32, linuxx86-64, windows, macosx-universal-32, solaris-sparc-32, solaris-sparc-64 and solaris-x86-32.

Before we start Continuum, we must change the port it will run on. Otherwise, we will encounter an Address already in use error during startup.

To change the port, edit the configuration below in the conf/jetty.xml file. Change the default value of the jetty.port system property from 8080 to 8082.

<!-- START SNIPPET: jetty_port -->
<Call name="addConnector">
 <Arg>
 <New class="org.mortbay.jetty.nio.SelectChannelConnector">
 <Set name="host"><SystemProperty name="jetty.host" /></Set>
 <Set name="port"><SystemProperty name="jetty.port"
 default="8082"/></Set>
 <Set name="maxIdleTime">30000</Set>
 <Set name="Acceptors">2</Set>
 <Set name="statsOn">false</Set>
 <Set name="confidentialPort">8443</Set>
 <Set name="lowResourcesConnections">5000</Set>
 <Set name="lowResourcesMaxIdleTime">5000</Set>
 </New>
 </Arg>
</Call>

Now let’s fire up Continuum. In the bin/ directory, execute ./continuum console if you want to see the logs in the console or ./continuum start to run it in the background. If you are running on Windows, you need to execute continuum.batconsole to execute it in console mode or continuum.bat install then continuum.bat start to run Continuum as a service.

I am sure you have noticed during startup that the Continuum standalone uses an embedded Jetty server similar to the one Archiva uses.

Did you know that the Archiva and Continuum standalone bundles were created using Maven plugins? These are the appassembler-mavenplugin that was used to generate the Java Service Wrappers including the wrapper configuration, and the maven-assembly-plugin, which was used to package the binaries bundle.

Once Continuum has started up successfully, we can access it in the web browser via the URL http://localhost:8082/continuum.

You can easily change Continuum’s URL either by setting it in the webapp through the General Configuration page or by manually setting it in the <baseUrl> of continuum.xml. One thing to keep in mind when changing the host or the context from the default in Jetty (or in the application server), is to make sure that you change the <baseUrl> in the configuration file as well.

Accessing the above URL gives us the Create Admin page (Yes, this is just like Archiva.)

For consistency purposes, we will use the same credentials we used in Archiva for the default System Administrator user. Fill in the required fields as follows:

• Full Name: Administrator
• Email Address: [email protected] (this is our official phony admin email address)
• Password: admin1

After creating the admin user, we will be directed to the General Configuration page, which looks like the following:

The Working Directory is where Continuum keeps the local checkouts of the projects it is building, while the Build Output Directory and Release Output Directory are where Continuum stores the respective output of the project builds and the releases done in Continuum. We have already tackled what the Base URL is for, so we will be skipping that. As for the last two fields—Number of Builds in Parallel and Enable Distributed Builds. The contents specified in the fields above will be saved in the continuum.xml file once we hit the Save button.

As a general practice, the Working Directory, Build Output Directory and Release Output Directory should be kept separate from the Continuum installation. This is to avoid accidentally deleting these directories when upgrading Continuum.

Using Continuum

Now, we will see how Continuum builds a project, and how it uses and works with Maven.

At a glance

Outlined below is a brief step-by-step explanation of how Continuum builds a project:

• Project build is triggered (either by the user explicitly triggering the build or through the schedule).
• Continuum adds the build to the build queue and determines the build definition and the build environment to be used for the build. You will understand what it is I am talking about once we get to the middle parts of this article.
• Continuum checks out a fresh copy or updates the project’s working copy (or working directory) from the source repository. A working directory is just a checkout of the project’s sources.
• Once the checkout or update is finished, Continuum builds the project using the build definition from Step 2. It actually just invokes the configured build tools such as Maven or Ant to build the project.
• When the build is finished, Continuum would collect the results of the build and stores them in the database.
• Continuum sends out notifications or messages to the project team regarding the result of the build execution—the build is broken, the build is fine, and so on.

If you later install Continuum on a server where you do not have access to the file system, you can browse the working directory through the Working Copy tab of the project in the Continuum web application.

Now enough with these concepts and let’s start exercising our fingers with some CI action.

The first thing we need to do is to check out our example application and its parent POM locally. As we did not host our SVN repository to be accessible via HTTP, we will be adding our project from the filesystem. Execute:

$ svn co file://localhost/path/to/data/svn/effectivemaven-parent/trunk
effectivemaven-parent
$ svn co file://localhost/path/to/data/svn/centrepoint/trunk centrepoint

Now we will be adding our example application to Continuum from the filesystem. In order to be able to do this, we must first enable the file:// protocol in apps/continuum/WEB-INF/classes/META-INF/plexus/application.xml. Otherwise, we will encounter a The specified resource isn’t a file or the protocol used isn’t allowed error message, if we attempt to add it without enabling it first.

By default, the file:// protocol for adding projects is disabled in Continuum due to security reasons. Only http, https and ftp are allowed.

Let’s stop Continuum fi rst. As we are running on console, we will stop it via Ctrl+C. Use ./continuum stop if Continuum is running in the background or as a service.