How can a company or organization minimize bandwidth costs when maintaining multiple Ubuntu installations? With bandwidth becoming the currency of the new millennium, being responsible with the bandwidth you have can be a real concern. In this article by Christer Edwards, we will learn how to create, maintain and make available a local Ubuntu repository mirror, allowing you to save bandwidth and improve network efficiency with each machine you add to your network.

(For more resources on Ubuntu, see here.)

Background

Before I begin, let me give you some background into what prompted me to initially come up with these solutions.

I used to volunteer at local university user groups whenever they held “installfests”. I always enjoyed offering my support and expertise with new Linux users. I have to say, the distribution that we helped deploy the most was Ubuntu. The fact that Ubuntu’s parent company, Canonical, provided us with pressed CDs didn’t hurt! Despite having more CDs than we could handle we still regularly ran into the same problem. Bandwidth. Fetching updates and security errata was always our bottleneck, consistently slowing down our deployments.

Imagine you are in a conference room with dozens of other Linux users, each of them trying to use the internet to fetch updates. Imagine how slow and bogged down that internet connection would be. If you’ve ever been to a large tech conference you’ve probably experienced this first hand!

After a few of these “installfests” I started looking for a solution to our bandwidth issue. I knew that all of these machines were downloading the same updates and security errata, therefore duplicating work that another user had already done. It just seemed so inefficient. There had to be a better way. Eventually I came across two solutions, which I will outline below.

Apt-Mirror

The first method makes use of a tool called “apt-mirror”. It is a perl-based utility for downloading and mirroring the entire contents of a public repository. This may likely include packages that you don’t use and will not use, but anything stored in a public repository will also be stored in your mirror.

To configure apt-mirror you will need the following:

• apt-mirror package (sudo aptitude install apt-mirror)
• apache2 package (sudo aptitude install apache2)
• roughly 15G of storage per release, per architecture

Once these requirements are met you can begin configuring the apt-mirror tool. The main items that you’ll need to define are:

• storage location (base_path)
• number of download threads (nthreads)
• release(s) and architecture(s) you want

The configuration is done in the /etc/apt/mirror.list file. A default config was put in place when you installed the package, but it is generic. You’ll need to update the values as mentioned above.

Below I have included a complete configuration which will mirror Ubuntu 8.04 LTS for both 32 and 64 bit installations. It will require nearly 30G of storage space, which I will be putting on a removable drive. The drive will be mounted at /media/STORAGE/.

# apt-mirror configuration file



##

## The default configuration options (uncomment and change to override)

##

#

set base_path /media/STORAGE/

# set mirror_path $base_path/mirror

# set skel_path $base_path/skel

# set var_path $base_path/var

#

# set defaultarch <running host architecture>

set nthreads 20

#



# 8.04 "hardy" i386 mirror

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy main restricted universe 
multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-updates main restricted 
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-security main restricted 
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-backports main restricted 
universe multiverse

deb-i386 http://us.archive.ubuntu.com/ubuntu hardy-proposed main restricted 
universe multiverse



deb-i386 http://us.archive.ubuntu.com/ubuntu hardy main/debian-installer 
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-i386 http://packages.medibuntu.org/ hardy free non-free



# 8.04 "hardy" amd64 mirror

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy main restricted universe 
multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-updates main restricted 
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-security main restricted 
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-backports main restricted 
universe multiverse

deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy-proposed main restricted 
universe multiverse



deb-amd64 http://us.archive.ubuntu.com/ubuntu hardy main/debian-installer 
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-amd64 http://packages.medibuntu.org/ hardy free non-free



# Cleaning section

clean http://us.archive.ubuntu.com/

clean http://packages.medibuntu.org/

It should be noted that each of the repository lines within the file should begin with deb-i386 or deb-amd64. Formatting may have changed based on the web formatting.

Once your configuration is saved you can begin to populate your mirror by running the command:

apt-mirror

Be warned that, based on your internet connection speeds, this could take quite a long time. Hours, if not more than a day for the initial 30G to download. Each time you run the command after the initial download will be much faster, as only incremental updates are downloaded.

You may be wondering if it is possible to cancel the transfer before it is finished and begin again where it left off. Yes, I have done this countless times and I have not run into any issues.

You should now have a copy of the repository stored locally on your machine. In order for this to be available to other clients you’ll need to share the contents over http. This is why we listed Apache as a requirement above. In my example configuration above I mirrored the repository on a removable drive, and mounted it at /media/STORAGE. What I need to do now is make that address available over the web. This can be done by way of a symbolic link.

cd /var/www/
sudo ln -s /media/STORAGE/mirror/us.archive.ubuntu.com/ubuntu/ ubuntu

The commands above will tell the filesystem to follow any requests for “ubuntu” back up to the mounted external drive, where it will find your mirrored contents. If you have any problems with this linking double-check your paths (as compared to those suggested here) and make sure your link points to the ubuntu directory, which is a subdirectory of the mirror you pulled from. If you point anywhere below this point your clients will not properly be able to find the contents.

The additional task of keeping this newly downloaded mirror updated with the latest updates can be automated by way of a cron job. By activating the cron job your machine will automatically run the apt-mirror command on a regular daily basis, keeping your mirror up to date without any additional effort on your part.

To activate the automated cron job, edit the file /etc/cron.d/apt-mirror. There will be a sample entry in the file. Simply uncomment the line and (optionally) change the “4” to an hour more convenient for you. If left with its defaults it will run the apt-mirror command, updating and synchronizing your mirror, every morning at 4:00am.

The final step needed, now that you have your repository created, shared over http and configured to automatically update, is to configure your clients to use it. Ubuntu clients define where they should grab errata and security updates in the /etc/apt/sources.list file. This will usually point to the default or regional mirror. To update your clients to use your local mirror instead you’ll need to edit the /etc/apt/sources.list and comment the existing entries (you may want to revert to them later!)

Once you’ve commented the entries you can create new entries, this time pointing to your mirror. A sample entry pointing to a local mirror might look something like this:

deb http://192.168.0.10/ubuntu hardy main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-updates main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-security main restricted universe multiverse

Basically what you are doing is recreating your existing entries but replacing the archive.ubuntu.com with the IP of your local mirror. As long as the mirrored contents are made available over http on the mirror-server itself you should have no problems. If you do run into problems check your apache logs for details.

By following these simple steps you’ll have your own private (even portable!) Ubuntu repository available within your LAN, saving you future bandwidth and avoiding redundant downloads.