The Sentinel

As I described in Building Custom .deb Packages and Chroots, since we’re using Ubuntu we wanted to use the Ubuntu package management tools—apt and dpkg—to install our custom stuff. After I built the package, I wanted to use apt to install it, because while dpkg will identify the dependencies, it won’t install them (which is lame, but a documented gap).

To do this, we need to tell apt where to find my package, which means we have to create a repository. Don’t worry, this sounded like a big deal to me with custom ftp servers, but apt/sources.list already supports file-based repositories.

Two easy steps:

Create the repository

First you need to create a special file called Packages.gz which lists the available packages. It’s built using the utility program dpkg-scanpackages.

mark@desktop$ mkdir repository
mark@desktop$ mv mypackage-0.1.deb repository
mark@desktop$ cd repository
mark@desktop$ sudo dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Let apt know about your new repository

Edit /etc/apt/sources.list to tell it about your new file. Here’s why mine (inside my chroot) looked like when I was done:

deb http://us.archive.ubuntu.com/ubuntu/ lucid main universe multiverse
deb file:/home/mrisher/repository/ /
deb-src http://us.archive.ubuntu.com/ubuntu/ lucid main

After that, you’ll have to tell apt to refresh its cache:

mark@desktop$ sudo apt-get update

That’s it! Now you can install your local packages (assuming no namespace collision, but you’d want to be careful about that anyway) by simply typing

mark@desktop$ sudo apt-get install mypackage

I haven’t figured out how to deal with the unsigned package yet, but I’m not really sure I care.

/m

Since we’re running Ubuntu LTS 10.04 on our servers, it seemed appropriate to deploy our software using the built-in package management utilities. Overkill for now, but the added expense was small enough that it felt worthwhile (as opposed to gold-plating; the general admonition is to avoid building for scale until just before you have to).

Ubuntu, being Debian-based,uses .deb packages and two tools: apt and dpkg. And, unfortunately, almost all of the HOWTOs out there are about repackaging someone else’s source code (available in a .dsc—Debian Source Code—file). These HOWTOs will point you to a program called Checkinstall, which basically monitors the results of a make install command to determine what goes where, then retroactively builds a .deb package for next time.

This didn’t quite work for me because I didn’t have a make install in my code. So I had to assemble my package by hand. Turns out it’s not all that difficult.

It’s pretty simple: For the simplest installation, create a directory structure with a special directory, DEBIAN and your target installation directories, à la:

-+ packagename/
—-+ DEBIAN/
——–+ control
—-+ usr/
——–+ local/
————+ bin/
—————-+ myfile
…

.deb packages are described from that file DEBIAN/control. The syntax is documented in the Ubuntu man pages and there are overviews at this site and in in the forums. With that documentation, all I really had to do was determine the dependencies. Fortunately, I had been keeping track during development of the required packages so I didn’t need to use this slick trick to list all the current mod_perl modules (though I tested it and it worked like a champ).
Editing the control file is fairly straightforward.

Package: apollo-api
Version: 0.1.0
Maintainer: 'Mark Risher <mark@edgesentinel.com>'
Description: Apache/mod_perl API for handling and logging events...
 [my long description]
Architecture: all
Depends: libapache2-mod-perl2, libapache2-request-perl, ...

The final step was to test out the install, and for that I wanted to have a “clean room,” i.e. an out-of-the-box, default server setup so I could be really sure I wasn’t missing any dependencies. I considered downloading a fresh VM to use with VMWare Fusion, but ultimately decided that a chroot jail would be lighter weight and use less local disk space. For those who don’t want to click through to Wikipedia, a chroot jail is basically a directory (e.g. /home/mrisher/lucid) with a special shell. When you log into that special shell, / (i.e. root) is remapped to point to that directory, meaning that while you’re inside the shell, all files and directories are all relative to the chroot directory rather than your actual, live system. For example, /etc/ld.so.cache from within the chroot will actually show you /home/mrisher/lucid/etc/ld.so.cache. Make sense? Here’s an example:

## Look at the ld.so cache on my "real" system
desktop:/home/mark$ ls -lrt /etc/ld.so.cache
-rw-r--r-- 1 root root 56599 2010-08-24 16:14 /etc/ld.so.cache
 
## now log into the chroot and run the same command
desktop:/home/mark$ schroot -c lucid
(lucid)desktop:/home/mark$ ls -lrt /etc/ld.so.cache
-rw-r--r-- 1 root root 12964 Aug 25 01:46 /etc/ld.so.cache

Notice in how the second time, I have a much smaller cache? That’s because the chroot is loading just a small subset of the modules, since it’s a fresh install. And the location of that ld.so.cache? It’s /home/mrisher/lucid/etc/ld.so.cache. Here’s where everything is mounted; notice the chroot thinks of itself as a rootfs device instead of a physical device.

desktop:/home/mark$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             39814552   3962772  33829280  11% /
...
 
desktop:/home/mark$ schroot -c lucid
(lucid)desktop:/home/mark$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                39814552   3962772  33829280  11% /
...

Okay, enough preamble. How do you set one up? With debootstrap, a program that downloads the necessary files from an Ubuntu or Debian mirror. I actually used a wrapper called pbuilder:

desktop:/$ sudo apt-get install pbuilder debootstrap schroot
...
desktop:/$ sudo pbuilder create
...

…but I wish I had just gone directly into debootstrap like this

desktop:/$ sudo apt-get install debootstrap schroot
...
desktop:/$ sudo mkdir /home/myroots
desktop:/$ sudo debootstrap lucid /home/myroots/lucid
...

This will run for several minutes downloading all the necessary packages. You can possibly speed it up by specifying a smaller variant like “minbase,” which “only includes essential packages and apt”:

desktop:/$ sudo debootstrap --variant=minbase lucid /home/myroots/lucid

Now you need to tell schroot about this new directory of goodness. Edit /etc/schroot/schroot.conf to add something like:

[lucid]
description=my new Lucid download
users=mrisher
root-users=mrisher
directory=/home/myroots/lucid
type=directory

And now you should be able to load it. If all goes according to plan…

…will dump you into a new shell where / is mounted in the directory you just downloaded. It will mount your home directory, but it won’t inherit permissions; for example, this chroot has a new /etc/sudoers file so, by default, you won’t have sudo.

Now, from here, you can get to work installing your package to make sure there are no implicit dependencies.

Good luck!

mod_perl provides a number of server lifecycle handlers, which allow you to hook various parts of the initiation sequence to log configuration files or logging before Apache starts handling your individual requests. Unfortunately, for reasons still not clear, when I tried to run initialization in the PerlChildInitHandler, it would correctly initialize but then the values would get re-initialized back to defaults when the PerlReponseHandler fired.

The solution, and not all that satisfying, was to switch to the PerlPostConfigHandler, which did not suffer from that problem and has the added benefit of using shared memory across the multiple processes.

PerlModule Handler
PerlModule Apache2::Reload
PerlOpenLogsHandler Handler::open_logs
PerlPostConfigHandler Handler::post_config
PerlSetVar ReloadDirectories ¨/mnt/hgfs/src/api/lib¨
<Location /event>
   SetHandler modperl   
   PerlResponseHandler Handler::handler
   PerlInitHandler Apache2::Reload
</Location>