When building an authentication system, it is common practice not to store the actual password (e.g. “opensesame”) but instead to store an encrypted or obfuscated version which can be confirmed against the password but cannot be used elsewhere. The standard operating procedure is to employ a cryptographic hash, a one-way function that cannot be deciphered. A layman’s example of a hash function might be hash(word) = (number of vowels) + (number of consonants), so hash(opensesame) = 5 + 5 = 10. As you can see, if someone submitted a password like “letmein” or “password” or “123456,” the system could count the vowels and consonants and determine whether there was a match, but do so without actually storing “opensesame” at all. Then, if miscreants broke into the database, they would know that your password has 5 vowels and 5 consonants, but they wouldn’t know the actual password, and if they tried to log into your Gmail or WordPress or Bank of America site they’d still have quite a task ahead of them.

Unfortunately, in this case it appears that instead of a one-way hash function, Gawker used a symmetric (i.e. two-way, i.e. reversible) function called DES. DES is fine (well, not really, it’s actually been deprecated for over a decade) for storing the passwords. What this means is that, if bad guys can determine the encryption key, they can decrypt the raw password and use it to log into other sites. If there were just a single key used, then the entire user list would be vulnerable as soon as someone found it.

Current forensic evidence suggests that Gawker did not use a single key across the board, but instead used the password itself as the encryption key, which does help avoid the problem of a single key being compromised. However, because encryption is predictable and deterministic, this also means that a given password will always encrypt to the same encrypted password, meaning once someone computes “encrypt(opensesame)” they can find all the other users who used that same password. That’s how the attackers published a list of all the users whose passwords were “password” or “qwerty” (a shockingly large number): They were looking at a table that looked something like:

and once they figure out what that string decrypts to, they know everybody’s password.

The recommendation to avoid this problem is to employ “salt,” a random number that is added to each password before the encryption or decryption happens. With this in place, you’re not storing encrypt(opensesame) again and again, you’re storing encrypt(opensesame+12345) and encrypt(opensesame+90210) and encrypt(opensesame+02138). The salt can be stored in plaintext in the database because it’s not providing actual security against decryption here, it’s simply fighting against one compromised password creating an advantage for deciphering a second one. In fact, the bigger problem it helps guard against is one known as “Rainbow Tables,” which are basically pre-computed lists of encrypted passwords. Without salt, an attacker can build or buy a list of common passwords and compare the encrypted results against your database; with salt, these tables are mostly impractical.

This essay is not meant to criticize Gawker individually, but rather to use this attack to help teach others to avoid the same pitfalls.

Several weeks back I wrote about setting up a local repository for .deb packages. Well we’re growing and adding machines, so it was time to move that local repository to a centralized location.

It shouldn’t be hard, but it did have a few more steps than I hoped. The essence was around building the repository through a framework, and for that, a number of posts led me to reprepro.

According to its man page and sparse documentation, reprepro is “a tool to handle local repositories of debian packages,” which sounded exactly like what I wanted.

The steps were fairly straightforward, albeit with one gotcha that took me about an hour to puzzle through.

Build the package

The first step is an obvious prerequisite. However, one step missing from what I described in Building Custom .deb Packages and Chroots was the requirement to specify Section (e.g. Main, non-free, etc.) and Priority sections. Without this, reprepro will give the cryptic (though, in retrospect, obvious) error “No section given.” (Maybe if the ‘s’ was capitalized I wouldn’t have assumed this was based on the reprepro config file, instead of the package itself). Oh well.

Create a repository

Joseph Ruscio explains this part pretty well; in essence, you:

1. Generate the PGP key: gpg --gen-key should get you pretty much all you need here. If you want the repository to be owned by another user, you’ll have to sudo into that account and ensure that user has a ~username directory (which system accounts will not, by default)
2. Create the repository control files: At the minimum, reprepro will need a subdirectory called conf underneath wherever your repository will live. Inside that dir, create a file distributions that contains the reprepro setup. Here’s mine, to diff against the other online examples:

   Origin: http://myhostname.path.com
   Label: Edge Apt Repository
   Codename: es
   Architectures: i386 amd64
   Components: non-free
   Description: A generic description.
   SignWith: yes
   

   Details of the various settings are in the man page; I think these are the only mandatory ones.

3. Create an options file: If you want to save yourself some command-line time, you can create a conf/options file like so:

   verbose
   ask-passphrase
   basedir .
   

4. Run reprepro: The command is simple:

   reprepro includedeb [codename] [path to .deb file]

5. Make the repository available over http: The various articles linked above show how to set up a virtual host or path pointing to your new repository. For security purposes you should hide the conf and db directories from the public, and depending on how broadly you’re allowing distribution, play with your webserver’s permissions to restrict by IP, domain, or password. Basically, you just need to make sure that apt on your client machine (where you’re installing to) can browse the directory you’ve created above.
6. Edit /etc/apt/sources.list on the client machine(s): You’ll simply add a row to point to this new directory, the codename you specified, and the source you specified in the package (e.g. non-free). Here’s mine:

   deb http://myhostname.edgesentinel.com/

7. apt-get update

And that’s it!
Happy distributing.

/m

To guard against this type of spoofing, many APIs require a “Message Authentication Code” (MAC) to be provided along with the API call, allowing the server to identify that a) the message originated from whom it is claimed to, b) that the message has not been modified in transit, and optionally c) that the message has not been previously submitted (i.e. “replayed”). However, many implementers—including some of the most famous sites—have made the naive assumption that an MD5 signature of the parameters of the request—i.e. signature=md5(key + param1 + param2 ... paramn)—can provide these guarantees.

The trouble with the above formula, however, is that most hash functions, MD5 very much included, are have “second preimage vulnerabilities,” meaning that, given Hash1 = hash(M1) it is computationally possible to find a second message, M2, that yields the same hash value. In this case, given the signature, S, from a captured, legitimate request, an attacker can come up with extra text (a.k.a. “chaff”) C such that the hash function’s output even without the secret key is a known valid hash of the original request S = MD5(key + param1 + param2) = MD5(param100 + param101 + C).

Moreover, switching the sequence of the MAC generator function to signature=md5(param1 + param2 ... paramn + key) (i.e. a “secret suffix”) still leaves the vulnerability that if the attacker can find a collision in the hash function, he knows that he’ll still have a collision when he appends the secret. Thus the security of the MAC is illusory.

The solution, as codified in RFC 2104, involves padding the originating message and computing both an outer and an inner hash. From Wikipedia…

Let:

• H(·) be a cryptographic hash function
• K be a secret key padded to the right with extra zeros to the block size of the hash function
• m be the message to be authenticated
• ∥ denote concatenation
• ⊕ denote exclusive or (XOR)
• opad be the outer padding (0x5c5c5c…5c5c, one-block-long hexadecimal constant)
• ipad be the inner padding (0×363636…3636, one-block-long hexadecimal constant)

Then HMAC(K,m) is mathematically defined by HMAC(K,m) = H((K ⊕ opad) ∥ H((K ⊕ ipad) ∥ m)).

In this way, the signature of the message can be made robust against spoofing, and can even remain strong despite the known collision vulnerabilities against MD5 and SHA1. That said, anybody implementing a new system should skip over MD5 and SHA1 and go straight to HMAC-SHA256.

1. In Fusion, select Virtual Machine | Install VMware Tools... menu option. This option emulates a CDROM being inserted, but without Nautilus your guest OS won’t do anything until you…
2. Mount the CDROM by typing sudo mount /dev/cdrom /mnt (substitute /mnt if you want the mount point to be elsewhere)
3. cd /mnt (or wherever your mount point was)
4. cp VMwareTools* ~/ to copy the archive to your home directory
5. cd to move yourself to that home directory
6. umount /mnt to unmount the CDROM
7. tar zxf VMwareTools* to uncompress the archive
8. cd vmware-tools-distrib to get into the contents
9. And finally, sudo ./vmware-install.pl to run the installer

At this point the installer will walk you through the options; I just accepted all the defaults.

To turn on shared folders, go to the Virtual Machine | Shared Folders menu and configure from there. The vmware tools daemon should detect the changes and make those folders appear under /mnt/hgfs (which stands for Host Guest FileSystem).

Good luck.
/m

Check it out.

/m

The solution, fortunately, was quite simple, and came compliments of Craig Box:

1. Run ifconfig inside the guest OS (the Ubuntu image, in my case) to find the MAC address of your virtual adapter
2. Back on the Mac side, edit the bottom of /Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf (outside the “DO NOT MODIFY SECTION” tags) to create a static assignment that looks like:

   host developer-vm {
           hardware ethernet 00:0c:29:0e:5f:3f;
           fixed-address 192.168.139.100;
   }

The only note is that whatever fixed-address you assign must be within the randomly-assigned class-C subnet that VMWare is already using (i.e. the 192.168.nnn.0/24 that you’ve been getting dynamic addresses within; mine is “139″ above) ; if you need to change yours, that needs to be updated in /Library/Application Support/VMware Fusion/networking

To restart the service, you can reboot your VM. Theoretically, you can also try
sudo "/Library/Application Support/VMware Fusion/boot.sh" --restart and then trigger a DHCP Refresh within the VM, but that seemed like overkill to me…

Good luck!
/m

To do this, we need to tell apt where to find my package, which means we have to create a repository. Don’t worry, this sounded like a big deal to me with custom ftp servers, but apt/sources.list already supports file-based repositories.

Two easy steps:

Create the repository

First you need to create a special file called Packages.gz which lists the available packages. It’s built using the utility program dpkg-scanpackages.

mark@desktop$ mkdir repository
mark@desktop$ mv mypackage-0.1.deb repository
mark@desktop$ cd repository
mark@desktop$ sudo dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Let apt know about your new repository

Edit /etc/apt/sources.list to tell it about your new file. Here’s why mine (inside my chroot) looked like when I was done:

deb http://us.archive.ubuntu.com/ubuntu/ lucid main universe multiverse
deb file:/home/mrisher/repository/ /
deb-src http://us.archive.ubuntu.com/ubuntu/ lucid main

After that, you’ll have to tell apt to refresh its cache:

mark@desktop$ sudo apt-get update

That’s it! Now you can install your local packages (assuming no namespace collision, but you’d want to be careful about that anyway) by simply typing

mark@desktop$ sudo apt-get install mypackage

I haven’t figured out how to deal with the unsigned package yet, but I’m not really sure I care.

/m

Ubuntu, being Debian-based,uses .deb packages and two tools: apt and dpkg. And, unfortunately, almost all of the HOWTOs out there are about repackaging someone else’s source code (available in a .dsc—Debian Source Code—file). These HOWTOs will point you to a program called Checkinstall, which basically monitors the results of a make install command to determine what goes where, then retroactively builds a .deb package for next time.

This didn’t quite work for me because I didn’t have a make install in my code. So I had to assemble my package by hand. Turns out it’s not all that difficult.

It’s pretty simple: For the simplest installation, create a directory structure with a special directory, DEBIAN and your target installation directories, à la:

-+ packagename/
—-+ DEBIAN/
——–+ control
—-+ usr/
——–+ local/
————+ bin/
—————-+ myfile
…

.deb packages are described from that file DEBIAN/control. The syntax is documented in the Ubuntu man pages and there are overviews at this site and in in the forums. With that documentation, all I really had to do was determine the dependencies. Fortunately, I had been keeping track during development of the required packages so I didn’t need to use this slick trick to list all the current mod_perl modules (though I tested it and it worked like a champ).
Editing the control file is fairly straightforward.

Package: apollo-api
Version: 0.1.0
Maintainer: 'Mark Risher <mark@edgesentinel.com>'
Description: Apache/mod_perl API for handling and logging events...
 [my long description]
Architecture: all
Depends: libapache2-mod-perl2, libapache2-request-perl, ...

The final step was to test out the install, and for that I wanted to have a “clean room,” i.e. an out-of-the-box, default server setup so I could be really sure I wasn’t missing any dependencies. I considered downloading a fresh VM to use with VMWare Fusion, but ultimately decided that a chroot jail would be lighter weight and use less local disk space. For those who don’t want to click through to Wikipedia, a chroot jail is basically a directory (e.g. /home/mrisher/lucid) with a special shell. When you log into that special shell, / (i.e. root) is remapped to point to that directory, meaning that while you’re inside the shell, all files and directories are all relative to the chroot directory rather than your actual, live system. For example, /etc/ld.so.cache from within the chroot will actually show you /home/mrisher/lucid/etc/ld.so.cache. Make sense? Here’s an example:

## Look at the ld.so cache on my "real" system
desktop:/home/mark$ ls -lrt /etc/ld.so.cache
-rw-r--r-- 1 root root 56599 2010-08-24 16:14 /etc/ld.so.cache
 
## now log into the chroot and run the same command
desktop:/home/mark$ schroot -c lucid
(lucid)desktop:/home/mark$ ls -lrt /etc/ld.so.cache
-rw-r--r-- 1 root root 12964 Aug 25 01:46 /etc/ld.so.cache

Notice in how the second time, I have a much smaller cache? That’s because the chroot is loading just a small subset of the modules, since it’s a fresh install. And the location of that ld.so.cache? It’s /home/mrisher/lucid/etc/ld.so.cache. Here’s where everything is mounted; notice the chroot thinks of itself as a rootfs device instead of a physical device.

desktop:/home/mark$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             39814552   3962772  33829280  11% /
...
 
desktop:/home/mark$ schroot -c lucid
(lucid)desktop:/home/mark$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                39814552   3962772  33829280  11% /
...

Okay, enough preamble. How do you set one up? With debootstrap, a program that downloads the necessary files from an Ubuntu or Debian mirror. I actually used a wrapper called pbuilder:

desktop:/$ sudo apt-get install pbuilder debootstrap schroot
...
desktop:/$ sudo pbuilder create
...

…but I wish I had just gone directly into debootstrap like this

desktop:/$ sudo apt-get install debootstrap schroot
...
desktop:/$ sudo mkdir /home/myroots
desktop:/$ sudo debootstrap lucid /home/myroots/lucid
...

This will run for several minutes downloading all the necessary packages. You can possibly speed it up by specifying a smaller variant like “minbase,” which “only includes essential packages and apt”:

desktop:/$ sudo debootstrap --variant=minbase lucid /home/myroots/lucid

Now you need to tell schroot about this new directory of goodness. Edit /etc/schroot/schroot.conf to add something like:

[lucid]
description=my new Lucid download
users=mrisher
root-users=mrisher
directory=/home/myroots/lucid
type=directory

And now you should be able to load it. If all goes according to plan…

…will dump you into a new shell where / is mounted in the directory you just downloaded. It will mount your home directory, but it won’t inherit permissions; for example, this chroot has a new /etc/sudoers file so, by default, you won’t have sudo.

Now, from here, you can get to work installing your package to make sure there are no implicit dependencies.

Good luck!

The solution, and not all that satisfying, was to switch to the PerlPostConfigHandler, which did not suffer from that problem and has the added benefit of using shared memory across the multiple processes.

PerlModule Handler
PerlModule Apache2::Reload
PerlOpenLogsHandler Handler::open_logs
PerlPostConfigHandler Handler::post_config
PerlSetVar ReloadDirectories ¨/mnt/hgfs/src/api/lib¨
<Location /event>
   SetHandler modperl   
   PerlResponseHandler Handler::handler
   PerlInitHandler Apache2::Reload
</Location>