Starling Sysadmin

Hey, it’s the one year anniversary of the Starling Software Sysadmin blog! I’m celebrating, but I’m also working. Why? Because I can see that, if I’m going to continue on at the astounding rate of two posts per year, I really need to get one out in the next few months. So why not now?

Today we’re going to talk about doing all sorts of twiddly things that the Ubuntu (or probably Debian, too) installer usually does for you. And, for those of you who read the title of this blog and are already thinking, “It’s simple! Just use dd or Ghost!” we’ll also spend a few minutes talking about why we don’t do that.

On this occasion the installer in question is the Ubuntu alternate install CD. As I write this, you need to use this alternate installer, rather than the standard graphical install CD, to install a system that uses full-disk encryption. When you set things up this way, the boot process (and thus, by definition, the recovery process) becomes rather more complex than the standard Linux one. Let me review this configuration.

The Layout of the Components

The IBM PC boot process has many Byzantine details, but we won’t worry (too much) about all the things that Grub does and instead start with the MBR partition table. As well as partitions for whatever other OSes you have kicking around, you’ll have two for your Linux system. One is a (small) unencrypted boot partition, and the other is a (large, I hope) encrypted everything else.

The small one will be a standard ext2fs (or other filesystem readable by Grub) filesystem that anybody can read; it’s usually mounted on /boot once the system is fully up and running. It contains the kernels you might wish to boot¹ and their corresponding initrd images. One of those images will be mounted as the initial root partition when you boot, and scripts that it contains will find and set up the real root partition.

The larger MBR partition will be a LUKS encrypted partition. This, when opened with cryptsetup, will in turn be found to contain several layers of LVM (Logical Volume Manager) components, that in turn contain your actual root and swap partitions.

To be precise, the unencrypted view of the encrypted partition will be configured as a single LVM physical volume which is the sole member of a single LVM volume group. This volume group will contain two logical volumes, one for root and one for swap. Once properly configured, these logical volumes should appear automatically under /dev/mapper once cryptsetup is used to open the encrypted LUKS partition.

Creating a System Clone: Overview

So, to create a clone of an existing installed system, one simply needs to recreate the above configuration, copy all of the files to it, change some configuration files (particularly those dealing with the UUIDs of the filesystems), and make sure that grub and the initrd images are configured properly.

Those familiar with the trick of using ‘dd’ or Ghost to copy existing partitions and resizing them later may wonder why we don’t do that here. The problem with doing so is the UUIDs I mentioned above: you end up with two different filesystems that have the same UUID. This causes havoc if you have both filesystems available at the same time (such as when you’re running the system you wish to clone) because parts of the OS that identify filesystems based on UUID get very confused. Re-creating the filesystems (and other partitions) with new UUIDs means that you can, during the process or afterwards, have both disks connected to your system at the same time. It also lets you resize things as you build the new filesystems, rather than afterwards, which can make for a slightly cleaner configuration.

So, all of that said, let’s move on to the details of how we create a new copy of an installation. For all of the commands I list below, if you’re not sure exactly how they work or how to use them, I strongly recommend spending a bit of time to study the manual pages first. If you don’t understand just what you’re doing it’s easy to substitute the wrong arguments for my sample ones and muck something up. Be particularly careful that you don’t modify in any way the disk containing the original system; if you can no longer boot that, you’ll probably be in trouble.

Below I use sdx as the name of the new disk we’re setting up, and fred as the name of the new system; substitute your particular parameters as appropriate.

Creating a System Clone: Step by Step

1. Create two partitions with fdisk, one of 128-256 MB for /boot, and the other as big as you like for the LVM. Let’s call these sdxB and sdxL. (Check the fdisk manual page for the details of how to do this.)

   While the standard Ubuntu installer creates the boot partition as an extended partition (e.g., sda5), it’s perfectly safe to create it as a primary partition. Just keep in mind, you can’t create a new extended partition unless you have a primary partition free, so you may want to avoid using your last primary partition if you have no extended partitions.

2. Use cryptsetup to encrypt the larger partition, and then open that partition to expose its unencrypted view:

    cryptsetup luksFormat /dev/sdxL
    cryptsetup luksOpen /dev/sdxL fred-crypt
    dd if=/dev/zero of=/dev/mapper/fred-crypt bs=1M

   If you’ve got a large drive, you can expect that last command to take a long time, possibly several hours.

   You may want to skip it completely if you’re using an SSD. While it’s good security practice to fill a new encrypted partition with random data (the zeros become essentially random numbers once encrypted), this can interfere with an SSD’s ability to do write-balancing of its flash memory.

3. Create an LVM physical volume filling the whole of the encrypted partition, and then create an LVM volume group, assigning that physical volume to it:

    pvcreate /dev/mapper/fred-crypt
    vgcreate fred /dev/mapper/fred-crypt

4. Create LVM logical volumes for your root and swap partitions. If you’re using a rotating disk you may want to create the swap volume first, to put it closer to the outside of the disk where I/O speed will be faster. In this example I create a 2 GiB swap partition, and use the rest of the disk for the root partition. When this step is complete, you will see entries for fred-root and fred-swap appear under /dev/mapper.

    lvcreate -L 2G       -n swap fred
    lvcreate -l 100%FREE -n root fred

5. Initialize the swap partition, and the filesystems on the root and boot partitions. Feel free to use whatever filesystem you like for the root partition; I use ext3 here just as an example.

    mkswap /dev/mapper/fred-swap
    mkfs -t ext3 /dev/mapper/fred-root
    mkfs -t ext2 /dev/sdxB

6. Mount the new root partition, and copy to it all of the files from the old root partition. I use rsync for this, and note that I explicitly do not copy any other partitions mounted under the root partition. We need to avoid copying /dev, /proc, or other special filesystems. If you have other filesystems mounted under root that you do want to copy, you’ll need to copy them separately.

    mount /dev/mapper/fred-root /mnt
    rsync -raHAXx -v / /mnt

7. Now mount the new boot partition in its usual home under the new root partition, and copy all of the old boot partition files to it:

    mount /dev/sdxB /mnt/boot
    rsync -raHAXx -v /boot/ /mnt/boot/

8. Now we need to reconfigure the new system to use the new partitions and filesystems when it starts up. There are quite a few different and sometimes unintuitive functions that use these, such as the configuration for restarting from hibernate mode, so your best bet is to use grep -rl on /mnt/etc, looking for all of the places your old UUIDs were mentioned, and converting them to the new ones. These will probably include crypttab, fstab, and initramfs-tools/conf.d/resume. You can, however, probably safely ignore obvious cache files, such as those for lvm and blkid.

   The easiest way to find out your old and new UUIDs is to use the blkid program.

   Note that for crypttab, if you’re using UUIDs in your fstab, the first word can be any arbitrary name, as it’s used only for the name under /dev/mapper, which is not referenced by fstab when you’re using UUIDs.

   If you get any of this wrong and you’re coming back to this step, you need to make sure that you re-run the chroot’d update-initramfs command in step 10 below, because the configuration generated for those temporary root partitions that set up your main root partition is generated from information under etc.

9. Grub’s configuration file usually also uses UUIDs; edit /mnt/boot/grub/menu.lst to fix anything there. This may include the following lines:

    # kopt=...
    ...
    # groot=...

10. Now we need to update the initrd and auto-generated portions of the grub files on our new boot partition. Because the Ubuntu tools to do this aren’t really designed to do it on anything but the current system, we’ll need to fool them with chroot. But before we do that, we need to make sure that the device and other special files that the tools read are available in the new root:

    mount --bind /dev /mnt/dev
    chroot /mnt
    mount /proc
    update-initramfs -u -k all
    update-grub
    exit

11. Finally, we just need to finish the grub install, and we’re set:

    grub-install -root-directory=/mnt /dev/sdx

Congratulations! After this long and arduous process, you now should be able to boot the new disk and run a clone of your old system. You’ve also probably learned a lot about the Ubuntu boot process as well.