Posted in technology | 5 Comments »
Have you ever been typing and clicking merrily along on a computer when, suddenly, it shuts off, makes a small hissing noise and emits a little puff of smoke? That happened twice to my 2.5-year-old Asus N71JQ notebook.
The notebook was a great deal when I bought it. About €1100 for a 17″+ monitor, Intel Core i7 4x2GHz CPU, AMD Radeon 4xxx integrated GPU, 2GB RAM (later upgraded to 8GB), and two 500GB SATA II hard drives. The thing weighs a ton, but I’d resolved never to buy a non-portable computer again.
A year after I bought it, it did the magic smoke trick described above. I sent it for warranty repair and it came back with a new motherboard and battery after about 3 weeks Then about 8 months after that, it smoked again. Warranty service, new motherboard and power supply. That was earlier this year.
This weekend I wanted to finally get the machine set up with a super-reliable disk configuration. I had been running full-disk encryption (dmcrypt/LUKS) on it for a while, but I wanted to add mirroring of the system and /home partitions and use the remaining space on the drives as a striped RAID0 array for downloads (mostly movies).
So I set up my partitions like this:
|/dev/sda2||1 GB||encrypted, RAID1 mirrored with /dev/sdb2 for LVM2 mirroring logs|
|/dev/sda3||498 GB||encrypted, LVM2|
|/dev/sdb1||1 GB||periodic bit-level backup of /dev/sda1, just in case|
|/dev/sdb2||1 GB||encrypted, RAID1 mirrored with /dev/sdba for LVM2 mirroring logs|
I ran cryptsetup luksFormat on /dev/sda3 and /dev/sdb3 and used the same passphrase I’d been using before. Since I wasn’t looking forward to entering 4 LUKS passwords at boot time, I set up the 1GB partitions with random passphrases which I stored in keyfiles and referenced from /etc/crypttab so that they’d come online automatically once the main volumes were unlocked and mounted. I then used mdadm to create a RAID1 array, /dev/md0, with /dev/sda2 and /dev/sdab2 mirroring each other.
Then I created a volume group, vg0, and added /dev/md0, /dev/sda3 and /dev/sdb3 to it. Nice, a 997GB virtual disk!
On vg0 I created logical volumes as follows:
|/||16 GB||LVM RAID1 mirroring between drives|
|/home||128 GB||LVM RAID1 mirroring between drives|
|swap1||16 GB||normal volume on /dev/sda3|
|swap2||16 GB||normal volume on /dev/sdb3|
|/download (symlinked to ~/Downloads)||842 GB||LVM RAID0 striping between drives|
|free space (on /dev/md0)||1 GB||Free for mirror logs of the RAID-ed volumes|
Now, I wasn’t doing a full reinstall, so I did this in a stepwise fashion. Move everything to one drive, repartition the other drive, set up crypto and LVM on it, move data, reboot, then do the other drive. It took me quite a while to plow my way through all the complexities of the partitioning, cryptsetup, mdadm and LVM configuration, mirror sync took many long hours, but eventually I got there. Everything worked. When I rebooted I expected to be asked for the passphrases to unlock /dev/sda3 and /dev/sdb3, and then the startup scripts would take care of unlocking /dev/sda2 and /dev/sdb2 for me using the password files now available on the unlocked /root volume. I got all my data back in place and celebrated my success, without rebooting but with a few beers.
The next day, the Ubuntu folks pushed out a few updates, one of which required a reboot (libc6? I don’t remember). I wasn’t deep into anything at the moment, so I applied the update and rebooted the machine. As expected, I was prompted for the passphrase for /dev/sda3, which I entered and which was accepted (I wish the Xubuntu greeter provided more useful details of the process, but that’s a complaint for another day). And then, disaster.
I was not asked for the passphrase for /dev/sdb3, but instead was dumped quite unceremoniously into a shell running off the initramfs. I was able to mount /boot, but nothing else. I manually unlocked /dev/sdb3 with cryptsetup luksOpen, but I still couldn’t mount /, /home or /download. What was wrong?
As it turns out, I made a major error in my thinking of how the boot process would proceed. Since / was a RAID1 mirrored volume, I figured that once I unlocked /dev/sda3 LVM would be able to mount it and then /etc/crypttab would be available to unlock /dev/md0. Wrong, wrong, wrong. To mount a mirrored volume, LUKS requires that both the members of the mirror pair and the mirror logs are available. (vg0/root)/etc/crypttab could not be accessed until /dev/sda3 (ok), /dev/sdb3 (ok) and /dev/md0 (not ok) were unlocked. A truly lovely circular dependency problem, I’m sure you’ll agree.
Naturally, I didn’t save the 256-character passwords I generated to unlock /dev/sda2 and /dev/sdb2 using 256 bytes of base64-encoded /dev/random output anywhere other than (vg0/root)/etc/md-key, so I had absolutely no way to get everything opened and mounted. Argh! Many hours of trying things using the Ubuntu CD in rescue mode and SysRescueCD off a USB stick, plus plenty of reading man pages, HOWTOs and forum posts on LUKS, dmcrypt, etc. yielded no progress.
So, I gave up. Time to reinstall. This time, I partitioned the disks the way I wanted them, created my /, /home and /download volumes under LUKS with encryption on just a single drive, and figured I’d take another crack at mirroring some other time. I did have actual work to do.
By late today I had everything reinstalled: familiar apps, my dev environment, custom scripts and some Xfce desktop customization. Somehow I lost my Firefox sync bookmarks during this process, but I’ll survive that. Restored my /home off the hulking black Darth Vader-ish NAS box I rsync it to daily with a cronjob. I read my mail, took care of a few things, and then let myself play The Battle for Wesnoth for a couple of hours before buckling down to work on a job I’m somewhat behind on.
My troops were wiping out orcs and brigands quite handily when, suddenly…
Black screen. Hissing sound. Puff of smoke.