Blowing up my notebook with Xubuntu, full-disk encryption, mdadm and LVM

23 October 2012 by Mike Gogulski
Posted in technology | 5 Comments »

Have you ever been typing and clicking merrily along on a computer when, suddenly, it shuts off, makes a small hissing noise and emits a little puff of smoke? That happened twice to my 2.5-year-old Asus N71JQ notebook.

The notebook was a great deal when I bought it. About €1100 for a 17″+ monitor, Intel Core i7 4x2GHz CPU, AMD Radeon 4xxx integrated GPU, 2GB RAM (later upgraded to 8GB), and two 500GB SATA II hard drives. The thing weighs a ton, but I’d resolved never to buy a non-portable computer again.

A year after I bought it, it did the magic smoke trick described above. I sent it for warranty repair and it came back with a new motherboard and battery after about 3 weeks Then about 8 months after that, it smoked again. Warranty service, new motherboard and power supply. That was earlier this year.

This weekend I wanted to finally get the machine set up with a super-reliable disk configuration. I had been running full-disk encryption (dmcrypt/LUKS) on it for a while, but I wanted to add mirroring of the system and /home partitions and use the remaining space on the drives as a striped RAID0 array for downloads (mostly movies).

So I set up my partitions like this:

PartitionSize Usage
/dev/sda11 GB/boot
/dev/sda21 GBencrypted, RAID1 mirrored with /dev/sdb2 for LVM2 mirroring logs
/dev/sda3498 GBencrypted, LVM2
/dev/sdb11 GBperiodic bit-level backup of /dev/sda1, just in case
/dev/sdb21 GBencrypted, RAID1 mirrored with /dev/sdba for LVM2 mirroring logs
/dev/sdb3498GBencrypted, LVM2

I ran cryptsetup luksFormat on /dev/sda3 and /dev/sdb3 and used the same passphrase I’d been using before. Since I wasn’t looking forward to entering 4 LUKS passwords at boot time, I set up the 1GB partitions with random passphrases which I stored in keyfiles and referenced from /etc/crypttab so that they’d come online automatically once the main volumes were unlocked and mounted. I then used mdadm to create a RAID1 array, /dev/md0, with /dev/sda2 and /dev/sdab2 mirroring each other.

Then I created a volume group, vg0, and added /dev/md0, /dev/sda3 and /dev/sdb3 to it. Nice, a 997GB virtual disk!

On vg0 I created logical volumes as follows:

Use/mount pointSizeSetup
/16 GBLVM RAID1 mirroring between drives
/home128 GBLVM RAID1 mirroring between drives
swap116 GBnormal volume on /dev/sda3
swap216 GBnormal volume on /dev/sdb3
/download (symlinked to ~/Downloads)842 GBLVM RAID0 striping between drives
free space (on /dev/md0)1 GBFree for mirror logs of the RAID-ed volumes

Now, I wasn’t doing a full reinstall, so I did this in a stepwise fashion. Move everything to one drive, repartition the other drive, set up crypto and LVM on it, move data, reboot, then do the other drive. It took me quite a while to plow my way through all the complexities of the partitioning, cryptsetup, mdadm and LVM configuration, mirror sync took many long hours, but eventually I got there. Everything worked. When I rebooted I expected to be asked for the passphrases to unlock /dev/sda3 and /dev/sdb3, and then the startup scripts would take care of unlocking /dev/sda2 and /dev/sdb2 for me using the password files now available on the unlocked /root volume. I got all my data back in place and celebrated my success, without rebooting but with a few beers.

The next day, the Ubuntu folks pushed out a few updates, one of which required a reboot (libc6? I don’t remember). I wasn’t deep into anything at the moment, so I applied the update and rebooted the machine. As expected, I was prompted for the passphrase for /dev/sda3, which I entered and which was accepted (I wish the Xubuntu greeter provided more useful details of the process, but that’s a complaint for another day). And then, disaster.

I was not asked for the passphrase for /dev/sdb3, but instead was dumped quite unceremoniously into a shell running off the initramfs. I was able to mount /boot, but nothing else. I manually unlocked /dev/sdb3 with cryptsetup luksOpen, but I still couldn’t mount /, /home or /download. What was wrong?

As it turns out, I made a major error in my thinking of how the boot process would proceed. Since / was a RAID1 mirrored volume, I figured that once I unlocked /dev/sda3 LVM would be able to mount it and then /etc/crypttab would be available to unlock /dev/md0. Wrong, wrong, wrong. To mount a mirrored volume, LUKS requires that both the members of the mirror pair and the mirror logs are available. (vg0/root)/etc/crypttab could not be accessed until /dev/sda3 (ok), /dev/sdb3 (ok) and /dev/md0 (not ok) were unlocked. A truly lovely circular dependency problem, I’m sure you’ll agree.

Naturally, I didn’t save the 256-character passwords I generated to unlock /dev/sda2 and /dev/sdb2 using 256 bytes of base64-encoded /dev/random output anywhere other than (vg0/root)/etc/md[01]-key, so I had absolutely no way to get everything opened and mounted. Argh! Many hours of trying things using the Ubuntu CD in rescue mode and SysRescueCD off a USB stick, plus plenty of reading man pages, HOWTOs and forum posts on LUKS, dmcrypt, etc. yielded no progress.

So, I gave up. Time to reinstall. This time, I partitioned the disks the way I wanted them, created my /, /home and /download volumes under LUKS with encryption on just a single drive, and figured I’d take another crack at mirroring some other time. I did have actual work to do.

By late today I had everything reinstalled: familiar apps, my dev environment, custom scripts and some Xfce desktop customization. Somehow I lost my Firefox sync bookmarks during this process, but I’ll survive that. Restored my /home off the hulking black Darth Vader-ish NAS box I rsync it to daily with a cronjob. I read my mail, took care of a few things, and then let myself play The Battle for Wesnoth for a couple of hours before buckling down to work on a job I’m somewhat behind on.

My troops were wiping out orcs and brigands quite handily when, suddenly…

Black screen. Hissing sound. Puff of smoke.

Oh, fuck.

  1. 5 Responses to “Blowing up my notebook with Xubuntu, full-disk encryption, mdadm and LVM”

  2. By Michael. on 23 October 2012

    Moral of the story, make sure you have a decent backup -_- which it sounds like you did.

  3. By Stephanie on 23 October 2012

    Hrm…you failed to mention this when you recommended Asus to me over the summer. Sorry to hear your laptop has gone kaput. I just started backing mine up. I have even more motivation to do so now after reading your story.

  4. By Arto Bendiken on 23 October 2012

    Ouch! What is it, specifically, that goes up in smoke? The PSU?

  5. By Thomas L. Knapp on 23 October 2012

    The second time it did that, they should have just written the whole machine off and given you a new one. Certainly THIS time (if it’s still under warranty.

  6. By Mike Gogulski on 23 October 2012

    @Michael: Yeah, effectively zero data loss.

    @Stephanie: Huh. Maybe I was just talking about price/performance and didn’t take my unit’s problems into account.

    @Tom: I agree. Unfortunately, it’s not under warranty any more.

    @Arto: There is a short somewhere near where the 19V active line connects to the motherboard just behind the power connector. The solder on the live pin there was nearly completely gone. This is, I believe, due a serious design flaw. The machine has two copper heat pipes coming off the CPU. These then go to the GPU and then to the cooling fan. In the first place, sharing the same heat pipes with both processors is just stupid. Worse, the heat pipes pass rather close to the power connector’s solder joints. Given that you’ve got 19V at a max of about 6A hitting the single hot lead off the power connector, that joint’s getting hot anyway. Then, add more radiant heat coming off that heat pipe, plus any convective heat inside the chassis. For the final bonus, the power connection is all of a) mechanical, not magnetic; b) straight-in rather than 90°; and c) located in the middle of the left side of the case, where it’s going to experience plenty of random movement if you use the machine actually on your lap, which I do almost always. So, high wattage plus excess heat plus mechanical strain leads to slugglishly flowing solder, eventually causing a simple failure of the joint or, in my case, a short circuit.

    I’m going to take the motherboard to the local hackerspace now and see if I can repair this. I don’t see any blown traces (on 2 layers of an at least 4-layer PCB, hahaha), so if there are no cooked components then fixing this might be straightforward.

    comments rss Comments RSS

Sorry, comments for this entry are closed at this time.

  • Categories

  • Archives

  • Core Dogma