A Data Recovery Story, Part III

Backstory: Part I, Part II

At this point, a cloned hard disk drive logic board utilizing factory calibrations and firmware programming rescued from my damaged drive was on its way back to my home.

I constructed a new Linux box with appropriate storage capacity, made images using ddrescue of the two failed drives in the LVM volume group, and awaited delivery.

When the time came, I carefully transferred the recovered logic board and placed it onto the failed disk drive. This was simply accomplished with a standard Torx screwdriver set. I hooked up the drive to the recovery server, crossed my fingers, and powered on.

The next few seconds would determine whether the gamble worked, or whether I'd have to resort to an expensive data recovery service.

To my great relief, the drive spun up. It registered with the BIOS, and was accessible to the recovery system!

Recovery imaging with ddrescue began immediately. With it, I then had (in principle) full access to the contents of all three physical volumes as binary image files.

Emergency imaging of a 400 GB drive with PCBSolution firmware transferred replica logic board (L) to a fresh SATA drive.

After the imaging was complete, I shut down the server and removed the repaired drive from operation while recovery operations were taking place. As it happens, it was the last time that the drive ever needed to be powered on.

Normally, given a dd disk image of a partition containing a file system, one can simply mount the image using a loopback device and recover its contents.

Recovery of a partition is simple when the file system (blue) resides on a single partition of a single drive (orange). A dd image of the partition (green) may be saved and directly mounted for file system access.

This was not possible in my scenario, as the file system of interest was residing within a LVM logical volume, itself located in a volume group comprised of dedicated LVM partitions spanning three physical volumes.

Recovery scenario. The file system of interest lies within a LVM logical volume (grey) spanning three physical volumes. ddrescue images of the drives containing the physical volume partitions exist.

Given this topology, my data recovery approach needed a bit more work. In the end, I found helpful hints from the Fedora forums, which I summarize here:

1. Create separate block-level loopback devices mapped to the ddrescue full disk image files:

#losetup -f /path/to/disk1.img
#losetup -f /path/to/disk2.img
#losetup -f /path/to/disk3.img

In my case, this created loopback devices loop0, loop1, and loop2. (You can find out the mappings via losetup -a.)

2. Scan the partition tables from the three loopback-mapped devices and create partition-level device mappings:

#kpartx -a /dev/loop0
#kpartx -a /dev/loop1
#kpartx -a /dev/loop2

This adds partition-level access at /dev/mapper.

3. Instruct LVM to re-scan for physical volumes:

#pvscan

This will detect the presence of the LVM volume group contained within the disk images.

4. Activate the volume group contained within the new physical volumes:

#vgchange -a y

After following these steps, the logical volumes were fully accessible at their usual location at /dev/mapper. Following a read-only mount of the system, I was ready to recover with rsync!

Several hours later, the data was entirely rescued. A great weight lifted from my shoulders -- and it was time to come back out of the doghouse! :)

I consider myself very lucky, and regret having found myself in this scenario. So, if there is a moral to this story, it's to have good, automated backups of your critical data. Preferably, an offsite backup too -- it's not clear that the mystery power event which destroyed my server (which was on a UPS!) wouldn't have affected other components on the same subcircuit.

I'll be describing my backup system sometime in the future, but for now, it's time to sign off.