A Data Recovery Story, Part III

Backstory: Part I, Part II

At this point, a cloned hard disk drive logic board utilizing factory calibrations and firmware programming rescued from my damaged drive was on its way back to my home.

I constructed a new Linux box with appropriate storage capacity, made images using ddrescue of the two failed drives in the LVM volume group, and awaited delivery.

When the time came, I carefully transferred the recovered logic board and placed it onto the failed disk drive. This was simply accomplished with a standard Torx screwdriver set. I hooked up the drive to the recovery server, crossed my fingers, and powered on.

The next few seconds would determine whether the gamble worked, or whether I'd have to resort to an expensive data recovery service.

To my great relief, the drive spun up. It registered with the BIOS, and was accessible to the recovery system!

Recovery imaging with ddrescue began immediately. With it, I then had (in principle) full access to the contents of all three physical volumes as binary image files.

Emergency imaging of a 400 GB drive with PCBSolution firmware transferred replica logic board (L) to a fresh SATA drive.

After the imaging was complete, I shut down the server and removed the repaired drive from operation while recovery operations were taking place. As it happens, it was the last time that the drive ever needed to be powered on.

Normally, given a dd disk image of a partition containing a file system, one can simply mount the image using a loopback device and recover its contents.

Recovery of a partition is simple when the file system (blue) resides on a single partition of a single drive (orange). A dd image of the partition (green) may be saved and directly mounted for file system access.

This was not possible in my scenario, as the file system of interest was residing within a LVM logical volume, itself located in a volume group comprised of dedicated LVM partitions spanning three physical volumes.

Recovery scenario. The file system of interest lies within a LVM logical volume (grey) spanning three physical volumes. ddrescue images of the drives containing the physical volume partitions exist.

Given this topology, my data recovery approach needed a bit more work. In the end, I found helpful hints from the Fedora forums, which I summarize here:

1. Create separate block-level loopback devices mapped to the ddrescue full disk image files:

#losetup -f /path/to/disk1.img
#losetup -f /path/to/disk2.img
#losetup -f /path/to/disk3.img

In my case, this created loopback devices loop0, loop1, and loop2. (You can find out the mappings via losetup -a.)

2. Scan the partition tables from the three loopback-mapped devices and create partition-level device mappings:

#kpartx -a /dev/loop0
#kpartx -a /dev/loop1
#kpartx -a /dev/loop2

This adds partition-level access at /dev/mapper.

3. Instruct LVM to re-scan for physical volumes:

#pvscan

This will detect the presence of the LVM volume group contained within the disk images.

4. Activate the volume group contained within the new physical volumes:

#vgchange -a y

After following these steps, the logical volumes were fully accessible at their usual location at /dev/mapper. Following a read-only mount of the system, I was ready to recover with rsync!

Several hours later, the data was entirely rescued. A great weight lifted from my shoulders -- and it was time to come back out of the doghouse! :)

I consider myself very lucky, and regret having found myself in this scenario. So, if there is a moral to this story, it's to have good, automated backups of your critical data. Preferably, an offsite backup too -- it's not clear that the mystery power event which destroyed my server (which was on a UPS!) wouldn't have affected other components on the same subcircuit.

I'll be describing my backup system sometime in the future, but for now, it's time to sign off.

AirPlay Fun

Those that know me know that I am not an Apple fan by nature. (I have my grudges against OS X due to my preference for Linux.)

However, since getting an iPhone in May for my graduation birthday, I've been enjoying some of the features that it offers.

In particular, for Christmas this year, I received an AirPort Express. It lets me stream music from my home computer or iPhone directly to my home stereo system. (Thank you, Remote App!)

Now that it's working, I'm very happy. It's a great implementation, and my hat's off to Apple for doing it.

I did have some problems with my wireless connectivity, though, which I thought were worth mentioning to those who may be searching for solutions.

In particular, while configuring Remote and attempting to pair it to my home iTunes library, there comes a step where the remote shows up as a 'Device' in the iTunes application. Clicking on it brings you to a validation screen where one enters a four-digit passcode shown on the iPhone running Remote. My machines were all hanging at 'Verifying remote passcode...' and failing to properly pair.

It was not a machine-specific thing; rather, due to my home wireless connection. It does not manage my home Internet connection, but provides wireless access to my LAN. I reconfigured the router to act solely as an access point, which immediately allowed Remote to pair to iTunes on my home computers.

While that solution may not work for everyone, I'm pretty glad that I found something that worked!

A Data Recovery Story, Part II

Where we last left off, I had a dead hard drive on my hands. It was part of a three drive combination which together contained a treasure trove of irreplaceable photos, documents, code, and work. Physical electrical damage had fried a protection diode on the drive's logic board and destroyed a servo control chip.

The plan: obtain a replacement logic board for the failed hard drive.

The first place I looked was eBay. There are several people out there selling logic boards or hard drives for exactly this purpose. I had no luck, however. After a couple of false leads with companies which claimed to have the drive (down to the model number and firmware revision) fell through, I was at my wits end and began soliciting quotes from data recovery firms, bracing myself for the $1,000+ numbers that trickled in.

Along the way, I found a company which specializes in hard drive logic boards: PCBSolution. Over the course of a few emails, they offered a very interesting alternative -- utilizing a physically identical controller board, and performing a firmware transfer from the damaged board to retain any/all factory calibrations, etc. used to encode the data in the disabled drive!

For $49 plus shipping (< 5% of data recovery service quotes), it was a steal to give it a shot. I shipped my board off to Canada, and within two days of receipt I received notice that the firmware transfer was successful and a replacement board was coming back to my home.

In the meantime, I constructed a replacement Linux server using new large disk drives. With approximately 3 TB of disk space accessible, I would be ready to image the drives and subsequently rescue their data. The usual rule of thumb is that if you can get the disk image, then there's some way to make Linux play nice with it. :)

Usually, the way I do this is with the trusty dd utility. For example,

#dd if=/dev/sda of=/path/to/image/file.img

would make a faithful reproduction of the contents of the disk addressed at /dev/sda into an image file named file.img.

However, in my case this trusty recipe failed. After several hours, the read would abort with an I/O error. This is likely because the disks I was imaging were likely damaged by the same power fault that took out the host system, motherboard, and ancillary components. What's galling is that most of the disk arrays were empty space. dd was giving up the read on a 750 GB disk because it could not read a single 512K sector!

It turns out that a specialized tool exists for exactly this situation: ddrescue, from the FSF. It is designed to recover as much data as possible from a (possibly failing) device and revisit pesky regions later. Example:

#ddrescue /dev/sda /path/to/image/file.img rescue_log

This creates the same net file.img from the failing /dev/sda, but keeps track of its progress in the human-readable rescue_log in the working directory. After the first pass, optional direct access / retry attempts can be made by adding the -d and -r flags, respectively.

After my primary imaging passes on the two available disks, I ran SpinRite on them to work out the problem sectors. (I had the time, and it took the better part of a week!)

Ultimately, by using ddrescue in combination with SpinRite I was able to recover ~1.5 TB of raw disk image from the first two disks of the damaged LVM set. I failed to recover less than 100 kB due to unrecoverable bad sectors, etc. resulting from physical damage to the devices.

When I next revisit this topic, the story will pick up when I received the replacement logic board in the mail. :)

A Data Recovery Story, Part I

or: Why I'm Still Married. :)

As my last post alluded, my home server died in mid-October as the result of a power supply failure. This turned out to be a much bigger recovery project than I had anticipated. In my prior experience, if a power supply unit failed, it might take out the motherboard and peripheral cards, but never hard drives. Maybe I was just lucky, but this time my luck ran out.

I  built a new system with a new power supply, motherboard, processor, and RAM. (In fact, I had to pilfer RAM from my main computer, as these modern motherboards no longer accept PC-3200...) I connected everything up and powered on. It turns out that two of the five drives in the machine failed to show up in the BIOS.

And one of those two drives was a LVM physical volume that was a critical component of our file server's entire storage area. Without all three physical volumes present and accounted for, its logical storage volume could not be created and Linux subsequently failed to boot. I knew I was in big trouble.

Inspection of the un-recognized drive made it immediately apparent as to why it wasn't working: the drive logic board's PCB had suffered damage from the PSU failure.

Damaged hard drive. Note blown diode (lower left) and motor driver controller (upper right).

A voltage-limiting protection diode had served it's purpose and fried itself. Unfortunately, it didn't do it fast enough to prevent the motor driver logic chip to fry itself, either. The drive was completely out of commission.

Dread began to set in. This storage volume contained photos and documents which were not backed up anywhere else. Lots of personal documents, archival stuff from college, etc. Should it have been backed up? Yes. The problem is that my hobbyist server slowly transitioned into a mission-critical system with a large networked hard drive... and now it was holding data captive that, if lost, would be a tremendous personal loss to myself and my wife.

Not only was the situation now out of my league, it was going to be expensive to attempt recovery. Data recovery firms I polled started around $1,000 to evaluate the drive, with no guarantee of success.

I knew that there was one last likely attempt in the bag of data recovery tricks: a logic-board swap. The trick is to find an identical make / model / firmware revision of the damaged drive and use its controller on the damaged drive. The difficulty is finding that exact match.

In my case, the drive was several years old and manufactured by Seagate. As I learned, Seagate has a wide variety of firmware versions in use which makes it extremely difficult to purchase an exact match. To some extent, drives are calibrated in the factory as well. (This is becoming increasingly common.)

Things didn't look good, but I had my task. And in the end, there was success. I'll describe how that worked out in a future post -- this one's clearly getting too long as it it!

The Joys of Home Servers

It's bound to happen eventually, but a couple of days ago I found myself unable to access my home Linux server from work. This is important to me, as I host my PhD thesis / other writing Subversion repository there, as well as using it as a general-purpose relay into my home network.

I figured that my home Internet connection was being flaky, or needed to be reset; no big deal, right?

When I got home, it turns out that the Internet link was just fine. Unfortunately, the server was emitting a very high-pitched whine from its power supply unit and was otherwise offline.

With the problem escalated, as my wife's business documents are in its file-serving backend, I rushed to buy a new power supply. After replacing it, the system could power up, but otherwise not POST.

D'oh! The motherboard and/or CPU are out of commission. New parts are on the way, but it's not a nice feeling. >: /

Fixed an odd SVN Error This Morning

I've been running my own SubVersion server for some time now. Until today, I'd never had any problems.

I was updating the working copy of my thesis (kinda important!) when my favorite Windows SVN client, TortoiseSVN, gave me the following error:


Can't find a temporary directory: Internal error

It turns out that on my SVN server, the disk hosting the / filesystem was full. Freeing up some room there fixed the problem immediately.

Others have suggested this might be the problem, but I figured it can't hurt to post solutions when things work!

A Fun Night

I saw Wicked last night at the Overture Center with Kristen.

It was nice to celebrate our anniversary at the location of our wedding ceremony and one of our earliest dates.

Mobile Blogging Fun, Revisited

Tonight I write this from the BloggerPlus application on my iPhone.

The great thing is that the latest version of the app works well for those of us who use Google Apps with their Google Account.

It turns out that prior versions were not properly authenticating to Google via their Blogger API. After a cordial email exchange with the developers, who took me up on an offer to debug and/or beta test, I was able to help the team resolve their authentication issues.

Not bad, not bad at all. Now I can be blogging on the go!

UPDATE: It seems like the app omits the ability to add a hyperlink to one's text. This seems like a pretty big omission if true!

Tonight, I'm A Typical Grad Student

Another round of insomnia tonight. Fun!

I decided to make the most of it, as I have the last few times. Instead of reading a book, or having a snack or a nightcap, what do I do?

Write an enhancement to my nonlinear χ² PDE function minimizer.

Yes, tonight, I'm a typical grad student... right?

Recovering from Nigerian Hackers...

In the last few days, my father-in-law and wife's web-based email accounts were compromised.

In each case, the attacker utilized an IP address originating from Nigeria, obtained access to the account, and proceeded to send an email to all contacts purporting to be in the UK, in a stressful situation, and needing money to avoid further stress. They then proceeded to delete all of the contacts in their contact lists.

Kristen was able to regain control of her Google account in a little over two hours -- which let us find that the attackers were forwarding her incoming mail to a phantom @ymail.com account and had changed her security questions. She was also briefly locked out of Facebook, which had shared authentication credentials with her Gmail account.

My father-in-law was only able to regain access to his Hotmail account after a little over two days of frustrating email and forum exchanges with Microsoft. He found that things were similarly infested once he regained control: changed security questions, mobile accounts registered for password resets, etc. -- fortunately, no email relaying on his part.

He's now migrating to Gmail, mainly out of disgust for his experience with Microsoft during the stressful time of account compromise.

After apologizing to everyone, and fielding many, many phone calls from people who were wondering how Kristen got into the UK in the first place, much less into trouble (!) the damage appears done, and we are recovering.

It really was a wake-up call, though -- there's a lot of personal information that can live in one's email account, much less damage to one's reputation if unauthorized people access it and spam people. Things get worse if one's authentication credentials (i.e. passwords) are shared across multiple sites.

If you're like me, you probably do share passwords, because a) passwords are hard to remember, especially cryptographically strong ones and b) people tend to be lazy. Of course, this multiplies the damage that can be done if a password is compromised.

In light of this event, I decided to migrate to a managed password solution: LastPass. More on that later, as I continue to evaluate it. Suffice to say for now that I'm very impressed, and appreciate the Gibson-esque TNO ("trust no one") security model it employs. I first heard about it on Security Now! episode 256, which introduced the concept and this particular solution to me.

All That Housework is Done

Work on the condo has finished. All the ~50 wall patches are sanded and re-painted, the carpets have been cleaned and repaired, toilets fixed, windows washed, light fixtures cleaned and dusted, walls scrubbed, bathroom and entryways re-caulked, and countertops cleaned.

Thank goodness!

I even got a chance to leave a little housewarming gift for our new tenants. (Ever since we received one from our landlord in Janesville, we've followed in his footsteps.)

Carpets and Paint and Caulk, Oh My!

Lots of work at the condo today.

I was amazed by the job that Service PLUS Carpet Repair Services did today at the condo. They fixed a long-standing oopsie of mine (stereo wiring under the carpet, vis-a-vis a razor blade cut -- bad idea!), re-stretched the carpet to fix a bad seam and cleaned up the entryway, which looked very ratty due to natural wear and tear.

The result: professionally-done carpet, in less than two hours and less than two hundred bucks. It was such a great improvement, I wish that I had called them earlier! They've got my future carpet cleaning business for life as well.

Painting is going well, with 25+ wall patches complete, the office completely patched and painted (can't tell where the work was done!), the living room painted, and the first coat on our newly-tan walls. The bathroom caulking has been stripped, due to Kristen's diligence.

More work tomorrow: ceiling repair in the second bath, finishing up the painting, and re-caulking the master bath!

Ah, the joys of being a landlord...

Checkout at the Condo

Today Kristen and I said goodbye to our first tenants at our condo. I feel very lucky to have been able to meet such nice people, and we wish them well as they move on to their brand-new home!

It was a nice afternoon, considering we were working on prepping the unit for its next occupants, who by all means are very nice as well. Kristen and I were masking off the living area in preparation for painting, while our tenant was finishing up the last of the cleaning and checkout. What made things fun was their infant daughter who "helped" us with the masking (she made an artistic rendering of a shark in blue masking tape), as well as listening to her and Kristen engage in spirited conversation.

Tomorrow will be a long day: painting and general repairs at the condo. Hooray...

Mobile Blogging Fun

In the spirit of trying to keep writing something new, more often, I've tried to find an app for my iPhone which can easily post, view, and edit things over here on my Blogger-hosted blog. You'd think that this would be something that is supported by umpteen bazillion apps, but it surprisingly is not.

So far, I've tried two: BlogWriter and Blogger+, with mixed results.

BlogWriter flat-out didn't work as advertised. It didn't work at all. Their support essentially turned me away after hearing that I had a custom domain, and instructed me to ask Apple for a refund. (Yikes, and good luck!)

Blogger+ did not work at first, but so far I've been getting pretty good responses from them. They helped me get things going on a test blog, which now does actually work successfully. Turns out one has to create a Picasa web album linked to the blog before things can be properly registered. (Odd, but OK...)

On the plus side, they're even looking into why things wouldn't work with the redirect offered by the custom domain names. So, good on them! It makes me feel a bit bad that my forum posting is prominently displayed on their app's front page...

So, I think I'll wait this one out a few days before posting a review for the app. I really think that having easy access to the posting is what will increase my posting volume!

This is a test... this is only a test...

Blogging from my iPhone here, via text messaging.

Who would have thought you could actually ever do this, much less want to? :)

In the Spirit of Migration

Another late-night post following an inability to sleep! I hope this doesn't get to be a habit...

I am currently spooling up my large photo collection to the cloud from my home-served Gallery2 system. While functional, it was time to make the change. I enjoyed the hobby aspect of running my own server, but keeping up with security patches, and lack of really good backup options finally made me decide to go with a professional service.

After consideration of two major services, Google Picasa and Yahoo's Flickr I went with the latter. Given the number of photos I have to store, the price is comparable; however, Flickr offered unlimited storage, providing  room for future expansion. It also really, really helped that there's a direct Gallery2 --> Flickr bridge module, gallery2flickr, to ease the pain of migrating thousands of annotated photos and photo albums to the service.

While it took a bit of code hacking to bring the 0.9.1 release to full operation, it's now working and well-spooling everything up to the Flickr-sphere. (Is that even a word?)

Migrated

Ok, all 162 posts successfully migrated. (Still tired!)

Turns out it was a maximum-daily-post limitation, which is ~ 50/day. However, Google lets you import/export an arbitrary number of posts. So, with the help of three helper blogs, the BlogSync tool was able to painlessly transfer over all my old blog postings.

I think this is pretty nifty, considering they date back to just after I graduated from college in 2004. At the time, they lived in a little PHP-Nuke environment; most recently, an ancient version of Drupal which I never quite got to work.

Now that I've just given up on hosting my services, having a nice shiny blog hosted here will be a nice convenience. With luck, I'll actually post more often to it, as the relative difficulty of doing so with my old Drupal interface was what hindered me in the first place!

Something new for the masses!

I’ve got a bit of insomnia tonight. Since I’m not in a good enough state to write thesis now, I figured I’d take a stab at reviving my blog posting a bit.

The first thing I tried to do was to import my old blog posts over to Blogger. I found a nifty little tool (that might have been abandoned) at  http://miian.com/en/blogsync/1.1. It got about 50-odd posts correctly, but failed to sync after that.

Perhaps that's just Google doing some anti-blog-spam measure. I'll have to try again later, I guess!