In the last installment, Hell-in-a-Handbasket System Rescue – Part I: PGP WDE we left the drama with a critical laptop system that had the scrambled PGP WDE system recovered and decrypted.
This was required as the system encountered some unknown type of hard-drive/system failure and the user had failed to maintain any backup data.
So now it was time to look at the disk.
Step One: Pick Wisely
Booting the system resulted in no system found.
This was a challenging position to be in. I had a suspect drive that had taken me the better part of three-days to get decrypted and accessible…against all odds. It would not boot. It could re-experience a physical failure.
I felt like any operations conducted on it were likely to be on borrowed time
I also had attached a 250 GB USB storage disk as well to the system.
Step Two: Raw Image Capture.
Using ADRC-DRT allowed me to capture an image file (sector by sector) of the entire physical drive. It didn’t take too long to complete, probably about an hour or so which was great as it was a 80 GB drive as an .IMG format file.
The reason why I wanted a sector-based image capture this time (as opposed to a file-based method like ImageX) was so that I could then use other file-recovery tools that would get at the files without needing to have the NTFS file tables intact.
I knew they were hosed as my pgpwde injected PE Boot disk system could see the physical drive, but could not see any system or files present on it.
Once captured, I shut everything down to preserve the laptop’s HDD from getting worse, if indeed it was going bad.
Step Three: Mounting the Image.
A second benefit of getting the image in an IMG file was that I could make a copy of it and then perform any number of file recovery attempts without worry about damaging the original copy.
After considering the options at my disposal, I went ahead and mounted the copy to my primary laptop system with the ImDisk Virtual Disk Driver. I liked this one in particular as it allows me to mount such a raw physical drive image file on my system and the system will recognize it as a physical drive (rather than just as a virtual optical drive). That allows some file-recovery applications that are expecting to see a physical drive to be fooled into using and accepting the image file as “the real deal.”
Another disk-based tool that allows mounting of a disk-image file I’ve used with great success is Partition Find and Mount. Also easy to use and really full-featured.
Step Four: Extracting the Damage.
Again, having a wealth of great freeware file recovery tools is a blessing and a curse. It is a curse in that you must really have spent some time with all of them to know the benefits that each one brings to the recovery table. Some are great for fast-rapid searches for specific items. Others are good for unattended bulk recovery. Still others are good when you want to try to preserve the original directory/file structure.
Know your tools.
I attempted to use TestDisk but was unable to successfully rebuild the partition information to allow in-place recovery of the decrypted/non-booting drive. Next I decided to stop taking the time to rebuild the drive geometry and file structure.
Refocused, my first-pass goal was to just see how much data I could get recovered and saved to the attached USB drive.
I then tossed PhotoRec at it which worked flawlessly. Only one problem.
The output is into a series of folders which is a jumble of recovered files. Good for the user or when working with a USB/Flash card. Bad when working with a hard disk drive full of data.
Looking at the After Using PhotoRec tip page, I followed the lead to find a really cool free “utility” PhotoRec Sorter from builtBackwards. Download it and place it in the same location as your PhotoRec recovery folders and then run it. What it does is to re-sort the recovered files into new folders based on file extension type. Really, really useful for re-organizing data pulled off a source with PhotoRec.
However, in my case I realized this was making almost as much mess as I had to start with.
The user’s key files really need to be recovered “in-place” with their original file structure.
Again I had a number of recovery tools that could do this, and in this case I went with NTFS Undelete as it seemed to behave well with the virtually-mounted IMG file representing the imaged physical disk.
Depending on the drive/health, you might need to “Run as Administrator” to access the drive/partition or you will get an access error. In this case that wasn’t an issue.
After waiting for it to thoroughly scan through the “physical-image” drive, it was able to identify for me the disk directory structures. From there I was able to select the key user profile folders as well as the database program and source folders the user had kindly provided me a list of.
These were off-loaded to yet another USB drive for safe-keeping.
It took me the better part of a day to make the image and recover the files, but at day’s end, I had 98% of the data recovered and ready.
Whew! Somebody had a lucky angel looking out for them.
With everything all safe and sound, I called the user who was ecstatic that what had looked like a lost cause had been snatched from the jaws of the hell-hounds and all was well again.
I turned my attention to the drive. I performed a round of additional tests on the real physical drive with various hard-drive testing utilities (see my Pocket Hard-Drive Utilities post for a roundup of choices). None of which reported any any SMART parameter errors.
A sector-test scan, however, found quite a few really bad sections. Really, really bad (in number found).
I then tried to do a DISKPART “Clean All” wipe of the drive to zero the physical drive out.
However it failed about 1/3 of the way in and balked.
Other command-line wipe/zero tools I then tried experienced the same thing. For some reason, there were enough bad sectors that it just wouldn’t finish. I also tossed both system stress-testing LiveCD’s (Inquisitor and Phoronix Test Suite LiveCD) at the system as a whole (to rule out mainboard, memory, controllers, etc.). Everything seemed fine but when they got to the hard-drive read/write tests, they failed.
As my final test, I used my official Dell Diagnostics boot disk to final-pass the system. All system points tested clean…except the drive.
So I clearly had a bad/failing drive. I called in for a replacement and the next day it was in. Because I couldn’t securely wipe the original drive, I kept it and it will be securely destroyed (apply hammer downward firmly, repeat multiple times…).
I zeroed out this new drive, formatted it, dropped a new laptop image on it, and did the primary setup for the user. I then created a “recovered files” folder for the user and dumped the rescued files back into it.
All in all, this process took approximately a week’s time to fully complete.
Lessons Learned: Part II
The most important thing I did correct at this stage was to take the image as soon as I could.
The biggest mistake I did for the overall recovery process was not taking an image of the physical drive, even while it was still in the whole-disk encrypted state, when I had the chance.
I should have done that at the get-go. As I had the tools and ability to capture the image of the physical drive…even while in an encrypted state…I should have done so.
I could have then mounted that encrypted image file and performed the same --recover actions on this one with pgpwde as if it were a physical drive.
With hindsight I cannot stress how lucky I was.
Remember from last time that once I had “recovered” the PGP encryption geometry, I then proceeded to target it with the PGP WDR boot disk, which ran a tremendously long (days) and stressful number of read/decrypt/write actions on every sector of that drive.
If during that process additional physical drive failures occurred or compounded, I might never have been able to get to the step of being able to do the file-recovery operations.
Sure capturing a raw-image of a flaky drive is very risky, but it doing it at the very start may minimize later damage and maximize your options for recovery if the physical drive later goes south.
Other lessons learned were having a variety of tools and utilities at my disposal and the ability to selectively use the correct ones to recreate the data in a format that was most useful and meaningful. Particularly in this case where database files and elements needed to be kept in the correct folder relations to one-another. Were it just image files off a memory card, that might not be as serious an issue.
Finally, you have to be willing to commit a tremendous amount of time, take copious notes through the process, and be honest with your customer. From the beginning he knew this was quite likely not to bear much success. So any level of file recovery was accepted. That I was able to recover so much was a incredible bonus.
It’s not something I would want to do every day, but I do have renewed confidence that I and our shop will be much better prepared the next time a system tanks.
Lessons learned and so noted.