In the last installment, Hell-in-a-Handbasket System Rescue – Part I: PGP WDE we left the drama with a critical laptop system that had the scrambled PGP WDE system recovered and decrypted.
This was required as the system encountered some unknown type of hard-drive/system failure and the user had failed to maintain any backup data.
So now it was time to look at the disk.
Step One: Pick Wisely
Booting the system resulted in no system found.
This was a challenging position to be in. I had a suspect drive that had taken me the better part of three-days to get decrypted and accessible…against all odds. It would not boot. It could re-experience a physical failure.
I felt like any operations conducted on it were likely to be on borrowed time
So, having a great selection of file recovery tools at hand I reached for the ADRC – Data Recovery Tools utility package.
I booted the system with my custom VistaPE boot disk (actually for additional speed and flexibility I’m using it off a bootable USB stick) where I had stashed a copy of ADRC-DRT.
I also had attached a 250 GB USB storage disk as well to the system.
Step Two: Raw Image Capture.
Using ADRC-DRT allowed me to capture an image file (sector by sector) of the entire physical drive. It didn’t take too long to complete, probably about an hour or so which was great as it was a 80 GB drive as an .IMG format file.
The reason why I wanted a sector-based image capture this time (as opposed to a file-based method like ImageX) was so that I could then use other file-recovery tools that would get at the files without needing to have the NTFS file tables intact.
I knew they were hosed as my pgpwde injected PE Boot disk system could see the physical drive, but could not see any system or files present on it.
Once captured, I shut everything down to preserve the laptop’s HDD from getting worse, if indeed it was going bad.
Step Three: Mounting the Image.
A second benefit of getting the image in an IMG file was that I could make a copy of it and then perform any number of file recovery attempts without worry about damaging the original copy.
After considering the options at my disposal, I went ahead and mounted the copy to my primary laptop system with the ImDisk Virtual Disk Driver. I liked this one in particular as it allows me to mount such a raw physical drive image file on my system and the system will recognize it as a physical drive (rather than just as a virtual optical drive). That allows some file-recovery applications that are expecting to see a physical drive to be fooled into using and accepting the image file as “the real deal.”
Another disk-based tool that allows mounting of a disk-image file I’ve used with great success is Partition Find and Mount. Also easy to use and really full-featured.
Step Four: Extracting the Damage.
Again, having a wealth of great freeware file recovery tools is a blessing and a curse. It is a curse in that you must really have spent some time with all of them to know the benefits that each one brings to the recovery table. Some are great for fast-rapid searches for specific items. Others are good for unattended bulk recovery. Still others are good when you want to try to preserve the original directory/file structure.
Know your tools.
I attempted to use TestDisk but was unable to successfully rebuild the partition information to allow in-place recovery of the decrypted/non-booting drive. Next I decided to stop taking the time to rebuild the drive geometry and file structure.
Refocused, my first-pass goal was to just see how much data I could get recovered and saved to the attached USB drive.
I then tossed PhotoRec at it which worked flawlessly. Only one problem.
The output is into a series of folders which is a jumble of recovered files. Good for the user or when working with a USB/Flash card. Bad when working with a hard disk drive full of data.
Looking at the After Using PhotoRec tip page, I followed the lead to find a really cool free “utility” PhotoRec Sorter from builtBackwards. Download it and place it in the same location as your PhotoRec recovery folders and then run it. What it does is to re-sort the recovered files into new folders based on file extension type. Really, really useful for re-organizing data pulled off a source with PhotoRec.
However, in my case I realized this was making almost as much mess as I had to start with.
The user’s key files really need to be recovered “in-place” with their original file structure.
Again I had a number of recovery tools that could do this, and in this case I went with NTFS Undelete as it seemed to behave well with the virtually-mounted IMG file representing the imaged physical disk.
Depending on the drive/health, you might need to “Run as Administrator” to access the drive/partition or you will get an access error. In this case that wasn’t an issue.
After waiting for it to thoroughly scan through the “physical-image” drive, it was able to identify for me the disk directory structures. From there I was able to select the key user profile folders as well as the database program and source folders the user had kindly provided me a list of.
These were off-loaded to yet another USB drive for safe-keeping.
It took me the better part of a day to make the image and recover the files, but at day’s end, I had 98% of the data recovered and ready.
Whew! Somebody had a lucky angel looking out for them.
Failure Thoughts:
With everything all safe and sound, I called the user who was ecstatic that what had looked like a lost cause had been snatched from the jaws of the hell-hounds and all was well again.
I turned my attention to the drive. I performed a round of additional tests on the real physical drive with various hard-drive testing utilities (see my Pocket Hard-Drive Utilities post for a roundup of choices). None of which reported any any SMART parameter errors.
A sector-test scan, however, found quite a few really bad sections. Really, really bad (in number found).
I then tried to do a DISKPART “Clean All” wipe of the drive to zero the physical drive out.
However it failed about 1/3 of the way in and balked.
Other command-line wipe/zero tools I then tried experienced the same thing. For some reason, there were enough bad sectors that it just wouldn’t finish. I also tossed both system stress-testing LiveCD’s (Inquisitor and Phoronix Test Suite LiveCD) at the system as a whole (to rule out mainboard, memory, controllers, etc.). Everything seemed fine but when they got to the hard-drive read/write tests, they failed.
As my final test, I used my official Dell Diagnostics boot disk to final-pass the system. All system points tested clean…except the drive.
So I clearly had a bad/failing drive. I called in for a replacement and the next day it was in. Because I couldn’t securely wipe the original drive, I kept it and it will be securely destroyed (apply hammer downward firmly, repeat multiple times…).
I zeroed out this new drive, formatted it, dropped a new laptop image on it, and did the primary setup for the user. I then created a “recovered files” folder for the user and dumped the rescued files back into it.
All in all, this process took approximately a week’s time to fully complete.
Lessons Learned: Part II
The most important thing I did correct at this stage was to take the image as soon as I could.
The biggest mistake I did for the overall recovery process was not taking an image of the physical drive, even while it was still in the whole-disk encrypted state, when I had the chance.
I should have done that at the get-go. As I had the tools and ability to capture the image of the physical drive…even while in an encrypted state…I should have done so.
I could have then mounted that encrypted image file and performed the same --recover actions on this one with pgpwde as if it were a physical drive.
With hindsight I cannot stress how lucky I was.
Remember from last time that once I had “recovered” the PGP encryption geometry, I then proceeded to target it with the PGP WDR boot disk, which ran a tremendously long (days) and stressful number of read/decrypt/write actions on every sector of that drive.
If during that process additional physical drive failures occurred or compounded, I might never have been able to get to the step of being able to do the file-recovery operations.
Sure capturing a raw-image of a flaky drive is very risky, but it doing it at the very start may minimize later damage and maximize your options for recovery if the physical drive later goes south.
Other lessons learned were having a variety of tools and utilities at my disposal and the ability to selectively use the correct ones to recreate the data in a format that was most useful and meaningful. Particularly in this case where database files and elements needed to be kept in the correct folder relations to one-another. Were it just image files off a memory card, that might not be as serious an issue.
Finally, you have to be willing to commit a tremendous amount of time, take copious notes through the process, and be honest with your customer. From the beginning he knew this was quite likely not to bear much success. So any level of file recovery was accepted. That I was able to recover so much was a incredible bonus.
It’s not something I would want to do every day, but I do have renewed confidence that I and our shop will be much better prepared the next time a system tanks.
Lessons learned and so noted.
Cheers!
Claus V.
4 comments:
About to try something like this procedure on my 16GB USB key that now shows up as Raw and 0 bytes after I right clicked on it and picked 'Eject' from the drop down menu; instead of using Stop from 'Safely remove USB Mass Storage Device' from the USB icon in the Notification area as I usually do on my Vista machine.
Great site Claus, been a regular reader for a couple of years now!
Update on how things went. Starting from your Step 2 above, I captured a raw image of the USB disk key using ADRC-DRT from a normal User account in Vista. I didn't bother booting the system from a boot disk because I figured; since my Vista system wasn't recognising the USB sticks partition in the first place, then it wasn’t going to be able to write anything to it. There was the danger of course, that I could accidentally respond 'Yes' to Windows invitation to format the USB stick, but I would just have to be careful. ADRC_DRT duly created an image file of just over 4GB in size which I immediately made another copy of before doing anything else. Next up was Step 3; mounting, the copy of, the image file using ImDisk. Getting ImDisk to run on my system took a little bit of perseverance, but basically when installing it, remember to run all the steps with elevated privileges even those CLI steps from the Command Prompt. Now, having a mounted image with a drive letter assigned, we could proceed to Step 4; extracting the damage, tried the NTFS Undelete tool, but to no avail. I had thought my USB stick had been formatted as NTFS but looks like maybe I my memory had failed me and it was FAT after all. I loaded up TestDisk, and run it against the image mount. At least TestDisk looked to be doing something, so I left it to do its thing for a hour or two. When I came back, after having walked the dog, it was still ‘analysing’ but the upshot was that when it did finish it showed 20 or so lines of warning to do with the FAT structure etc. I re-ran the analysis using the option for the more ‘detailed search’ but the result was the same. The warnings I was seeing were what looked like multiple corrupted FAT32 partitions ending with a FAT16 partition which TestDisk was able to display. What the display showed was a whole list of directories with names that were just a jumble of characters and there was nothing I recognised from what had been on my USB stick, sigh! When I wnet and inspected each of these directories they were all completely empty.
What happened next. When I’d discovered the problem with my USB stick a few days previously, my first port of call had been the internet were I came across this article in the Linux Journal which delves into fixing the Boot Sector and FATs directly by editing certain hex values using Linux commands. Before even contemplating going down this route I wanted to exhaust all other avenues first. So the following evening, and carrying on from my ‘Update’ I once again loaded up the image file using ImDisk. I ran the TestDisk tool again just to make sure I was still seeing the same warnings reported previously, I was. I hit the internet again looking for other partition recover freeware tools, and downloaded Partition Find and Mount from Atola Technology. This tool found the mounted image, but when I tried the 3 different depths of scan that it provides, none of these found any partitions. At this point I went back to the output from TestDisk that I had run earlier, taking a closer look at the messages, this time. I then went back up a level to the top level menu from where I’d previously kicked off the ‘Analysis’ option. From this menu, I went into ‘Options’ but I nothing that seemed of any benefit, I then chose ‘Configure’ from the same menu and found that this would allow me to adjust the Cylinders, Heads, Sectors and Sector Size, [CHS] the tool was using while trying to analyse the image. I had a look at the setting for each of these and remembering back to what I’d read earlier on the Linux Journal these default setting didn’t look sensible. So, it was back to the internet to try and find the correct CHS values for a 16GB USB stick. I found this Boot Land forum item on CHS to LBA translations which indicated what the geometry for as 16GB USB stick might be. When I plugged the values gleaned from this post into TestDisk and re-analysed my image, it immediately identified just one partition with a name I instantly recognised! On selecting this partition I was presented with a listing of directory names and file names, some in white, some in red, that I knew had been on my USB key from before the corruption. This looked very promising! I now have the option to copy those files and directories, that are highlighted in white, to a new location. Result!!!
Thank you! Thank you! Thank you! These two posts just helped me recover a hard drive for a top level manager at my company! Using the --recover switch in PGP in conjunction with Testdisc allowed me to get everything back. I had experience with Windows 7 AIK, but learning how to inject PGP into WinPE was invaluable! Really excellent posts. I am going to be checking this blog all the time now!
Post a Comment