Hell-in-a-Handbasket System Rescue – Part I: PGP WDE

Some time ago in a previous post, I alluded to a series of lesson-learned posts I had regarding a particularly brutal system rescue.

I doubt anyone even remembers it, but it was so worthwhile to me that I felt obligated to post.

So here is Part I of this saga.

Hell-in-a-Handbasket System Rescue

The day started like any other typical day.  One of our VIP users brought to us his PGP Whole Disk Encryption (WDE) encrypted laptop.  It held one physical drive with two Windows NTFS formatted partitions. Both included in the whole-disk encryption.

It apparently experienced a HDD crash/failure of some sort which upon reboot caused PGP’s BootGuard pre-boot authentication wrapper to not accept the user’s PGP passphrase.  There was little doubt that the passphrase was good and known to user.  The user reported they were doing (local to system) database maintenance work when the system locked up hard—totally non-responsive.  The user hard-powered the laptop off and then back on.  When the system came back on, it appeared to end-user that BootGuard was damaged (?) as it would now not accept his known-good passphrase on the offered authentication screen.

The system contained local logs, reports, and database files of an organizationally critical nature.

The first-level technician attempted to fix the problem by using a Windows Setup disk from the Repair Console to check-disk (CHKDSK /R) to scan for and repair sector errors.  Unfortunately this seems to have caused additional errors and rendered BootGuard damaged.  A final pass at using a Windows XP Recovery console to FIXMBR (prior to decryption) left BootGuard seemingly no longer present.

It was at this stage that I got involved, apparently seen as being one of the PGP WDE mojo holders.

Off-line booting the system with my pgpwde injected PE Boot disk and executing the “pgpwde --disk # --status” command showed that drive was still reporting as PGP “instrumented” but would not boot to BootGuard.

Using the command-line pgpwde –auth --disk # --passphrase 'user passphrase' command failed.  This was unsettling as I frequently use this with great success to access a PGP WDE system “off-line” on the fly with no issues.

The user was also quite concerned at this point.

Step Two: The Tease

I attempted to use a PGP WDR boot-disk but it did not seem to allow any options to perform recovery or decryption of drive…for whatever reason, it was not recognizing drive as PGP encrypted.

At this stage it looked like the data was a total loss.

I checked in with our Enterprise level PGP support consultant who suggested that it appeared we had covered all the known bases.  The only alternative suggested was to send it to a professional data-recovery company (at considerable cost) and see if they could recover the system.  (Big Sigh) That was a non-starter despite the fact this was a critical system for this user and the production group he supports.

So I set the system aside, took a very deep breath, and hit the books.

Step Three: Hope!

Combing through the Internet and PGP Forums I seemed to find some leads that there existed an apparently undocumented command line argument to pass the pgpwde driver to attempt recovery of the PGP key/geometry from a backup sector on the encrypted drive.

It took me the better part of a day to work it out, but by Friday I felt confident enough to toss it at the system.  We had done everything else.  There was little to loose and after a week of not having his system back, the user was quite resigned that the ultimate result was total loss of the data. (BTW, he was not making backups….)

I again used my custom Win PE 3.0 pgpwde injected boot disk to boot the laptop “off-line’ then the undocumented PGP WDE “recovery” command.  (See a bit further down in the post for the technique.)

It scanned through the physical drive for the better part of a half-hour and eventually seemed to report that it had found and repaired the damaged (?) PGP WDE geometry references.

It still would not allow me to use pgpwde to authenticate to the encrypted volume.

I was mentally exhausted by this point at the end of the day and recognized I was a bit “punchy”.  I decided to sleep on it before doing anything else.

Step Four: Decryption 

On a hunch, first thing the next morning, I then again attempted to use the official PGP Recovery Boot Disk to decrypt the physical drive with the user’s known-good passphrase.

It now recognized his passphrase and began the decryption process! Hurrah!

I locked it down, ensured the A/C source was stable, and left it going.  By the end of the day on Friday, the progress meter was still at 99% encrypted (8-hours later) but the drive indicator light was churning away.

Hoping for the best, I left it secured and running over the three-day holiday weekend.

When I came back, the PGP Recovery boot disk had a message that it had successfully decrypted the physical drive.

I was pleasantly stunned and amazed.

Back from the brink.

But the story is only half-way through.  What would I find next on the unencrypted drive?  Stay tuned for Part II.

…as well as my thoughts about a root-cause analysis on what happed to start the cascaded failure.

Now on to the technical lessons….

Using the –Recovery command argument: Explanation #1 from Claus Valca

I’ve cobbled the following below together from various PGP technical and support form pages (liked at the end of the post) as well as my own field notes.

Symptoms

On rare occasions internal or external disks that are Whole Disk Encrypted may experience the following issues:

• Inability to decrypt or read the contents of a secondary or non-system disk.
• System displays "Error loading operating system_" after entering the passphrase at the WDE Login screen.
• Master Boot Record (MBR) corruption causing the system to no longer boot.
• After starting the system with the hard disk encrypted to a passphrase and an eToken, valid passphrases are not accepted.
• Using a PGP Recovery disk displays a “Internal error accessing disk, 0×80" error.

The following commands will help diagnose and decrypt the disk.

(Important Note!: These steps all assume you have “slaved” the encrypted drive to another system that has PGP WDE installed and running.  In my case I didn’t do that, instead I used my own pgpwde injected PE Boot disk although you could also use PGP’s instructions PGP WDE: Customizing the PE for PGP Whole Disk Encryption to build your one BartPE based version as well.)  

1. To begin working with the PGPWDE interface open a command prompt and change to the PGP installation directory (default directory shown) C:\Program Files\PGP Corporation\PGP desktop.

2. To list all installed hard disks in the system type: pgpwde --enum. Entering this command will give us a list of disks with numbers we will use in the next few steps.

3. Now type pgpwde --status --disk #. Substitute the WDE disk number listed in the previous step for the number # in the command if different. The output of this command will tell us whether the disk is still encrypted.

    * If the disk is not encrypted, "Disk 1 is not instrumented by bootguard" will be the output.
    * If the disk is encrypted, the output will display:
          o "Disk 1 is instrumented by Bootguard."
          o The total number of sectors.
          o A Highwater value (number of sectors encrypted).
          o Whether the current key is valid.

4. Type pgpwde --list-user --disk #. This will tell us the user information contained on the disk. This will help in multi-user environments to determine which user passphrase was used to implement WDE.

5.  Type pgpwde --recover --disk # --passphrase “xxxxxxx”     --recover will search your entire drive looking for the file information that contains the data for authenticating your disk and attempt to rebuild/restore them. This is necessary when the references that are stored become corrupted on your machine. It does not necessarily restore/repair damaged Boot Guard instrumentation.

Now you have some more choices to make.

• you could place the “repaired” drive back in the original system and use your PGP Whole Disk Encryption Recovery Disk Image boot disk to decrypt the drive, or
• try the following from either the working system you have slaved the drive to, or
• try the following from your PGP injected Windows PE boot disk in the original system.

6. Type pgpwde --decrypt --disk # --passphrase “xxxxxxx”  This will start the decryption process. To view progress, type the status command listed in step 3 and note the “highwater” number.  This number will get smaller and smaller as the number of sectors encrypted decreases.   . --decrypt expects the “recover” references to be correct, and therefore sometimes cannot locate the data in order to check the passphrase.

Clear?

If not look below.

Using the –Recovery command argument: Explanation #2 from the University of Alabama at Birmingham IT Department

Only later after I worked out the above culling though the Net and PGP forums did I discover the following PGP Guide for Campus Administrators which had a clear “how-to” use of the –Recovery command:

Hard Drive Recovery

If you have the drive slaved to a working machine with the same version of PGP Desktop try the following:

1. Open a CMD prompt.
2. Go to: c:\Program Files\PGP Corporation\PGP Desktop\
3. Run pgpwde -enum (this will list all the drives available on your machine, find the drive number for the encrypted drive, the first will be disk 0 (your boot drive) then disk 1, then disk 2 and so on)
4. Once you have your disk number, try: pgpwde disk #(one u found) --recover (so if its disk 1 it would be: pgpwde --disk 1 --recover), the pgpwde will search your disk for a backup sector, if it finds one it will restore it.
5. If it restores the sector, then do: pgpwde --disk # --decrypt --passphrase “enter within double-quotes”
6. To determine whether the drive is still instrumented (MBR Swapped) run: pgpwde –status –disk #
7. If the disk is instrumented, run: pgpwde --uninstrument --disk #

Whew!  Either one should give you a better chance and understanding on how to use and apply this powerfully useful argument.

Claus’s Lessons Learned

In reading around, it seems the preferred method of such a recovery is to “slave” the PGP damaged drive to a working system that has PGP WDE installed.  Apparently the time to decrypt may be much faster than running it from/with the PGP WDR boot disk due to it occurring under a x32 bit OS rather than the x8 or x16 bit (I’m not sure which) OS that the PGP WDR boot disk operates under.  I could be wrong on that and stand ready to be corrected by any in the know.  It seems to me that using a Windows PE boot disk with PGP WDE pgpwde support injected in it would see similar performance as if it were “slaved” to a working system.  This would still be my own preferred solution.

Regardless, if you do use the PGP Recovery boot disk to search drive and/or repair/decrypt it can take a VERY long time….days.  Seriously.

Let it run and step away.  It stayed on 99% decrypting for almost 8 hrs. when I went home.  Sometime in the intervening three days later timeframe, it had completed without observation.

In hindsight, this was probably the case for when it is searching through a PGPMBR damaged drive for recovery with the PGP WDR boot disk, prior to requesting authentication.  I had thought it was non-responsive but it probably was (very, very) slowly churning through the encrypted drive’s contents looking for the backup data file section needed for recovery.  Had I let it complete I suspect it might have then offered me a chance to authenticate and decrypt without going through all this fuss. Unfortunately I wasn’t experienced with the process at that stage and was both impatient and frustrated by the lack of “official” documentation on the PGP site. So regardless, if you are recovering/decrypting a PGP encrypted drive, you need to seriously give it LOTS of time to run/complete.

Stay tuned for Part II where I then transition to attempting to salvage the data off the system.

It’s almost as good as I hope this post has been!

Cheers and happy --recover ‘ing

Claus V.

Bonus Linkage

Supporting PGP WDE “Recovery” linkage and reference material for the curious or desperate offered kindly below to save anyone time Googling this all up on you own.  I promise it is all valuable stuff!

• PGP Guide for Campus Administrators – University of Alabama, Birmingham.
• PGP Whole Disk Encryption Command Line for Windows (PDF) – Official User’s Guide from PGP Corporation that documents many awesomely useful pgpwde command line operations and explains the terms PGP uses for its WDE solution.  It doesn’t seem to explain, however, use of the --recover command.  I’d call that a major FAIL.
• Recovering a PGP Whole-Disk-Encrypted Drive – Securism Blog.  Great and encouraging post by Jon Janego on his own frustrating attempt (eventually successful) to work out how to recover a borked PGP WDE drive.  Slightly different technique than mine documented here, but all good and painfully similar in lessons learned.
• PGP WDE: Customizing the PE for PGP Whole Disk Encryption – PGP Corporation Knowledgebase Answer.
• PGP WDE will not boot - Tried Most Solutions - HELP - PGP Whole Disk Encryption for Macintosh – PGP Forum Post.
• WDE Recovery: "Boot harddisk is not instrumented; advanced recovery is needed" - PGP Whole Disk Encryption for Windows – PGP Forum Post.
• Master boot record became deleted - PGP Desktop 9.x for Windows – PGP Forum Post.
• PGP Whole Disk Encryption Recovery Disk Image(s)– PGP Corporation Knowledgebase Answer.
• PGP Whole Disk Diagnosis and Recovery - Windows– PGP Corporation Knowledgebase Answer.
• PGP Whole Disk Encryption: Data Recovery (Mac OS X)– PGP Corporation Knowledgebase Answer.
• List Of Commands/Options for PGP (8.x) – PGP Corporation Knowledgebase Answer.
• Still seeing bootguard after decrypting whole disk encrypted hard drive – PGP Corporation Knowledgebase Answer.
• HOW TO: Uninstall PGP Products in Windows Safe Mode – PGP Corporation Knowledgebase Answer.
• PGP Whole Disk Encryption - Barely Acknowledged Intentional Bypass – Securology Blog.  Not related (except in that it is another “undocumented” pgpwde command-line argument that is powerfully useful to technicians and system admins who work in a PGP WDE deployed environment.  I’ve used it in a few situations when remotely servicing a system over-the-wires and the user needed to take off for a lunch-break.

--CV