So one of the many assignments I have been working on lately is the decommissioning of a hoard of old Dell PowerEdge servers which will be sent to pasture. New technology has been deployed and these are obsolete.
Models range from 1300’s to a 2800; and each contains a pair of SCSI drives.
Accordingly, we need to be sure that all resident data has been purged from the internal drives. (Since apparently just yanking and tossing them into a super-shredder or piercing them with a ram-driven spike isn’t an option.)
The Great Wipe Hope
Instead, I turned to wiping them using DBAN / Darik's Boot And Nuke
I had to use a combo of builds as in some cases the CD ROM drive in the server didn’t work and I didn’t feel like tearing the chassis apart to swap them (or the drive bay carriages). For that I used the last DBAN version that came in a floppy drive format. Otherwise I used the latest CD-based version.
Either way they both seemed to do a good job with a DoD Short 3-pass wipe/zero session. (Yes, yes, I know..a 3-pass now days? The powers that be require a 3-pass minimum wipe if we are passing them on.)
So while DBAN seemed to handle the SCSI drives and daughter-board (?) adapter hardware just fine, every now and then on some of them I would get a “non-fatal errors were encountered” message when the wiping was done.
Obviously I wasn’t tending them full-time so I couldn’t find the error as the logging didn’t work too well; something about writing to a full floppy and/or the CD version obviously couldn’t write to itself.
Anyway…all’s wiped is good correct?
Nay! “Constant Vigilance!”
Reality Check-in
I just remained uncomfortable releasing these wiped servers without doing an independent verification that the drives really showed all sectors as “zero-out”. Didn’t want the mess of answering questions if one turned up on E-bay unverified with data still on it. That would be a Bad Thing™.
So my plan was simple.
I’d just pop in one of my custom WinPE CD’s and run a Windows-based "sector editor” against the drives and quickly confirm we found an all-zero pattern.
Simple right?
Not really.
See what I quickly found out is that while my custom WinPE CD is a really cool tool, (big surprise) it didn’t have the needed SCSI adapter board drivers. So while I could boot the iron, I couldn’t see the drives.
That sucks and the plan is now in jeopardy.
I did some reconnoitering via Google and while I found all kinds of things regarding adding drivers to WinPE (something I already knew), trying to find a source for the specific mass-storage SCSI drives that would run under WinPE 2.0 seemed fruitless.
In Which “Plan U” is Formed…
So I regrouped.
I knew that Linux seemed to be handling the particular SCSI drives/adapters OK since DBAN was working without any issues.
Since I have been working in Linux quite comfortably, I went directly to Ubuntu for a LiveCD solution.
Sure enough, a check of the Installation Media page found that SCSI CD-ROMs were supported as was the following:
Generally, the Ubuntu installation system includes support for floppies, IDE (also known as PATA) drives, IDE floppies, parallel port IDE devices, SATA and SCSI controllers and drives, USB, and FireWire. The supported file systems include FAT, Win-32 FAT extensions (VFAT) and NTFS.
Disk interfaces that emulate the “AT” hard disk interface — often called MFM, RLL, IDE, or PATA — are supported. SATA and SCSI disk controllers from many different manufacturers are supported. See the Linux Hardware Compatibility HOWTO for more details.
Following that link I confirmed that the particular SCSI controllers I was encountering in these systems was also supported.
Great.
So what process should I use to verify a total zero-out of the drives under Ubuntu?
…and DD is summoned
I knew from my previous secure drive wiping studies that the terminal command tool “dd” could be invoked, not just to wipe a drive but also examine sector data and contents.
From Disk Wiping with dcfldd - Anti-Forensics blog, MAX shows us some techniques (using dcfldd though dd worked fine with the same commands seen below since dcfldd wasn’t on my Ubuntu build):
To view the first sector (with dd) of the first attached drive;
sudo dd if=/dev/sda count=1 bs=512 | hexdump -C
For additional sector spotting, just change the value for count= accordingly.
If the drive had been fully zero-ed out, then running the following command…
sudo dd if=/dev/sda | hexdump -C
…should result in the following output as seen on a blank virutal drive I have handy:
All zero’ed out!
I found some additional tips from Dr. Nikolai Bezroukov over on this super-detailed dd usage information page, including this gem:
I just want to make sure my drive is really zeroed out!!
dd if=/dev/sda | hexdump -C | grep [^00]
... will return output of every nonzero byte on the drive. Play around with it. Sometimes drives don't completely zero out on the first try.
And finally, Raymond McKendall has some super-useful “dd-fu” secure wiping verification tips at his Ray’s Notebook site post Erasing Disk Drives. (Note: no page-scrape offense is intended to Mr. McKendall. I want to record all this in case I need to print this post in entirety and take a single hard-copy output with me for quick reference. Full props to him as I doubt I wouldn’t have easily figured this out on my own!)
It's not a bad idea to verify that the drive has been completely overwritten. When the fill pattern is a constant, like 0x00 or 0xFF, then hexdump provides an easy means to check shredding:
-> time hexdump -C /dev/sdb
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
df8f90000
real 40m21.654s
user 9m16.822s
sys 1m47.435sHere, hexdump reports that the file contains 0xdf8f90000 bytes and that each byte has the value 0x00. To verify this byte count, let bc convert from hexadecimal to decimal:
-> echo $bytes; echo "ibase=16; DF8F90000" | bc 60011642880 60011642880As it turns out, hexdump offers formatting options that can be used to condense the previous report:
-> time (hexdump -e ' "%d " "%_Ad bytes\n" ' /dev/sdb; echo $bytes)
0 *
60011642880 bytes
60011642880
real 40m34.971s
user 30m37.536s
sys 1m56.239sThe first output line indicates that all of the bytes are zero, the second line reports the total number of bytes examined, and the third line echos the value of $bytes for easy comparison.
Alternatively couple dd with hexdump to get a record count and timer all in one. This next example also specifies an ouput format to hexdump, just to be fancy:
-> dd if=/dev/sdb | hexdump -e '"hexdump: " "%d"'
hexdump: 0*
117210240+0 records in
117210240+0 records out
60011642880 bytes (60 GB) copied, 2685.18 s, 22.3 MB/sThe "0*" verifies that all bytes are zero, and the record and byte counts show how many bytes were examined. It's easy to verify that these counts meet expectations:
-> echo $sectors sectors $(( $sectors*512)) bytes
117210240 sectors 60011642880 bytesAnd to see the elapsed time in minutes:
-> echo "scale=1; 2685/60" | bc -l
44.7Or, use grep
-> grep -v -P '\0' -c -m1 junk 0
Which was all well and good, but…
This whole process was very time-consuming. To say it nicely.
Running any of even the basic dd verification methods, though quite definitive, would require many hours…per disk. And since the hard-drives I had were in the 16-80 GB range, and I had a gazillon of them to check…this technique while thorough, was a time-killer.
…pssst, Buddy. Got a good Linux sector editor GUI?
Despite my best efforts, I couldn’t find a GUI-based sector editor pre-loaded in the Ubuntu 12.04 base distro.
So I went looking wider.
Experimenting with GNU/Linux: Five gui hex editors for ubuntu - UnixLab blog
This was a great starting point, and I spent last weekend trialing them and others on my virtual Ubuntu system.
Bless Hex Editor - Ubuntu Apps Directory. Bless worked great, was able to be downloaded directly from the Ubuntu Software Center app and was easy to use. When launched via the terminal with “sudo” (a process I had to follow with all the apps listed below), I was able to access the local hard drives sda/sdb and quickly perform a visual inspection of the sector contents.
Tweak: an efficient hex editor - Ubuntu Apps Directory. This is a terminal-based sector-editor, but does provide a GUI-like interface view. See this Ubuntu Manpage: tweak for the terminal command syntax as well as this developer’s Man page for tweak. It worked well and was also installable via the Ubuntu Software Center.
Ncurses-hexedit: Edit files/disks in hex, ASCII and EBCDIC - Ubuntu Apps Directory. This is another terminal-based sector editor in a GUI-like interface view. See this [N]Curses Hexedit manual. It worked well and was again installable via the Ubuntu Software Center.
hexedit - Ubuntu Apps Directory, likewise again could be installed via the Ubuntu Software Center. Invoke from terminal via : sudo hexedit /dev/sda
KDE Utilities - Okteta - This is very nice, but to fetch it you must run: sudo apt-get install okteta I also had issues accessing and viewing /dev/sda with it that I couldn’t overcome. So despite the nice look, it didn’t fit my bill easily.
There were a few others, but these seemed to be the biggest and easiest to use “out of the box”.
Then there was wxHexEditor…
wxHexEditor - a Free Hex Editor / Disk Editor for Huge Files or Devices on Linux, Windows and MacOSX
I really, really liked this tool. It is seriously good. Really!
Not only was it rocket-fast and easy to see/navigate around it, as an added bonus it has a helpful “infoPanel” docked in the bottom left corner to verify the drive/size you are looking at. So if I had a 16 GB drive but was only seeing 8.0 GB reporting, I’d be worried.
Getting it on Ubuntu was a particular challenge to me since it was not listed in the Ubuntu Software Center and I am a Linux noobie.
The “easiest” way I could come up with it was to go over to GetDeb.net V2 - Software for Ubuntu Linux’s wxHexEditor page.
Using that source required me to first install the “get-deb” package to then fetch wxHexEditor.
That wasn’t a problem, but for some reason it didn’t work via downloading from Firefox.
So I re-read the instructions and found that bug 476853 applied to me. Which basically says Firefox hoses the downloads. So I then had to download Chrome into my Ubuntu build to fetch things. That worked and I was eventually able to successfully get wxHexEditor running nicely in Ubuntu.
Did I mention it was compiled in Linux, Mac OS X and Windows binaries? Sweet!
Success? Let’s call it a Pyrrhic victory…
OK. So here’s the rub.
To accomplish my GUI-sector-viewing quick-verification goal on each of these servers, since I was using a LiveCD of Ubuntu I had to…
- Wipe the system with DBAN, wait to finish.
- Reboot the system with Ubuntu
- Did CD-ROM drive work? Yes? Continue to step 3.
- No? pull SCSI drive carriage out of system and place into sever that had working CD-ROM drive. Continue to step 3.
- Carefully select option to try LiveCD rather than install Ubuntu onto (hopefully) wiped drive.
- Wait for Ubuntu to load into limited server system RAM bank.
- Wait some more.
- Connect network cable to server.
- Wait for network detection.
- Download “Bless” via the Ubuntu Software Center.
- Install “Bless”
- Run Bless to then inspect /dev/sda & /dev/sdb
- OK?
- Shut down Ubuntu/eject disk.
- Open server chassis to visually verify no additional non-detected SCSI drives were present. Two were expected, confirm that only two physical drives are inside the box.
- If step 2.2 was invoked, replace drives back into original server chassis.
- Repeat
This took a frigging long time.
Successful? yes.
Efficient? hardly.
So whatcha gonna do? Build a custom Ubuntu LiveCD?
Well, I suppose I could re-master a Linux LiveCD of Ubuntu that has Bless (or wxHexEditor) installed in it. Way, way before I built Windows PE disks, I cut my teeth making custom Knoppix Live CD’s to use to off-line boot and recover files from DOA Windows NTFS partitions. That was how I first got acquainted with LiveCD formats. That was when having a DOS boot floppy disk for system servicing was still required carry.
I checked out that possibility and found a number of techniques and methods to do so:
- LiveCDCustomization - Community Ubuntu Documentation
- MakeALiveCD/DVD/BootableFlashFromHarddiskInstall - Community Ubuntu Documentation
- How to Create Your Own Customized Ubuntu Live CD - How-To Geek
- How to Create a Custom Ubuntu Live CD or USB the Easy Way - How-To Geek
- Make Your Own Live CD With Ubuntu Builder - Digital blog
- Remastering an “Ubuntu alternate” ISO with Ubuntu Customization Kit – the complete guide - Fabrizio Balliano
Here is the deal. While I was pretty confident I could successfully do it, my project timeline to wipe and get the severs returned for surplusing didn’t give me enough time to do so.
Then again, maybe I could find another Linux LiveCD distro that had a GUI-based sector-editor pre-installed and that would support the particular SCSI drive/controller hardware I had.
Not really, I tried but burning lots of CD’s to trial-n-error them didn’t appeal to me and I didn’t want to attempt using my iODD hard drive to emulate ISO’s for this task.
I might have been able to save a “persistent user profile” to a USB/floppy drive…but the USB support on these servers was spotty at best and I didn’t always have the luxury of both a working floppy drive AND a working CD ROM drive in any given server (seriously) so that really wasn’t worth messing with if I could even get the required planets to align.
Concurrently, another enterprise-wide deployment storm was brewing and the squall-line looked serious. Time was of the essence and I had to complete this project. It was bad enough that I was adding in the self-imposed secondary-verification level for this job.
I’d learned a lot and improved my Linux/Ubuntu user skill set, but I was still not satisfied this was the path to continue my investment in.
Stay tuned for Scratching at a SCSI Drive Itch - Part II
…in which the problem is elegantly solved and I learn and find even more awesomeness!
Cheers,
--Claus V.
Now for the “GUI-Gotcha’s” Postscript
Note, one “gotcha” of this GUI-editor method over the “DD” methods noted earlier is that just dragging the scroll bar down the drive doesn’t really allow you a sector-by-sector inspection of the contents. You are more likely “skimming” them and potentially missing data in your speed reading. That said, it should (possibly) be pretty obvious for a heavily used drive that large blocks of data would be present if the drive wasn’t successfully zero-ed out.
YMMV with this technique so use according to your comfort level.
No comments:
Post a Comment