Thoughts on USB drive duplication for Sysadmin work

Deployment of system images has matured over the years at the sausage factory.

In the wild-west days, we would have just a handful of hardware platforms. So when they came off the truck from the factory, we would use Clonezilla to grab an image of the system.  As time went on we would periodically apply the respective image back to each of the platforms, run all our update processes, do a new Sysprep shut it down, capture an updated image, rinse, repeat.

I later shifted image capture from Clonezilla to Microsoft’s ImageX platform and could capture multiple platforms in a single WIM and deploy it depending on the specific hardware. That really helped cut down on having “separate” image DVD’s and external HDD’s to carry around and saved a lot of space due to the way ImageX and WIM files worked.

As the organization grew (became more departmentalized and specialized) other teams gradually took over the imaging process and standardized it.

Today we no longer handle the image building and distribution work ourselves but get the latest images built and shipped to us on a master USB stick.

Hopefully we will move on from this method to PXE booting and image deployments. Won’t that be exiting!

So now the challenge has shifted from system image duplication and distribution to image device duplication and distribution.

Typically the method involves taking the (bootable) USB master stick, and capturing a single IMG file of the entire device, then writing that disk image back to all the target drives.

I’ve struggled with this method for a number of reasons:

• As the USB sticks have gotten larger, the time to capture an image of the drive has gotten longer.
• As the USB sticks have gotten larger, the time to write an image of the drive to other devices has gotten longer.
• It takes a lot of space to store these IMG files; particularly as I tend to hold on to a few of the previous versions--just in case.
• Since we rarely use the entire space on the USB drive, there is a lot of wasted space and time capturing and then re-writing back that unused space.
• As different USB sticks are procured, variances in actual space on disk can wreak havoc when your master stick turns out to be a “larger” 64 GB USB stick than the purchased 64 GB USB stick you were given.

The popular tool we were asked to used for this duty was PassMark Software’s ImageUSB.  This tool is “standalone” software, works great on capturing a single IMG file of a removable USB device, and handles writing the image back great; even to multiple devices. So if you set up a bunch of USB hubs you can quickly write the image back to them all. Well, assuming that the IMG file represents a “smaller” USB storage amount than the devices you are writing back to.

I also like Alexander Beug’s USB Image Tool which can capture both device and volume-based images of removable USB devices. It has a few more bells and whistles.

Steve Si also details how you can use RMPrepUSB to capture a more efficient image of a USB drive in his Bulk duplication of USB drive images post.

In my post All Kinds of USB Cloning Challenges… I outlined how I had used both ImageX to capture a WIM-based image of the master drive and then after some DISKPART and BOOTSECT work, restored the WIM image to the drives for cloning. That works as well and saves a ton of time and space overhead storing the files that make up the master image stick rather than taking\writing a sector-based device image of the master.

I really prefer to stick with Microsoft based solutions where I can due to the health-department standards of system cleanliness on the sausage factory floor. There is always a risk bringing in and using third-party tools and I’m not a coder so I can’t audit the code these apps run on. Can’t be too safe I suppose. So I continue to fiddle around with alternatives to the WIM based image capture/cloning method.

I’ve got a library of WIM images of these master USB images and they are pretty small. They are stored on a external USB drive. However sometimes I don’t want to go through the extra steps of connecting up the drive, authenticating past it’s encryption, and then deploying the WIM to the USB stick after re DISKPART prepping the target device.

It takes me about a minute tops to go through the IMAGEX steps to re-prep a USB drive I want to apply an image to:

And did this from the command line window.

>DISKPART

DISKPART>LIST DISK

note: this is to confirm which disk number the USB drive was showing at…in this case the USB stick was disk 1 as my system disk was disk 0. Be sure you get this part identified correctly or bad things can happen! Your system will almost certainly vary!

DISKPART>SELECT DISK 1

DISKPART>CLEAN

DISKPART>CREATE PARTITION PRIMARY

DISKPART>ACTIVE

DISKPART>ASSIGN LETTER = E

DISKPART>EXIT

>format E: /fs:ntfs /q /y

note: at this point I have a freshly formatted (smaller) USB stick that is empty. Next I need to make it “bootable” so the USB stick will work as designed for system booting/imaging after the files are restored. I used the bootsect.exe tool to do this. You should be able to find it under the Windows ADK that you probably installed to get to this point to first have snagged imagex.

> bootsect /NT60 E:

Then I have just used a working “master” USB stick I had cloned off the true USB master stick I got and do a simple XCOPY to copy the files from the master to the copy.

XCOPY D:\*.* E: /e

That seems to work pretty well and duplicate USB sticks perform as needed.

I’ve also seen a variant command thusly:

XCOPY D:\*.* E: /e /h /k

The “/e” switch makes sure that directories and subdirectories -- including empty ones -- get copied.

The “/h” switch makes sure that hidden and system files get copied.

the “/k” switch makes sure that attributes get copied also as the normal xcopy will reset read-only attributes.

More here: MS-DOS xcopy command help - Computer Hope

However, Win 7 and up also now come with RoboCopy as part of the arsenal.

So I should be able to do the DISKPART parts, then use the following command to make my clone:

At the simplest I could do the following:

Robocopy D:\ E:\ /E

But since I am really trying to make a mirrored duplicate of the “master” I could do thus: 

Robocopy D:\ E:\ /MIR

• Robocopy and a Few Examples - TechNet Wiki
• Windows command line robocopy command help and information - Computer Hope
• Use Robocopy's multi-threaded feature to quickly back up your data ... - TechRepublic
• Is it posible to use RoboCopy to mirror a whole hard drive ? - Logitech Squeezebox
• Synchronous USB Clone/Mirror - Script Center - Spiceworks

At this point it seems that the biggest choke-point in the image clone deployment process is the speed at which the system can shuffle data through the USB port(s) and onto the USB stick itself. Man those things get warm when done!

And here is the other thing.

Our new Windows 8 Surface Pro 3 tablets use a different USB stick for their image/deployment.

And they aren’t just any “removable” USB stick. Nope. They have to be seen as “fixed disk” type USB drive hardware as they contain two separate and distinct partitions. Windows can’t recognize those on “removable” type USB flash drives.

So out goes PassMark’s ImageUSB tool as it can’t handle the fixed disk type USB devices with more than one partition. Nor does USB Image tool seem to handle device duplication of these multi-partition fixed disk USB drives. At least as far as I have been able to figure out.

However, being able to see and replicate the fixed-disk USB flash drive partition structure and make it bootable using just DISKPART and BOOTSEC, AND being able to restore the files to each of the partitions using either IMAGEX WIM deployments, XCOPY, or ROBOCOPY these methods make duplication of even these “fancy-pants” USB image deployment master sticks pretty straightforward.

Now, if I can only convince the powers that be to invest in USB 3.0 drives and hardware platforms that use USB 3.0 ports, I should be able to really cut down the image capture and image/file writing times for duplication.

If you have any other suggestions or recommended additional switches for use in these scenarios with XCOPY or ROBOCOPY please drop some tips in the comments.

I’d love to refine the process and steps even more.

Cheers!

Claus Valca