RoboCopy (& a few alternatives) for network file copying

Growing up as kids in Texas, children of an manager at a local oil refinery, in a town surrounded by oil derricks, pump jacks, and refinery noises and smells, it only made sense that one of the most fun games we had to play at home was King Oil. I need to see if mom and my brother know where our game set is being stored. Hopefully we still have it around!

These days, I wonder if someone should make a game called “King Bandwidth” with the goal to successfully manage the needs of the customers, the decisions made by web-based application programmers and OS updating, video feeds and training, “soft phone” communication software, and third party facility service operators who just assume that if there is a network available, they should have the right to hook into it for their own equipment provision and administration. Oh. And you need to manage all potential security threats as well.

One of the challenges we face in our environment is that some tasks for the system administration side of operations involves shuttling very large files across the network. If we do that during primary production hours we run the risk of slowing down the network for our users and the web/cloud-based applications they depend on. If we try it during off-production hours we often get slowed down running into automated nightly infrastructure processes that get priority scheduling.

What I was hoping to do was find some alternatives to allow us to “trickle” copy files around the network that would minimize bandwidth impact.

One might think that if running out of bandwidth is the problem, then just increase the bandwidth.  But as any sys/network admin knows, that gets very complicated very quick. Besides just the raw dollar cost of adding additional capacity (standard copper circuits, coax, fiber) you often have infrastructure costs for additional cabling, upgraded routers/switches, and the like. If you over-buy capacity, your wasting money and facing irate fiscal managers, if you under-plan for capacity you are looking at wasting money and having irate end users and other IT admins. Bother. It is a delicate balancing act to be sure.

So anything we can do to live within the existing parameters, but be more efficient is a Good Thing™ .

What would be great would be to find a file-copy/transfer tool that would support some argument options that respected bandwidth so we could “slow-copy” where needed or “fast-copy” when bandwidth wasn’t an issue.

The very first tool that came to mind was Microsoft’s (Robust File Copy) tool robocopy.exe.  What made this tool particularly nice is that is is Windows-native and is on almost all our systems already.

Robocopy - TechNet Windows Server & Robocopy - Wikipedia

Aside from all the powerful and cool features it brings, the one that really interested me was the following argument: /ipg:n - specifies the inter-packet gap to free bandwidth on slow lines.

Robocopy - SS64.com - More details on the cli usage and some examples.

One potential “gotcha” with Robocopy is that it doesn’t handle open files. From the Wikipedia article:

Robocopy will not copy open files. Any process may open files for exclusive read access by withholding the FILE_SHARE_READ^[4] flag during opening. Even robocopy's Backup mode will not touch those files. (Backup mode instead runs Robocopy as a "Backup Operator". This allows Robocopy to override permissions settings (specifically, NTFS ACLs).^{[5] [6]})

The Windows Volume Shadow Copy service is used for such situations, but Robocopy does not use it. Therefore Robocopy is not useful for backing up live operating system volumes. However, one can use a separate utility, such as DiskShadow.exe^[7] (included with Windows Server 2008), to create a shadow copy of a given volume, which Robocopy can then be directed to back up.

Mmkay?

Updated: the exceptionally sharp TinyApps bloggist reminds us there are some great apps to help work around the locked-file issue with RoboCopy. For a full and amazing rundown of applications that can help and solution pop over right now (or at least bookmark) the TinyApps post Copy in-use files from the command line. Great tips and resources as always!

Other points to be aware of, Windows XP doesn’t come with it native, you need to get it from the Windows Server 2003 Resource Kit Tools then add it onto an XP system.

Beginning with Windows Vista through Windows 7 and 8, it was included as part of the base Windows OS package. That’s nicer.

All that to say that depending on where you got your robocopy.exe file, it may be one of several versions…each with their own slight idiosyncrasies;

• Copy files faster in Windows 7 with robocopy - FAQforge
• Robocopy Appears to be Broken in Windows 8 - Microsoft Community

So robocopy.exe may just be the perfect tool our sysadmins need to use proactively when copying files across our networks to play it safe. Only there is one small hurdle to overcome.

robocopy.exe is a CLI tool.  Not a nice, sweet GUI that many (but definitely not all) of todays IT folks are used to. In my mind, that is a major plus, but for some, eyes glaze over quickly…

Fortunately there is a solution for that:

Utility Spotlight: Robocopy GUI - TechNet magazine post from November 2006.  Robocopy GUI is an unsupported GUI wrapper for robocopy.exe

And under the “Monitoring Options” is the “IPG” field to manage copy actions over the network where bandwidth concerns exist:

How great is that?!

The Robocopy GUI post notes that there is now a newer tool “RichCopy” that offers improvements over Robocopy GUI:

Free Utility: RichCopy, an Advanced Alternative to RoboCopy - TechNet Magazine post from April 2009.

A careful look in the overwhelming number of options it provides shows a “Trickle” feature under the “Mode” area.

Unfortunately for us, while RichCopy may be a great tool in other applications, for this particular case, application documentation (and subsequent forum crawls) report the “trickle” feature is not supported.

More details on RichCopy here: How do I ... use RichCopy for high-powered file copy and transfers? - TechRepublic

So that brings us back to robocopy.exe and/or the RoboCopy GUI wrapper and the “/IPG:N” option to control bandwidth during the copy process.

From what I have read in the forums, there is something to a black-art of trying to best calculate the impact of the correct “N” value you are looking for.

Fortunately, the awesome crew at ZEDA.nl has some great tips and tools to help dial in your range finding.

ZEDA Windows Tip: Copy files on slow links - ZEDA.nl

The delay is calculated in the formula:
[filesize] / 64KB * IPG
A 300MB file and an IPG value of 500 means a delay of:
300MB / 64KB * 500ms = 307200KB / 64KB * 0,5sec = 2400 sec = 40 min.

Example

I copy a 300MB file over a 1MBit/sec line. If all bandwidth is available this will take 40 minutes; 1Mbit/sec = 1/8MB/Sec = 300MB/2400sec = 300MB/40min

Robocopy IPG Calculator - ZEDA Tools - ZEDA.nl - This tool helps automate the estimation of how long it will take to copy a particular file using Robocopy /IPG:N when provided the filesize, the current transfer rate across the network, and the IPG rate you provide. Pretty handy and cool.

Need more examples on how to use Robocopy? Then Bob’s your Uncle! (or rather PramodK who wrote and is actively maintaining it:

Robocopy and a Few Examples - TechNet Wiki

Not satisfied with either the robocopy.exe CLI or RoboCopy GUI tool?  Want some more options or maybe a better way to save you own advanced robocopy commands for reuse?

Here are some more RoboCopy GUI tools:

Better Robocopy GUI - CodePlex - “Provide intuitive GUI for editing optional parameters of Robocopy command line. It was designed for computer literates who like Robocopy but don't use it very often, and need to review legacy commands from time to time. This program is alternative to Microsoft's Robocopy GUI.”

WinRoboCopy - UpWay2Late.com Software - Though I can’t see where the /IPG:N option can be selected and called…you can however manually add it to your cli string.

Getting off the RoboCopy trail…

Up to this point I was specifically looking at RoboCopy itself, because being a “native” Microsoft application carries some additional benefits inside our organization.

However it that isn’t a concern, there are some alternative, third-party copy applications that may have some network bandwidth conservation options as well.

Ultracopier/SuperCopier - freeware - (Teracopy, SuperCopier like), replacement for files copy dialogs. It has an option under “Default options” to set a speed limit. More overview details in this AddictiveTips post: UltraCopier - Limit Speed & Pause / Resume File Copy Operation. Get the bits here: Download. And for more options, Download all version of ultracopier including a portable version.

Advanced LAN Pump - SoftSolo - ($$) - has controls for dealing with network bandwidth usage. Previous versions were freeware. If you want the last free version (v2.32) then you can grab it from The Portable Freeware Collection - Advanced LAN Pump.

I really started thinking about a blog post on this again after TinyApps.org bloggist posted some nice tiny apps for copy actions:

• "Access is denied" when attempting to copy green files - TinyApps.org blog
• FastCopy and Ycopy - TinyApps.org blog

So I reached out and TinyApps bloggist kindly offered the following additional gem of an application that I hadn’t yet found in a forum:

backup - How do you limit the bandwidth for a file copy? - Server Fault - comment left by user “levitation”

I use KillCopy for this purpose. I find it very convenient.
It has separate speed and other settings profiles for local and network copy.
It is also able to resume when a transfer of a large file (or files) is interrupted in case of bad connection, or even when the computer hosting the program crashes.

KillCopy - freeware - tiny GUI app that is skinnable and has advanced LAN speed control options

Advances settings:

• Speed limit: You can limit maximum copy transfer speed in Kbps units or change what part of current traffic used by KillCopy.

LkN -where N - number 0..10 - speed limit in kbps (see following table)
LcN -where N - number 0..10 - speed limit in bps (see following table)
LpN -where N - number 0..10 - speed limit in % (see following table)

N	kbps	%
0	16	10
1	32	20
2	128	30
3	256	40
4	512	50
5	1024	60
6	2048	70
7	4096	80
8	8192	90
9	16384	100
10	No limit	No limit

Then again, if you just don’t care about sucking all the bandwidth from your site…

…and bringing down the wrath of the end-users and management, then these alternative file copy tools may also be worth looking into. Some have been previously covered here at GSD:

• TeraCopy - (free for personal use) – Very nice, fast and tiny.
• Copy Handler - (freeware) – Dependable and fast file-copy activity. Updated periodically.
• Roadkil.Net's Unstoppable Copier - (freeware) – Special-use tool to get copy files that are “damaged" like off optical media or that give other programs copy-errors.
• FastCopy - (freeware) - New to me that some claim is the fastest file-copier tool out-there today.

Finally, since we are way off topic now,

Windows 7 native file-copy process is a big leap forward over the XP dialog windows.

And Windows 8 takes it even further.

• Windows 8 Explorer: improved copy, delete, and conflict resolution - ExtremeTech
• Improving our file management basics: copy, move, rename, and delete - Building Windows 8 - MSDN Blog
• Acting on file management feedback - Building Windows 8 - MSDN Blog

Which is one feature on Lavie’s Windows 8 laptop I am envious of that my Windows 7 laptop doesn’t’ have.

Cheers and again a special thanks for TinyApps.Org bloggist helping me to track down some additional alternatives to robocopy for file copy bandwidth management over networks.

Claus Valca