Tweak SharePoint and NAS Links

While working on a recent project, one issue we ran into was trying to upload very large files to the SharePoint server.

Some files were in excess of 500MB and the SharePoint server rejected them outright.  So then we used a compression program to slice them up in smaller chunks.  This worked, but if the files took to long to transfer the file transfer was terminated.

Long story short it took a lot of extra time and effort to transfer the files to a network share when everyone needing them could access them.  What compounded the problem was that the persons needing the files were using a shared system otherwise we might have had some more clever options.

Had the SharePoint administrator been available they could have made some temporary tweaks to the SharePoint parameter to allow us some extra room.

• Error message when you try to upload a large file to a document library on a Windows SharePoint Services 3.0 site: "Request timed out"

• SharePoint File Upload Size Limit - Bryan Worthy

• SharePoint Upload Size Limit on Server 2008 - IIS 7 - Windows Live

Typically, we just jockey files around between protected volumes of our local Novell servers.  The only problem with this is that not only do we have to share bandwidth with the production sites, we also have to busy (not really but to some degree) up the production server with the file transfer/access.

What this exercise really illustrated is that it might be nice to look at putting some Network Access Storage (NAS) points around at key locations that have some large pipes.  These would get around file-size restrictions on a single SharePoint server and as they aren’t performing production work, would be better behaved.

In my mind having a NAS solution that supported multiple NIC’s, had a web-client interface (for roaming technicians to access), and could be highly configured with both group and user policy and storage allocation limits would be great.  It would helpful if the software allowed for both file and folder/container transfers to save time.

We have lots of slightly older equipment and drives that could be easily repurposed for this task.  Certainly there are a lot of commercial solutions for NAS devices that range from enterprise-level support to SOHO needs.  And although there are some really disaster-event supporting solutions, we really wouldn’t need to use it for any critical or secure data.  Mostly just installer programs, utilities, and sharing one-time file-transfers across the map.

I’ve looked into Open Source NAS solutions and these two seem to stand out.

• FreeNAS: The Free NAS Server - Home

• Openfiler — Openfiler - Open Source Storage Management Appliance

I know there are others as well but I think I will focus on these for now.

Storage Utility Roundup

In the process of researching (starting with this Intel’s NASty little test tool InfoWorld post) I also found a few nice utilities to help test/benchmark storage media.  So here they are as well:

• Intel® NAS Performance Toolkit - Intel® Software Network – Really nifty tool.

  The Intel® NAS Performance Toolkit (NASPT) is a file system exerciser and analysis tool designed to enable direct measurement of home network attached storage (NAS) performance. Designed to emulate the behavior of an actual application, NASPT uses a set of real world workload traces gathered from typical digital home applications. Traces of high definition video playback and recording, officeproductivity applications, video rendering/content creation and more provide a broad range of different application behaviors. With the latest version of NASPT, users may even add their own custom traces. NASPT reproduces the file system traffic recorded in these traces onto whatever storage solution the user provides, records the system response, and reports a rich variety of performance information.

  NASPT includes an intuitive graphical user interface to get teams up and running quickly, a graphical data analyzer for in-depth performance investigations, and a convenient batch mode feature for performing multiple test runs with a single click.

• Iometer project – Storage device stress-testing and performance measurement tool.

• Iozone Filesystem Benchmark – Filesystem benchmarking tool.  Really cool in that it provides a variety of graphical data outputs so that you can get a better sense of what is going on.

• Bart's Stuff Test 5 - Says Bart, "Bst5...is a small win32 application for long term heavy stress testing storage devices. Bst5 supports testing at file and device level. File level support enables you to test any local or remote volume by file access. This makes it possible to test almost any storage device. As long as the operating system can write or read files from it, you can use bst5 to test it. In Bst5 this is seen as a "high" level test, you write/read data to/from a file using the file system support from your operating system. Device level support enables you to test local devices directly block-by-block. You can use this to test any removable or fixed logical drive, physical hard disk, or tape device. In bst5 this is seen as a "low" level test, bst5 writes/reads data directly to/from the storage device without the use of any file system. In other words, the storage device or media does not need to be partitioned or formatted before testing. If any file system exists on a storage device or media, a non-read only test will overwrite any data on it. Bst5 supports very large volumes, up to 16 exabyte (17.179.869.184 Gigabyte) enough to last for at least 30 years."

• smartmontools – Two “CLI” programs “…(smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA and SCSI harddisks.” Versions are available for most all OS flavors.  The primary download link provided to the “stable” versions was down but this Index of /smartmontools link provides some to beta/testing versions that are very new (but maybe not as stable). Note: the installer used often is flagged as “malware” by AV software.  I had to disable mine to download the file and then unpack it.  That said it seemed otherwise safe.

• GSmartControl – Provides a GUI wrapper that may be easier to use for folks.  Also offered in numerous OS supporting versions, though expects some additional supporting applications (GTK+ for Windows Runtime) to be installed on the system

• USBDeview – Nirsoft – Use this tool to get information on USB devices that have been connected to a system. However, for this discussion the latest version from Mr. Sofer now contains a USB read/write test so you can check the performance of the device. Designed to work with flash-media it does seem to work with most drive-based USB storage devices as well.

• Grand Stream Dreams: Pocket Hard-Drive Utilities – For even more storage utilities and tools, please see this recent post.

Cheers!

--Claus V.