On the Hunt…

(no, this is not a picture from one of our network rooms, though the similarity looks uncanny.)
cc image credit: mrtom on flickr

One of the (many) critical projects I’m currently working on has our team upgrading the network switch hardware across our enterprise.

That alone should be fairly simple, get new switches as needed, pre-configure new switches, schedule swap-time with customer, un-patch cables from old switches, put in new switches, re-patch cables into new switches, move on to next site.  Easy right?

However a few very critical things (from a network security standpoint) are causing a lot of work and late nights.  Until recently, there was no real documentation kept on where all the network cables/jacks in the facilities were located, patch panel labeling at “old” sites was spotty at best, and furniture and office improvements left access to jack pates and trust in their labels weak at best.

So to take control back of the physical layer, my partner and I have to physically survey and account for the location and labeling of every cable we patch down into the new switches.  Considering the size of some of our facilities and number of users, this is a tremendously daunting process.  Oh yeah, the two of us typically have just a day on-site to do everything…from survey to final patch down.

Semper paratus, we load up and head out.

When we complete this project for all our facilities, we will have up-to-date floor plans of our physical cable topology and the documentation to match. Couple that with being able to administratively disable the actual ports (not in use) on the switches now, and we can go a long way to extending our network security and troubleshooting.  And this is just laying the foundation.

So, here are some free tips and tools and the methodology I’ve painfully worked out as our project and techniques have matured, that maybe can help others taking on this task; YMMV.

• Recon work and data-gathering is the key.
• A day or two prior to the facility upgrade day, I remotely run a series of scans from a box at the location to collect key data off the local network.
• Free IP Address Tracker from SolarWinds - This tells me which IP addresses are active (at that moment), their hosts name (in most cases), and some supplement data which could be useful. Results are exported into a CSV file.
• Colasoft MAC Scanner - This free tool very quickly rips though the local network and provides me a list of active IP addresses, hosts names (in most cases), and, very importantly, the MAC address of the machine. Results are exported into a CSV file.
• There are some other free tools such as Nir Sofer’s FastResolver and SoftPerfect Network Scanner and Radmin’s Advanced IP Scanner 2.0 that can also handle those tasks but for speed of scan and ease of export, I prefer the first two myself.
• Once those scans are in hand (usually both in less than 10 minutes), I then prepare a MyIPS.txt file for the location that contains all the IP addresses (one per line) that subnet contains.
• I then couple that TXT file with the following simple DOS BAT file I worked up. While the ultimate source to be credited for the technique is lost to me at the moment to give credit to, I suspect it is related to some tips found on this page: Ping list of computers from a txt file 

IF EXIST SCAN-RESULTS.txt DEL SCAN-RESULTS.txt

echo IP,HOSTNAME,MAC >> SCAN-RESULTS.txt
FOR /f %%i IN (MyIPS.txt) DO echo %%i & echo %%i >> SCAN-RESULTS.txt & nbtstat -A %%i | find "<00>  UNIQUE">> SCAN-RESULTS.txt >> SCAN-RESULTS.txt & nbtstat -A %%i | find "MAC Address">> SCAN-RESULTS.txt

• Normally I include ALL the IP’s for the location in the MyIPS.txt file that is feeding the dos-bat file above. I do so to ensure full coverage. However the drawback is that depending on the number of IP’s that your subnet provides, that can take a REALLY long time to complete. So if you want to save some time, and are willing to accept some possible skips, you could filter one of your Colasoft/SolarWinds export files for active IP’s only and feed it that instead.
• Note: I typically run these scans around mid-morning or mid-afternoon when I am most likely to catch the maximum number of users at their desks and PC’s turned on.
• Now that I have my SCAN-RESULTS.txt file which provides me the IP address, the HOSTNAME, and the MAC address of each “active/responding” IP, I have to clean it up into a nice CSV format.  Some quick cut/trim/replace work using Notepad++ usually does the trick in a short order.
• Lastly, I need one more CRITICAL piece of information, switch/switch-port/MAC mapping.
• I Telnet onto each of the local switches at the site and after authenticating, I run a “show mac-address-table” command.  I copy this output into a text file.  This proves me the MAC address being reported for each switch/port.  Your command may vary depending on switch manufacturer, model, and firmware version. However, if it is a managed switch, you should have something similar.
• Whew, get up and stretch and grab a beverage.
• Returning to my desk, I then use a combination of Notepad ++, Excel, and some clever multi-tab/multi-view work to “basically” create a spreadsheet that uses the MAC as the commonality for matching the information in the various logs.  My final spreadsheet contains rows for the IP address, the HostName, a device-name field (to be used for printers and other non-pc network items that HostName may not apply to), MAC address, switch number, port number of that switch. If you do this, you will have to work out the technique but I think you will get the general idea quickly.
• For rows where I got an IP address with a MAC address only, after all this work I perform some additional network discovery tricks (attempt to connect via HTTP/FTP to the device), a fresh NBTSTAT -A on just the IP (in case someone turned on their PC late in the scan and got skipped) or some other tricks.  Usually I achieve a 98% success rate.
• I then create two versions of this spreadsheet; one sorted by HOSTNAME and the other sorted out by switch number/port number.
• With these now printed out and on hand we hit the site and perform a physical survey:
  • I sketch out our data/relay racks and the patch panels on them.  I later convert this into a sweet Visio diagram using cool object figures of rack components.
  • I have a template sheet that represents our patch panels. I use this during the project rollout to document the physical panel/slot numbers, the actual labels for the ports, the room numbers where the jacks are located, if they are “active”, and if they are patched into a switch.
  • With floor plan in hand, we then perform a physical survey of the site, room by room, wall by wall, public and non-public spaces.  We note the actual data jacks found on the hard-copy, what they are labeled, and the name of the host system/device attached.
• With results in hand, I then sit down to reconcile the patch-panel documentation against the physical survey. Sometimes it matches nicely, sometimes it does not. Sometimes cables may have been abandoned (in the ceiling, in the wall, etc.) or are lost behind filing cabinets that cannot be moved. These are so noted and all “unknown” cable ends are not patched down.
• For cases where we were not able to see the jack to get a jack number (behind a desk) I can then pull out my spreadsheet and look up the system’s host-name to find its corresponding switch/port association. My partner or I then back-trace the cable from that switch/port back to the patch-panel to discover the panel/jack label.
• For rare cases where we were not able to “network discover” the PC-to-Jack-to-Panel-Switch association (example a cable that is found to be “hot” into a switch but has no PC on it and the jack is not labeled), we normally would have to tone it out. However, as anyone who has attempted to tone-out a cable known to be plugged into a switch, it can be a real challenge.  Luckily, I recently found a very reasonably-priced toning tool that has a “cable ID” feature: Psiber Data Systems Inc. Cable Tracker.  Set this little gizmo to Cable ID in one of three “pattern settings” and it will flash a beacon pattern on your switch. Just look for that beacon and you then are able to back-trace up the panel.  Alternatively, you could attach your laptop to the jack, note your MAC address, then telnet to the switch and find what switch/port it is on.
• Note, a partner and a set of good heavy-duty radio units (does anyone call them walkie-talkies” anymore?) make this a fast-two-person job; one person stationed at the patch-panel the other roams the field.
• Note: A very cheap but indispensible network testing tool IMHO is the (Amazon linked) Test-Um TP 100 - Network tool/tester kit. Plug this non-powered device into a network (or telco) jack and if it lights up, you will know the other end is connected to your network/switch.
• Once we are comfortable we actually know where are the jacks are and who is plugged into them (can be done pretty non-intrusively during business hours) and the patch-panel documentation template sheets have been updated with the findings, we wait until operations shut down and start pulling off all the patch cables, out all the switches, and mounting the new ones.
• Then we pull two hard-copy documents; the first is the patch-panel documentation sheet that tells us where all our active users are, and what jacks they are associated with.  Also I have a hard-copy template of our new switches/ports in tabular format.
• We look on the patch-panel sheet to find the first active port, patch it down into the switch, note on the patch-panel sheet we patched it, and note on the switch port sheet which physical patch panel/port number it came from. (Note I prefer that notation over “labels” as labels can change but the physical panel/port numbers are less likely to.)  And so we repeat the process until we are done.
• Final step (no customer accidently left unpatched) is to use the aforementioned Test-Um TP 100 unit at the patch-panel to back-check all the unpatched patch panel ports to see if there is attached network equipment we overlooked.
• If so these also get patched (for now) and noted on the patch down sheets. More on that in a moment.

The final documents are then converted into electronic versions to share with our other network administrators for use on an ongoing basis. When new cables are added to the site, the electronic floor plan copy showing the found jacks/numbers/locations gets updated, the panel sheet gets updated also. When customers/equipment is pulled, the sheets get updated, the ports get disabled on the switch. When customers/equipment is added/changed, likewise the sheets.

• I also use this information to the “label” the switch ports inside the switches with the corresponding panel number and port number.  This lets us find the systems physically very quick if we have an incident and are provided a MAC address or IP.  A few quick searches and I can not only disable the switch port immediately, but can then direct responding staff to the physical location of the system using all the documentation.
• Yes. All inactive/unpatched switch ports are administratively disabled to discourage local site staff from being creative and moving network equipment around into areas without IT approval and handling.

Later (very soon after we are done), I address the few “unknown” patch-downs we did where we found a hot jack on the panel that didn’t correspond to any physical network items during my scan discovery process or physical survey.

• Because I had documented the switch/port we patched it into, I can then telnet on the switch and get the corresponding MAC address.
• Then I run a Wireshark or Network Monitor capture session at the site filtering only for that MAC address.  That almost always nets me the host-name or other identifying information about the device. With that and our asset inventory at my disposal I can trace out the owner/name and assign that to a technician to perform a site-visit to perform another physical location check. Once that is confirmed, they can provide me the missing jack location and documentation is so updated.

We don’t (usually) have to contend with (authorized) wireless devices/access point hardware in our network so that makes things a bit simpler.

Also, after a while, you learn (and are not surprised) to find the odd non-authorized mini-switch/hub unit the local customer brought in without consulting IT (…we thought it would cost too much to request you to run a new cable, …it’s just for a few days, …we are having a meeting and the conference room has just one data jack, etc.).

Eventually once this phase is done, the IT policy makers/managers will need to decide if it would be good policy to implement and enforce MAC filtering on the switches to only allow known and approved hardware/devices to connect.  That will certainly lock down the switches even more but will provide even more IT network administration overhead to keep up with our constantly moving customers in all the offices we support.

See. Like I said.  Easy.

If anyone has any recommendations or additional tips/tools/utilities you have found helpful in your own network surveys and documentation acquisition, please drop your suggestions in the comments.  I’d value anything to refine this process even more.

Cheers!

--Claus V.