Sunday, April 29, 2012

Forensically Sound: Quick Post #3

While I cannot say the past week was light, it definitely was quieter than most I encounter.

I’m still digging out the trench but the skies are clear.

Here are a couple of items that caught my attention this week.

Utilities and Tools

  • PDF Stream Dumper was recently updated to version 0.9.320. Check the second link for a summary of the new features; one is a VirusTotal plugin.
  • usboblivion - Google Project Hosting. This is actually an “anti-forensics” tool of sorts to strip out evidence of USB connected drives from the registry. It would be interesting to see if the tool itself leaves a signature of its usage (besides a clean registry I suppose…) behind.
  • Exploring Symbol Type Information with PdbXtract - Mandiant blog - New tool to explore programming database files. Probably most interesting to malware analysts.
  • triage-ir - Triage: Incident Response - Google Project Hosting. Another script-based tool to collect key information from a suspect system. Based on the Sysinternals Suite along with a few other key utilities. Kenneth Johnson has some thoughts recently in his Tools in the Toolbox - Triage post at the Random Thoughts of Forensics blog. Triage was updated to version 0.7 back on April 16th. More on the Automated Triage Utility here.
  • For those that still haven’t tried WinFE…. - Windows Forensic Environment blog. Brett Shavers shares a quick-start guide to encourage the hesitant on just how easy it is to build your own WinFE boot disk. Check it out.
  • Z-VSScopy Freeware - Z-DBackup - (free for personal use/$ for commercial use) - Very interesting tool new to me that allows you to browse VSS snapshots, cerate new ones, and copy files from a snapshot back out. It is actually a module of their Z-DBackup backup software, which makes sense as being able to leverage VSS shadow copies makes running backup jobs a bit smoother. Spotted in this AddictiveTips blog post: Create, Access, Delete & Mount Shadow Copies On Any Windows Version - Z-VSSCopy .   Other well-known tools for monitoring/accessing VSS: ShadowExplorer and the VSC Toolset: A GUI Tool for Shadow Copies.

Tips and Reminders

More Mandiant Goodies!

Investigating Indicators of Compromise In Your Environment With Latest Version of Redline - This is an outstanding overview of the use and functionality of Mandiant’s free Redline tool. It really shows the power this tool can provide during a system assessment and incident response…if you are very familiar with it! 

If not, after you have read Doug Wilson’s guided walk-through above, dive deeper into the Redline User Guide.

Then hop over to the OpenIOC Framework page and check out the details there. Need some more Indicators of Compromise (IOC)? Drop into the IOCs on the MANDIANT Forums.

One more item: IOC Finder to collect host system data and report IOC’s.

 An Eye on the Malware Front

Windows 8 forensic previews

The forensic learning and exploring is underway for the new Windows 8 system.  Here are just a few posts I’ve found touching on the new system.

Windows 8 Forensics - Recent post by Ethan Fleisher at the Senator Patrick Leahy Center for Digital Investigation, Champlain College. Ethan goes long and in this first review covers passes at Recycle Bin properties and USB Drive activity.

Windows 8 Forensics Part 2 - Ethan pickups up at Internet History.

Future topics of coverage promised by Ethan include Win 8 “reset and reload” feature, Event logs, Prefetching, Jump Lists, and File History features.

The Computer Forensics at Champlain College Blog where these posts came from contains a great collection of fresh material and the addition of this blog to my RSS feed list seemed a no-brainer!

Windows 8 Forensic Overview - Random Thoughts of Forensics blog - An extensive post by Kenneth Johnson covering Windows Registry artifacts.  Note, Kenneth updated his original post to reflect changes in observations between the Win8 Developer version and the newer Win8 Consumer version. Kenneth’s experience does highlight the challenge examiners and students have when a new OS is released in alpha/beta versions. It’s a great start to the learning process, however the path may be fraught with dead branches and dead-ends. Nothing will be 100% certain until the final release comes out. And even then, I suspect it will take some time for the forensic knowledge-base to be fully built-up.  There is still much to learn about Windows XP systems, and the books are still being written on Windows Vista/Win7 even as Windows 8 appears on the horizon!

The “X” Factor

Beyond the bits and bytes, deeper than the registry keys and that which lurks in unallocated space at the far-end of the hard drive, there is something special that sets some incident responders and forensic investigators apart from the rest.

Whenever I get a bit discouraged of the drudgery and lack of “play-time” learning new tools and techniques and getting my boots dirty in the trenches on a good investigation, I take heart from posts like these that are reminders that it really does take something special--an “X” factor--to be a great responder.

The Core Duo - The Digital Standard blog - From cepoug’s post

So I have recently been doing a lot of speaking and teaching, and came to an interesting conclusion about what are the core (an in my opinion, critical) skills of our trade, which I have affectingly dubbed, "The Core Duo".

When I really started to think about it, what we do (Forensics and Incident Response) really boils down to only two things. 

1. Spotting Patterns

2. Spotting Anomalies

Now, I know this sounds really simple...maybe too simple, but let me explain.  First of all, simplicity is something that I think is frequently minimized as being undesirable.  I think there are a lot of folks who think something to the effect of, "If something can be explained in simple, easy to understand terms, it must not be very complex".  I challenge that this is not the case.  I think, that even the most complex situations (which we all know, cyber investigations are among the most technical and convoluted anywhere) is made up of components that can be broken down and simplified.  Being able to do this is a critical element in actually understanding what you are doing and why you are doing it.  That in turn leads to be successful at what you are doing.  Which finally, leads to you solving the case, and potentially, some bad guy going to jail.

What makes a good forensicator? or how to get a job in Digital Forensics... - WriteBlocked. Michael Wilkinson opens up his review of key traits this way:

If you are already working in IT, it is possible to complete either an industry certification or graduate study or even transfer directly into a forensic position, although this is becoming harder as the pool of qualified applicants continues to grow. However no matter how qualified you are this will never guarantee you a job. Certifications and qualifications are only good for getting past the HR screening process. After that the decision will be based on other factors, partially on your performance in the interview and partly on your performance in previous jobs. When I am looking for employees I am looking for two things, motivation and the ability to solve problems. I will take these attributes over certifications any day.

A Fistful of Dongles: Border Collies - A Fistful of Dongles - Eric Huber turns to the four-legged friends for a nice analogy.

You will live and die by the people you hire and the leadership that you give them. The most critical element of your security program is having the right people on your team and providing them with the leadership and resources that they need.  You absolutely need proper tools to secure your enterprise, but the tools are secondary to the people who use them. The purpose of the tools is to help your people do their jobs. Too many organizations treat their people as glorified tool drivers rather than security professionals. If you are spending more money each year on your tools than you are on your people, you’re probably in a very bad place with your security posture.

Information security is very hard. It takes tremendous time, effort, and expense to even come close to mastery of critical information security skills such as incident response, malware analysis, and digital forensics. There is no tool that can ever substitute for a highly skilled and well led information security professional.


Meet Jet the Border Collie. You will find no creature on Earth more in the moment than a Border Collie like Jet chasing sheep. This is what they live to do. They are fantastic at it and they enjoy it immensely.  Incident response people are the modern day information security Border Collies.  We live in a time where we have an information security community made up of incident responders who absolutely live to get up in the morning and chase people out of our networks.

Eric goes on to expand his meme wonderfully.

This week I’m going to walk into the workplace with a Border Collie mentality; motivated, focused, and ready to perform.


--Claus V.

Saturday, April 28, 2012

Bits and Pieces for the Admins - Quick Post #2

OK. This next collection is a mashup of various items. Mostly new utility finds as well as few tips/tricks.

Probably something in here to play with for a while.

Microsoft Security Essentials 4.0 Released - Quietly…and related curiosities

I noticed last week that I had some out-of-cycle updates waiting. Curious, I checked on them and found that one was a new version of MSE. Most all of the changes are “under-the-hood” items, although there are a few GUI tweaks you might catch if you look close enough.  If you use MSE and haven’t gotten an update for it yet, I recommend you manually apply it.


Image Writer in Launchpad - Neat tool to write images to USB or memory sticks - Cool project by Michael Casadevall. Read this important announcement as well: Version 0.6 release is back - with a warning.

Image Resizer for Windows - CodePlex project. I’ve posted this utility before but for some reason never got around to installing on my own Windows system at home. Egads! Realized it when I had to resize some pics to send to my brother and wanted to tell him about it for quick-resizing work.

Known Folders Browser 1.0 (for Vista and Beyond) - Kenny Kerr - Super neat tool to show virtualized folders and linking to the actual paths going on behind the scenes. A must-have tool under Vista/Win7. Spotted at ToolTip: Known Folders Browser - Anything about IT blog.

The above tool reminded me of the similar tool (and beloved) SpecialFoldersView over at NirSoft. Take your pick. Both it and Known Folders Browser come in x32/x64 flavors.

Portable WinCDEmu - Not really “portable” but once you run it and do your business, you can then uninstall the driver from within the application. I prefer ImDisk Virtual Disk Driver and Pismo File Mount Audit Package supplemented with SlySoft Virtual CloneDrive on my system. You may also want to check out OSFMount (based on ImDisk), Gizmo Drive, or MagicISO for some more full-featured ISO mounting apps. Spotted over at ToolTip: Portable WinCDEmu - Anything about IT blog.

Fixing Remote Desktop Annoyances - What the.....? blog - a collection of clever tweaks for you power Remote Desktop junkies.

JunctionMaster (or MoveAndLink) - Clever and powerful tool to create NFTS Junctions on your system. Really neat and handy tool as long as you manage to keep your real and virtual link relations straight! Spotted via this JunctionMaster: Move A Folder Without Changing Its Path (Hard Link) post at Addictive Tips blog. Similar (though it doesn’t let you create them) NTFSLinksView utility over at NirSoft. Sysinternals’ Junction tool is a CLI tool to let you view existing ones and create them.  Junction Link Magic is another freeware GUI tool. Check out Link Shell Extension as well. Finally there is ntfs link over at for one more tool.

Tips of Note

Windows and boot disks larger than 2TB - Awesome roundup of tips and tricks at TinyApps blog. With large-storage (1TB+) getting more and more common at the consumer level, dealing with prepping these drives for maximum compatibility and usage can be challenging. Thanks TinyApps bloggist! Your awesomeness remains supreme!

Disconnect USB Devices Without Using Safely Remove Hardware Option - AddictiveTIps blog. For those of you who prefer the “snatch-n-go” technique this may be beneficial.  See also USB Safely Remove v5.1 (not free), EjectUSB (via freewaregenius/Softpedia), or USB Disk Ejector Free (via Softpedia). For more feature-rich USB removal support there is my new favorite, Dev Eject (beta version 1.0.23 released 4/25/12).

Windows Prefers Wired Connections - Clint Huffman's Windows Performance Blog - I really-did already know this but it was a nicely composed reminder.


--Claus V.

WinPE 4.0 - Quick Post #1

No time for extended commenting. I’ve got to clean out the “to-blog” hopper and time is winding down for the night.

In my work in the Case of the Unexplained Donut of Death post I tripped over the Windows ADK for Windows 8 Consumer Preview and installed it to get the Windows Performance Analysis Toolkit.

However, I was wonderfully surprised to find that WinPE 4.0 (WinPE for Windows 8) came along for the ride.


I haven’t had time to break it open and start playing with it in custom WinPE builds (as I am known to do) however I now have a watchful eye out for tips, tricks and material related to WinPE 4.0 building so when the time is right, I’ll be ready to go.

Sometimes--ok most times--once you start working with WIM files, particularly the BOOT.WIM file that forms the core of your WinPE build, you will need to add some items to it. While you can mount/commit/dismount the WIM file from the command line, it may be a bit clunky to the uninitiated. A nice GUI-manager for DISM will go a long way.


--Claus V.

Saturday, April 21, 2012

Case of the Unexplained Donut of Death

A few weeks ago, I had dropped in at the church-house to bring down some updates to the PC we run the services on. I also took advantage of the time to do some drafting work on that Sunday’s service material.

As I was working, one of the other ministers asked me if the worship-leader/sysadmin had gotten with me to look at one of the church administrative assistant’s PC’s that was not running smoothly.

I hadn’t gotten the message yet but had free time, so I popped into her office to take a look.

The user explained that a few weeks prior, another church member had their PC infected with some malware. One consequence was that system mailed out malware link spam from their email client. The church admin got one of those emails, it looked legit, followed the link, and ended up with a malware infection on her own system.

In the end the system had to be paved and reloaded from scratch.

Ever-since, the user reported it was constantly locking up at random times, though launching Internet Explorer sessions seemed to aggrieve it the most.

The dreaded Windows 7 Donut of Death had appeared. Luckily I’ve got an appetite for donuts.

First thing I did was take a look at the system hardware; i7 core processor, Windows 7 x64 Pro, 12 GB system RAM (wow!), really big SATA HDD. This PC was probably one of the most well-built ones in our church. After some checking and tests, I couldn’t find any issues that could be hardware related.

Next, I fired up Process Explorer.  Since the PC had once been infected, but was wiped/reloaded, I didn’t expect to find any unfamiliar processes and that was indeed the case. Everything running looked legit. No one process seemed to be showing evidence of a “CPU hog.”

With Process Explorer running on the 2nd Monitor and some changes made to the default columns, I started running some applications. Once I fired up IE there appeared the donut of death. The system seemed “locked” an unresponsive for almost a minute before control restored and applications became responsive again. Looking over at the Process Explorer, I didn’t see anything jump in terms of CPU hogging. I clicked around and ran some additional apps with the donut of death appearing again from time to time, more frequently than not. The behavior didn’t seem limited to IE. Other apps triggered the same result. Also, IE could run fine for a while and then suddenly when clicking a new page-link, the system freeze occurred again.

Next I fired up Process Monitor and began a capture. I wasn’t paying close attention to the time, but was able to get the donut of death appear a few more times before closing the capture and saving the file. When I was done I had acquired a 7GB Process Monitor trace file. Yikes!

While I was there, I also grabbed the available System Logs with Nir Sofer’s MyEventViewer tool and exported them into a txt tile for carryout consumption.

A quick review of the Process Monitor log saw a few tool-bar-related processes (Ask toolbar, Bing toolbar, etc.). These seemed to show some time-jumps after execution. The time jumps were a few seconds, not the up-to-a-minute differentials I was expecting from the donut time. The user didn’t use them, so I uninstalled them. Unfortunately, while the donut subjectively seemed to take a bit longer to appear, it still was there.

I off-loaded both these bits to my USB stick for more analysis when I got home.

I was stumped but donut hungry.

Initial Observations: While it was possible there was some hardware/driver problem, my initial feeling was that we had a rogue process taking control of the CPU’s. Since I couldn’t see any cpu spikes during the lockup in the temporarily frozen Process Explorer I wasn’t sure which one it could be.

Back at the homestead, I poured over the system logs. While they were quite interesting, I failed to find any smoking guns.

Next I analyzed the Process Monitor data. Poring over a 7GB file was very tedious.

Part of the problem was that I didn’t ever get an error dialog, BSOD, or some other “failure” to help me narrow down the search. The system would lock up and then unlock after some time and keep running “fine”. Hmm.

I was hoping to focus on a period when I saw the time-counter take a significant jump. Unfortunately, I didn’t find any obvious time jumps.  So while it appeared the system locked up significantly I didn’t find any time-jumps to point to a rouge process.  However, I did make a few notes. Suspiciously, there were a whole lot of logging going on with “coreServiceShell.exe” and “TmListen.exe”.  These are processes related to Trend Micro AV. Looking at the Process Monitor activity related to “coreServiceShell” it was a busy little child while “TmListen” seemed to be looking for related Trend Micro log(s). uiWnMgr.exe was also present and related to Trend Micro and seemed to be focused on certificates.

I discussed this with the sysadmin who was doubtful as this behavior wasn’t seen on any of the additional systems he had used Trend Micro on, including other systems in the church. So I set it aside.

Next week I stopped by the church-house again, but the sysadmin was out. My hope was to try to disable Trend Micro to see if that banished the dreaded donut.  Unfortunately, Trend Micro is protected with an administrative layer which requires a password to disable/turn-off. I contacted the sysadmin by phone but he couldn’t remember it and couldn’t access the location where he had it stored. I was on my own again.

This time I was a bit more prepared.

Resource Monitor

First I tried using the built in Windows 7 “Resource Monitor” tool.

Resource Monitor_2012-04-15_13-20-32

The sample shot above shows some of the data it collects and displays. Running it is very simple. Just type Resmon.exe in the run-line.

While I did get the system to lock up with the donut of death again, I still couldn’t find anything obvious here. My suspect processes still seemed to be behaving themselves in terms of CPU usages.

More on using Resource Monitor:

Performance Monitor

I also considered using Windows Performance Monitor as it also comes native on the systems. Just type perfmon.exe in the run-line to execute.

Performance Monitor_2012-04-15_13-25-14

While this is a good and powerful tool as well, it just wasn’t providing me the data I was looking for. The image above shows a sample view (not related to this particular case). It can be powerful in the right hands, but is not quite as intuitive to use “out-of-the-box” in a meaningful way.

More on using Windows Performance Monitor

Windows Performance Analysis Tools (Xperf)

However, all was not lost.  I had come better prepared and now had a powerful tool in my performance troubleshooting arsenal loaded up on my USB stick; Xperf.

Basically, I had previously downloaded the Windows Performance Toolkit at home on my own Windows 7 system. I then copied the C:\Program Files\Microsoft Windows Performance Toolkit folder over to my USB stick for deployment and usage in the field.

I practiced using it first at home, but mysteriously kept getting the following error:

C:\Program Files\Microsoft Windows Performance Toolkit>xperf.exe -on DiagEasy
xperf: error: NT Kernel Logger: Cannot create a file when that file already exists. (0xb7).

That was weird as I didn’t have any traces set to run on my home system (that I was aware of at least).

That took a bit of troubleshooting but I eventually found the issue:

Basically it was tripping over Process Explorer which was running by default on my system at all times. Process Explorer uses NT Kernel Logging to capture data it uses and they were fighting. Once I disabled Process Explorer temporarily, Xperf worked fine.

Anyway, on the target system back at the church I copied the folder to the local drive, then opened a DOS box and pointed to the location and ran the following command.

C:\Microsoft Windows Performance Toolkit>xperf.exe -on DiagEasy

I then launched some applications, Internet Explorer, loaded a few web sites, ran paint, notepad, calc, etc.  Each time I would get the donut of death. Good data!

I would then stop the trace and export the file with an appropriate file name.

xperf -d testdata1.etl

I noticed complaints about dropped trace elements with a recommendation to increase buffer size. I probably should have heeded the advice, but didn’t since I was using the default diagnostic capture mode.

With three traces in hand now with sizes of 48MB, 50 MB, and a whopper of 301MB, I felt I had a pretty good sample set. During the testing, I made written notes of what I was doing (which app launched) and when the donut appeared. I hoped to correlate these events.

Now that I had the files back home, I fired up the GUI “Windows Performance Analyzer” tool, xperfview.exe.  Actually, since I had installed it on my home system the “ETL” file extension was pre-associated with that application.

CPU Scheduling Aggregate Summary Table - C__Users_Tatiana_Desktop_CM PC Logs_CM-_2012-04-15_13-32-18

You can select different “Frames” to view data from which then load into the horizontal panes. From here you can compare events as well as hone-in on specific time-slices. You can also display the results in a tabular form. The image above is a screen-capture from a real trace from the problem system.

Poking around in this data from the three captures I had taken showed time-after-time that (overall) coreServiceShell.exe was the heaviest user of CPU processes far-and-wide.

This information, coupled with additional data with filtered Process Monitor session data I had captured observing the behavior of that process made me feel much more certain that Trend Micro was in fact the cause of the donut of death.

More on using Xperf and the Windows Performance Toolkit

Bonus Xperf Material:

Today while poking away at the CodePlex - Open Source Project Hosting site I found three projects that may leverage the Xperf data in helpful ways:

  • XPerfUI - This is a GUI wrapper for the Xperf command-line performance analysis tool.  I I mentioned, my own performance traces dropped (lost) anywhere from 26301 to 194970 events during the trace capture process. The CLI arguments are too tough once you have your confidence up, but this tool might make the process a bit easier.
  • Xperf123 - Xperf perf data collection made as easy as 1-2-3 - This project provides a Wizard-based method of selecting a trace profile (it does the CLI arguments automagically for you) and it creates and fires off the command. Then you can analyze the results.

As the warning dialog I mentioned said when loading my trace files in xperfview, “This is usually caused by insufficient disk bandwidth for ETW logging. Please try increasing the minimum and maximum number of buffers and/or the buffer size. Doubling these values would be a good first attempt. Please note that this action increased the amount of memory reserved for ETW buffers, increasing memory pressure on your scenario. See “xperf -help start” for the associated command line options.”  I probably should have set -maxbuffers 1024. Will try that next time.

Getting the Windows Performance Analysis Toolkit bits

As I said earlier, I had to install the toolkit on my home Windows system. However, once I did so, I was able to copy the installation folder in the Program Files folder to a USB stick for deployment and usage on a per-system basis. As I understand it, because you can download the entire package, you could (in theory) extract the downloaded file-set and snag them that way. I found the download/install/copy-to-USB method painless myself.

Here are some links to guide you through the download/installation process.

So, back to the donut of death…I diverged for a moment.

I had some great trace data, and my eyes on some culprits, but while the output in the xperfview.exe application (and table views) was helpful, I kept feeling I was still not able to see the big-picture as clearly as I wanted. After all, I’ve got more than a few Windows process/internals books but I’m no Windows programmer or systems engineer so I was still digging around slowly in the data.

Then I found this.

Windows Performance Analysis Toolkit (WPT) for Windows (SDK 8)


This is the next-generation WPT and has some fantastic power and Xperf analysis toolsets!

Windows Perfmance Analyzer SDK 8

Compare the data views I had selected and was working with above in the Windows Performance Analyzer (wpa.exe) as compared to the prior screenshot in xperfview.exe. The difference is almost night-to-day. The image above is a screen-capture from a real trace from the problem system.

I found this interface much more powerful and intuitive to use.

xperf.exe, xperfview.exe and all the previous tools are still present but wpa.exe is much more user-friendly for analysis work IMHO. Also added to the mix is the tool Windows Performance Recorder (wpr.exe) as well as a GUI-based wizard WPRUI.exe to help set your performance recording sessions. I didn’t use this tool in this troubleshooting but will be working to figure out what neat things I can do with it beyond xperf.

Getting and installing this one on my Windows 7 x64 bit system was a bit more challenging getting started. It doesn’t seem to have a full download package set to just extract these bits out of. Rather you seem to download a web-based pre-installer to pick your packages and it then fetches and installs the bits. All that said, once I figured out what I needed, the process went smoothly, and I was also able to copy the installed folder location to my USB to take with me in the field for captures on other Windows 7 systems.

Here are the links I studied to download and install it:

With the additional views and drill-downs I was able to do in the Windows Performance Analyzer, I had all the proof I needed to convince me that the most likely culprit for the donut-of-death on the church’s system was Trend Micro in general and the “coreServiceShell” process in particular.

Apparently “coreServiceShell.exe” thinks it’s always time to make the donuts!

Armed with this information, I now invested myself in the Google with “coreServiceShell” and found some interesting stuff. Lots of complaints were seen about CPU hogging with Trend Micro and coreServiceShell in particular.

Turns out that process in particular is a busy little bugger (as I’ve seen in my 7 GB Process Monitor capture). It appears to be the main scan-engine for the AV product, not only checking files upon access and execution, but also acting as a web-proxy while pulling in pages in the web-browsers. I could find and locate the proxy activity in the Process Monitor traces. Very interesting thing.

In fact, the issues were so bad that Trend Micro offers a Hot Fix for just this performance issue.

I copied the Hot Fix down to the sysadmin’s network share folder with a few more useful links to the problem.

Since I couldn’t get admin rights to install the Hot Fix myself on the user’s system (yet) I’m waiting for feedback if the Hot Fix has been applied and if it fixes the issue. Based on all my work I’m confident this is the trick.

I suspect that while the CPU activity loads looked normal while I was watching them in Process Explorer -- just before donut of death -- when it went overboard Process Explorer couldn’t keep up and froze so I didn’t see the jump in CPU’s while it hogged/chugalug’ed away. Once it was done, things returned to normal and there I was at normal CPU levels again…until the next lockup.

Additionally, since I didn’t set my Xperf CLI to -maxbuffers 1024, I probably dropped a good portion of the trace capture events during the lockup process as well. That said, the tools above gave me sufficient information to say pretty confidently this was the donut maker.

While this isn’t for the average Joe, these tools and techniques are extremely powerful and once mastered, can give the sysadmin tremendous confidence in working your way through performance issues on a Windows system. Since they can tag-along on a USB stick, you can deploy them as needed as long as you have them handy.

Additional Windows Performance and Analysis Linkage

Here are a few more links and blog and resources I thought were very insightful to getting up to speed tracking out Windows Performance issues. More than a few of these have been added to my RSS feed list.

Hopefully this super-long post and linkage has provided some good comparison views between different tools and techniques tracing out Windows performance problems.

I learned a lot in the process of this one system, and it just another reason why I find benefit in taking a look at problem systems. I usually walk away with a more honed skill-set in the process.

I know I will be coming back to this post myself for some time to come re-reviewing the linkage and information here.


--Claus V.

Sunday, April 15, 2012

Bits and Pieces: Mini Link Rundown

I probably should be pleased to have crammed in three posts this weekend.

Alas I am not. I’d intended to get one more “biggie” out the door this weekend…aimed for all you sysadmins. I have in mind a “Case of the Unexplained…” type theme on running down some crazy Windows 7 system behavior on a system at the church-house, multi-GB trace file captures, and sundry stuff like chasing a white rabbit down CPU process utilization percentages and disk utilization by process IO type.

I’m back from that chase with lots of notes, but to do it justice, I’ve got to wait till next week.

So let’s just enjoy our company at final call over these late-breaking weekend links. Hopefully they will carry us into the week with some inspiration and a few shiny new utility toys to play with at our desks.

Adobe April 2012 Black Tuesday Update - ISC Diary - In case you missed it, there were a number of critical Adobe patch updates this week

APSB12-08 - Security updates available for Adobe Reader and Acrobat - Adobe Security Bulletin - Updates now to 9.5.1 and 10.1.3. This goes for both the PDF “reader” versions as well as the “full” Acrobat PDF generating software application. Patch!

At the end of last month some Adobe Flash Player updates came out, one feature of which is to now include an “auto-updater” feature for Flash Player (if so selected in the options). That release back on March 29th was

Guess what snuck out of Adobe Friday (the 13th?). Version of Flash Player.

  • 4/13/2012 - Flash Player Update - Adobe Forums
  • Flash Player 11.2, AIR 3.2 - Adobe Release Notes
  • Adobe - Flash Player - Lists your installed version (check page with each browser you use) and a table of the current version for all platforms.
  • Installation problems | Flash Player | Windows - Adobe. I dropped over to this page, then scrolled just a bit lower to the “Install in a firewall proxy server environment” section to grab all of the direct download installer links there.  It’s a one-stop shopping session!  Then I spent some time manually updating my portable browser plugins to all the newest versions. Sheesh. Sadly I’m getting very good at it and have now even crafted a custom batch-file to auto-copy/overwrite the new Flash/Reader version DLL’s to the plugin directories in my browsers to save me time.

If in doubt, try running Qualys BrowserCheck page in each of your web-browsers to check your patch-level or use the Secunia Online Software Inspector (OSI). Either of these tools will help tell you if your browsers are securely patched.

Download just imagex.exe (568k) - TinyApps blog. I LOVE Microsoft’s ImageX.exe imaging tool. It has become second-nature for me to use. If you do a lot of WinPE building and use you probably have already extracted it and keep it handy.  However, if not, TinyApps blog shares a quick tip on getting your hands on it from the WAIK without all the drama of installing the WAIK on your system.

Increase hard disk size in VirtualBox 4.x - TinyApps blog. I know no-one actually creates a virtual hard-drive without first considering (and allocating) all the size they will every need (and then some) before they first get started. Right? TinyApps bloggist has a great walk-though on how to enlarge your drive size without having to mail off for sketchy blue pills. Lots of supporting linkage at the end as well.

Value of Targeted Timeline Analysis in Research - Windows Incident Response blog - Keydet89 provides a great post on the work that goes in towards gaining a better understanding of event timelines and Windows behavior. It’s through detailed work like this that our knowledge gets sharper.

Challenge: What can you do with funky directory names? - ISC Diary post - Mark Baggett warns us to beware those funky file/directory names in Windows! Check out the comments carefully for more feedback. On a related note, the Hexacorn Blog Forensic Riddles posts contain a whole lot more of file-name and directory name tricky shenanigans to be aware of!

NetworkMiner 1.3 Released - NetRecSec has released v1.3 of the amazing (and still free) NetworkMiner NFAT. This release contains a number of new parsing and extraction features. Go get it now! Of course, if you are lucky enough to be able to purchase a copy of the NetworkMiner Professional version -- sadly I’m not ;-( -- that too has been updated and you can get your upgraded version for free from their customer portal with login. Happy upgrading free and pro’s alike!

eXtra Buttons: utility buttons in the title of the window - freeware - clever little utility that adds a few extra option buttons to your Windows windows. The default windows options in the top-right corner are minimize, maximize, and close. This app gives you up to thirteen options for managing your window, including roll-up/unroll the window at the caption bar, minimize to System Tray, transparency effects, and minimize to a predefined box area on your desktop. I don’t usually use windows tweaking utilities, but this one could be very useful for you multi-window-multi-taskers.

Synkron - freeware - Folder synchronization application. Yeah, I hear you. Claus, really? After that super-long roundup of sync/backup apps you recently posted? Just had to add another one? Yep. This one has a pretty intuitive interface and also comes in a Synkron Portable | PortableApps version as well. More details in this older AddictiveTips blog post.

Colasoft Ping Tool - freeware - Colasoft has a great and super-handy ping tool that supports pinging multiple IP addresses as well as useful charting tools for monitoring and analysis.

Anti-virus scanning exclusions - ISC Diary post - Daniel Wesemann kickstarts a discussion on setting exclusions in your AV scanning policies. Some vendors have recommendations on file/folder exclusions to improve system performance. On the other hand, the thought of creating “safe-zones” that could be exploited by malware for APT landing could outweigh the benefits of following the recommendations. Check out the post and the lively comments that follow. Do you even know if/what your own (or your customers’) policies are regarding AV exclusion settings? Worth looking into.

Malware blocks booting - The H Security. News post about a pretty new ransomware attack that hits the MBR discovered by TrendLabs. While the vector itself isn’t necessarily anything new (messing around with the MBR) apparently the combination of using it in a ransomware attack is. Trend Micros also has instructions for removing the infection if you encounter this bad-boy.

And then there was this “bad news getting worse” over the weekend:

Medicaid hack update: 500,000 records and 280,000 SSNs stolen - ZDNet Zero Day blog.  Original post here: Medicaid hacked: over 181,000 records and 25,000 SSNs stolen.

Expect the fallout from this one to be pretty massive. Quoting from Emil Protalinski’s article linked above:

DTS had recently moved the claims records to a new server, which had a configuration error at the password authentication level, allowing hackers to circumvent the security system. DTS says it shut down the affected server, implemented new security measures, is reviewing every server in the state to ensure proper security measures are in place, identified where the breakdown occurred, and has implemented new processes to ensure this type of breach will not happen again.

It was just a year ago we were dealing with a similar mess here in Texas. Although in that case, it seemed to be more an issues of inside IT data mismanagement rather than a hacker attack. 

Hoping the week ahead gets better even though it hasn’t started yet.

Hang tough and remember “Constant Vigilance!”

--Claus V.

Saturday, April 14, 2012

Malware Analysis Resources

This is meant to be a complimentary post to the URL Scanner roundup post back in January.

Let me be the first to say I am not a malware reverse-engineering analyst.

On the other hand, when I am responding to an incident involving a system compromise, and/or am trying to both clean the system as well as understand the potential impact of what happened, being able to analyze a suspect file is critical.

It can not only give me a better understanding of how to clean it, but possibly how it got there in the first place. This lessoned-learned may help strengthen our security perimeter.

So having a collection of resources that can help analyze a malware (or potential malware) file is important to me.

The following resources are a collection of on-line file scanners, analysis-report-generating, and local sandbox creating tools to aid in that process.

There are a number of similar “list-of-lists” like this one. I’ve just tried to collect them for my own personal reference.  Major hat-tip and credit goes to the following sources which have already paved the way before me. You may find some more more resources here that I haven’t linked to as well as additional descriptions and feedback.

And as Sketchymoose points out in the close of that post, before you start uploading files to any of these resources:

So now, keep in mind-- your submitted file is now out on the internet and is now on some database. Some of these may be owned by AV companies which look for new juicy malware to add to their signatures. So, if you are really worried about that:
(A) read documentation on their website to see what happens with collected data
(B) do your own analysis
(C) Ask customer/boss what their position is about submitting files to these sites -- make sure you know the answer for choice 'A' too for this one
Remember collaboration is one of the biggest deciding factors in incident response, but use common sense and discretion.

On-Line Scanners and Virus/Malware Analysis Tools

PDF File Analysis Tools

Not a PDF but Malware Tracker’s +Cryptam service can scan "Office” documents for malicious content as well.

Sandbox Tools for Malware Analysis 

Adobe Shockwave/Flash Analysis Tools

Mandiant - When One Word will do…

  • MANDIANT - Red Curtain - From their product description: “MRC examines executable files (e.g., .exe, .dll, and so on) to determine how suspicious they are based on a set of criteria. It examines multiple aspects of an executable, looking at things such as the entropy (in other words, randomness), indications of packing, compiler and packing signatures, the presence of digital signatures, and other characteristics to generate a threat "score." This score can be used to identify whether a set of files is worthy of further investigation.”
  • MANDIANT Find Evil - tool that uses disassembly to detect packed executables.
  • Be sure to check out all Mandiant’s Free Software offerings as many other tools here may aid in a malware response investigation.

Lessons Learned and Wisdom Shared by the Malware Analysis Pros

Thanks to the hard work and community-spirit of malware analysts, we can “sharpen-our-saw” against their efforts. These are some of the best places to start.

I sincerely hope you find several good take-aways from this post. It’s been simmering a while and I think it will greatly aid me in my own efforts and responses.


--Claus V.

Zalman ZM-VE series Enclosures: Next-Gen Virtual ODD

Last week I received an anonymous comment on my iodd : Multi-boot madness! post. -- Thanks Tipster!

You may recall the IODD device is an external drive enclosure that supports multi-mode operation:

You can use it in an external hard-drive mode to just copy files back and forth and access them as needed.

You can use it in an ODD (optical disk drive) virtualization mode. In this mode you load a bunch of ISO image files onto it. Then while operating in this mode, you can select the ISO file and PC will see it as a virtual optical disk. If the ISO represents a bootable disk image, you can likewise boot the system with it. This massively cuts down on the number of “burned” boot disks as well as installation media disks you need to carry. Just carry this device and you are limited only by the size of your external drive capacity.

Then there is a multi-mode where it operated in both an external drive/ODD mode.

I love the device I have and it has made my life so much cooler easier when I roll out on an troubleshooting/incident response call.

The model I have and cherish is an older iODD 2501 model. It has both an eSATA connection (requires independent power connection via dual USB plugs), and a USB 2.0 connector.

It’s rock solid, and firmware updates were available to allow it to operate all partitions including any that include your _ISO store to be NTFS. download page. My primary (_ISO holding) partition remains FAT32 because I’ve yet to create a 4 GB+ ISO file I need to boot with or access and that’s all I presently use it for on that partition.

Anyway… as my tipster points out, it appears there is a new model out from the IODD manufacturer. Based on the main page, it looks like “worldwide sales” is seeing it marketed/distributed under the “Zalman Tech” name now.

The features of the newest model appear pretty much the same except for the addition now of a USB 3.0 port rather than the older USB 2.0 connection and eSATA combo. The USB 3.0 can net you up to 5 Gbps transfer rate if you have the hardware to support it. Compare that to the 480 Mbps USB 2.0 rate. Wowzers. It also appears to provide some SMART drive stats on the display now as well.

Here are some useful links based on some quick research I did over the last week.

Still Super Cool. Still Valca Recommended.

Related Concept:

As I said, my iODD 2501 model is running strong and I don’t have any systems that support the USB 3.0 hardware (personally or in the trenches) so I’m not rushing out to pick a Zalman model up just yet. However I can’t help but be a bit envious of the transfer rates in this new model.


--Claus V.

Windows 8 Linkage: “Passage Public Metro” version

cc image credit image by david.nikonvscanon on flickr


So Claus, where do you stand on Windows 8 at this point?

Well, to be honest, I’m really liking the under-the-hood improvement talk on how things are working in Windows 8.

What I still find very hard to overcome are the end-user interface changes and the challenges trying to restore it to some form of a Windows “Classic” interface and operation.

I get that Microsoft wants to forge ahead with a new interface and blending between the Windows Phone interface and the computer system interface. I get that “apps” are all the rage. I like old-school design but am pretty comfortable with moving to new designs. (I no longer curse the Ribbon interface in Office 2007/2010.)

So…when Windows 8 gets released in a final version, I’m not going to be rushing out to upgrade all our systems to it from Windows 7. Windows 7 is super-stable for our needs at home and everyone is very happy. That said, once it looks like things are stable and I’ve come to terms with the new interface, I’ll probably upgrade just one of our systems here at the Valca ranch to Windows 8 and see how things go.

In the meantime, here are an updated collection of Windows 8 linkage building on the prior grand stream dreams: Windows 8 Linkage: “Majestic Metro” version post. It definitely requires an update since the Consumer Preview release of Windows 8 has come out a while back and some of the things from that one either no longer apply or may not work in this release.

Truth be told, tonight will be the first opportunity I’ve budgeted myself to load up with Windows 8 Consumer Preview version in a virtual machine.

Windows 8 “Consumer Preview” Version - Start Here to Get It

Windows 8 - Related Betas

OK. None of these are required for Windows 8 Consumer Preview. However they are related to it and I thought some folks might be interested in playing around with them as well. If you don’t know what these even are, then just skip down to the next section.

Windows 8 - Install It

There are a number of methods and platforms to install Windows 8. Review all the ones below carefully to figure out which works best for you. I personally am currently going with installing it into a Virtualbox session. When I get closer to pulling the trigger on installation to one of my laptops at home, I’ll first install it into a VHD and then dual-boot my system so the Win8 install can run on real “hardware” to check driver compatibility and system performance on the iron rather than in a virtual system. I did that before with Win7 and found it very beneficial.

Windows 8 - Under the Hood Stuff

These are the things that make me look forward to Windows 8 despite the stupid fact the “classic” interface is stripped out and requires considerable effort to restore using tips/tweaks/third-party tools to accomplish. Note: when I say “classic” I’m not talking about the theme that is a toss-back to Windows 2K/XP but rather the “classic” GUI with the program bar, the “start” menu, the system tray icons, etc.

Windows 8 - To Go

Windows “To Go” is basically a feature in Windows 8 that allows it to run “full OS” from a supported USB storage device like a flash drive or external hard-disk drive. I guess it could be considered an advanced version of WinPE but with all the benefits of the OS with no feature strip-out or additional “hacking” required that custom WinPE builds require to get past a plain DOS box environment “out of the box”. It is very intriguing to me and should be a cool option…if you meet the license requirements as well have a robust and super-fast USB device/port.

Windows 8 - Tweakages

Getting Windows to the way you like it is still important. Here are some important (to me) tweaks, tips, and tweaking tools to make it possible.

Windows 8 - Deeper Insights

Windows 8 - DaRT (Diagnostic and Recovery Toolset)

This off-line system boot tool is kitted out with a collection of system administration tools to aid in the diagnosis and recovery of a tanked Windows system.  This isn’t as easy to get your hands onto so some work and signup with Microsoft is required to get it.

Indirectly related but still interesting.

Windows 8 - Usage Tips

New user interface, new things to learn navigating around and completing basic tasks without beating head on desk…

Windows 8 - Miscellanea & Rumor Mongering

Already mostly covered in the linkage above, but sometimes you just can’t resist poking that ant-pile with a stick…


--Claus V.