Sunday, October 18, 2009

Final Push Linkfest

Just a few more links to wrap up a delightful weekend.

  • Native boot from VHD on a Windows XP computer – Mark Wilson’s blog – Mark has brilliantly worked out a solution to native booting that I haven’t seen yet.  Windows 7 allows you to Boot to VHD with just a modicum of effort and run your VHD system on the physical hardware layer rather than a virtualized one.  Then some clever folks figured out how you can Dual Boot Windows 7 on Vista via VHD file by swapping Vista’s boot manager file version with the one from Windows 7 which does support VHD booting.  I’ve been running Windows 7 RC x64 on both my and Lavie’s personal laptops for some time this way all the while preserving our existing Vista installs until Windows 7 final comes out.  I had hoped to do the same thing in keeping a native XP system in place and then boot Windows 7 from a VHD but XP didn’t seem able to be VHD booted under W7.  Mark’s solution doesn’t change my issue (Windows 7 as the base install and XP in the VHD being booted).  Instead he just simply reverses the equation.  Keep XP as the base install, do some freaky (but clever) boot chain-loading configuring.  This gets around both issues.  First the Windows 7 bootloader is forced upon the native XP system.  Then the Windows 7 VHD system is attached and set to be a native booting option. Finally the legacy (XP) boot settings are chained to the boot loader configuration.  Like I said, very clever and this solution method actually works.

  • F-SECURE Releases New Rescue CD – CyberSec blog. According to the release notes, it is based on Knoppix and contains a minimal set of tools to off-line scan a system against malware/viruses. It also now contains a few tools to recover deleted files and messed up partitions and check the S.M.A.R.T data of the local hard-drive. DAT files can be accessed/updated using a prepped USB stick.  More Info from the F-Secure blog: What is F-Secure Rescue CD? and Rescue CD 3.11

  • EraserDrop Portable - PortableApps.com. I love Heidi Eraser for secure file erasing and zero-ing out freespace on a drive. as well as the Eraser Portable version.  However it isn’t always the most convenient tool to use as you have to drag files into a schedule task, then run the task.  EraserDrop takes the nuissance out of it by displaying an icon on the desktop to which you just drag-drop-n-secure-erase your files on the fly.

  • Sysinternals Site Discussion : Updates: Autoruns v9.56. “Autoruns v9.56: This update enables Autoruns to view registry entries that have permissions only allowing the System account access and fixes a bug that caused some rundll32-hosted entries to not display correctly.”

  • Internet Evidence Finder – Updated – JADsoftware.  “Version 2.0.5 updates: Fixed bug where physical locations of located artifacts beyond 4GB were sometimes incorrect. IEF is also now compatible with Mount Image Pro! Tested/verified with version 3.26.522.” 

  • Patch Registration Cleanup Tool – Microsoft Download Center. “Brief Description: On a computer that has a Windows Installer based product installed, you may receive an error while installing an update for the product and the installation of the update may fail.”  See KB976220 for more information.

  • Microsoft SharedView – Version 1.0 Release – Spotted via Mark Wilson’s blog post SharedView: Free desktop sharing across the ‘net.  Take a look at the extensive release notes for a roundup of the bug-fixes and compatibility enhancements.  As Mark kindly noted, I’ve posted about it on GSD (Microsoft SharedView: OMG this is Free?!!!) and while it is no replacement for a dedicated remote control client solution (Remote Desktop or ShowMyPC), for collaboration efforts with several participants across networks, it works great.

  • Deploying Windows® 7 Essential Guidance from the Windows 7 Resource Kit and TechNet Magazine – Microsoft Download Center. Stop right now and go download this resource PDF file.  Regardless on your feelings on Windows 7, it contains a smorgasbord of valuable information, all rounded up into a single volume!  For example, Chapter 6: Developing Disk Images, Chapter 7: Migrating User State Data, Chapter 9: Preparing Windows PE.  There’s a lot more in the other chapters as well, but if you know the terms Win PE, ImageX, or WIM you really should download and print out this resource collection.

“…creates a virtual envelope for Windows XP N, the version Microsoft created without Media Player 9 pre-installed, to appease the European Commission. But there's a very good reason Windows 7 users may want the (N) version over regular Windows XP Mode: There's no really good reason for most users to be virtualizing Media Player 9 anyway, since XP Mode's real purpose in Windows 7 is for running older Windows apps whose developers didn't make the transition to Vista. If you want to watch a movie, you've got Media Player 11 in Win7.”

Now you know!

--Claus V.

Saturday, October 17, 2009

Tracking down a pagefile.sys mystery

A Prologue

About every four months, I build updated base images for our technicians.  We have about ten different hardware platforms current deployed for our end-user desktop and laptop systems.

It’s a touch time-consuming but the following procedure works well and gives us flexibility and redundancy.

I take the “official” company image and apply it to a cleaned and formatted system drive.  I then apply a round of Microsoft updates, plug-in updates, and a few others to the system, add in any system configuration or policy tweaks that we have discovered are needed or desired since the last image-build. I run CCleaner to dump temp files and stuff. I finally sysprep it and shut it down.

I then capture the image using both ImageX and Clonezilla.  ImageX (and the WinPE disk) seem to like 512 MB RAM or better to function. Clonezilla can tolerate 512 or less.  I know it’s kinda silly to have the same image in two different formats but the guys seem to enjoy the flexibility and with large USB hard-drives cheap, space isn’t an issue.

I could (and have) make a multi-image ImageX single WIM that contained all of the different systems in one WIM but it wasn’t very popular.  So I just stick with what works.

So this go-round I had prepped and captured a system WIM and was shocked to find that the size had jumped from about a 2.6 GB WIM file image to over 4 GB.  While there were over 34 Microsoft patches to deploy in October alone, I didn’t expect to see that big of an incremental jump.

Something seemed very, very off here.  Why the big jump?

I mounted up the WIM file on my main system and then tossed SpaceSniffer at the mount folder.

Immediately I found the issue; a 1.6 GB file called “pagingfile.sys”.

But the standard Windows pagefile.sys should be excluded by ImageX by default capture parameters.  What was going on?

That was the mystery to be solved.

Chapter One: “Fraternal Twins”

At first I thought I must have made some really bone-headed mistake during my system image preparation.

So I cleaned the last system I was building, rolled back to the original image, and went through the process again.

Before I ran sysprep I checked the root and looked for any hidden system page files.

Curiously, I found not only “pagefile.sys” with an early time/date stamp but also the “pagingfile.sys” file as well with a later time stamp.  Both were approximately the same 1.6 GB size for a combined total of just over 3 GB of drive-space taken up.

Hmmm.

Chapter Two: “A Windows Page File by Any Other Name…”

While I have from time to time “tweaked” a Windows system to adjust the page file I’ve never attempted to change the name of it and didn’t have a clue how that was done.

Turns out it is not controlled from a GUI but rather from the SYSTEM Registry hive.

 How to Move or Rename Your Paging File in Your Runtime – MSDN

Regarding the paging file, an uncommon question in the past has been "How can I rename the page file from 'pagefile.sys' to something like 'andy.sys'?"

A more common question on the pagefile has been "How can I relocate pagefile.sys to another partition that is not protected by EWF?"

The information for both of these questions can be found and changed at the following registry key:

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
Name: PagingFiles
Type: REG_MULT_SZ
Data: C:\pagefile.sys 150 500

Within that Data field you can adjust the NAME of the pagefile, its LOCATION and its MINIMUM & MAXIMUM values.

I off-line booted the system with my custom Win PE 3.0 bootable USB stick, loaded up the SYSTEM registry hive (how-to link #1 and how-to link #2) and inspected the key value.  Sure enough, it was set to “C:\pagingfile.sys 0 0”

However when I went back and compared the key value in the original “base” image I was building from, the key value was the default “C:\pagefile.sys 0 0”

Somehow between the original base image and the end of the building process, the page file name was being changed.  Was it a virus? A trojan? Some bone-headed system building step I was doing or missing?

Chapter Three: “The Inspector orders broiled Red Herring” 

Sysprep perhaps?

My first thought was that somehow the page file was getting changed in the pre/post Sysprep process.

I inspected both my sysprep.ini and unattended.txt file looking for any clues by didn’t see anything to cause the behavior.

While you can control some pagefile settings in sysprep.ini, nothing I found dealt with renaming it.

This was getting crazy.

Chapter Four: “Hide and Evade”

I even ran BareGrep against the whole mounted WIM image file looking for “pagingfile.sys” with no luck.

So I got my system image all ready and sysprepped, shut it down, and off-line booted to manually reset the SYSTEM Registry hive already noted above to “pagefile.sys” and manually deleted both the pagefile.sys and pagingfile.sys files from the system root.

I then let the system power up and went through the semi-automated setup process.

When it was all done after a few reboots I checked the system.

Sure enough, both the pagefile.sys and pagingfile.sys were present again and a check of the Registry hive value showed it had been reset to “pagingfile.sys”.

Grrrr.

Chapter Five: “Fresh Evidence”

I was pretty frustrated at this point.

I needed to capture data that would tell me just what was causing the registry key to be changed and when.  Only one tool came to mind; Process Monitor.

In addition to logging all file, disk, process, and registry actions, Process Monitor also has an option for boot-time logging.  I’ve used it before but had no idea if it would survive through the sysprep process.

So I reprepped the system one more time, copied Process Monitor to the root, fired it up and enabled boot-logging, sysprepped the system and shut it down.  Then I (again) off-line Win PE booted the system, mounted the SYSTEM hive off-line and reset the key back (again) to “pagefile.sys” and deleted the paging files.

I then brought the system back up and walked through the setup process again.

For some reason it went through the initial setup process just fine but when it got to the custom company post-configuration wizard, the wizard locked up.  (This was because I was rushing a bit out of frustration and didn’t bother uninstalling an application the wizard automatically deploys/installs locally as part of the post-setup process.) 

Because the system seemed “hung” I wasn’t able to get at my Process Monitor logging data.  Bummer.  However both page files were again present as well as the modified registry key.

So I went back again and this time, did not copy over the post-configuration wizard exe files knowing it would just toss a “file not found” error when Sysprep first-boot hit that section.  I rearmed Process Monitor, sysprepped the system, off-line fixed the registry key, deleted both of the page files, and rebooted.

This time when the dust settled, the page file was still just “pagefile.sys” and the registry key was preserved.  Process Monitor logs didn’t find any evidence of the change to the registry key.

Very curious.

Chapter Six: “A Suspect is Identified.”

The only thing I did differently was to not run the post-configuration wizard exe.

I figured this needed some investigation as though I knew what we “did” interacting with it, I clearly might not be seeing what was going on beneath the surface.

Right after the post-boot Sysprep process completes, a line in the sysprep.ini files calls to a custom folder we deploy.  In the folder are a number of MSI and EXE install files for some base applications.

The “master” program that calls those is the executable noted in the sysprep.ini file.  When ran it requests the technician input some system naming information, a hard-drive label name, what kind (if any) of third-party network client to load, and what domain it should be attached to. Pretty simple stuff and nothing to do with page files.

It would be nice if I could take a look at the executable’s code and inspect it.  However I’m not a programmer and definitely not experienced in reverse engineering executable files.

But for kicks I tossed the file at Universal Extractor on the off chance it might just be a fancy pants packed batch file of sorts.

That didn’t work too well but the error code suggested it was an AutoIt package.  Now that was interesting.

AutoIt allows creation of automated and advanced scripts which can be packed into an EXE file for deployment.  I’ve seen some great utilities in the past that use AutoIt to create their functionality.

How can I decompile it to get at the underlying script?

Chapter Seven: “The Inspector Gets Help from Didier”

Didier Stevens has two great (but older) posts exploring AutoIt executables and extraction of the au3 script file I was looking for.

I downloaded AutoIt and quickly found the Exe2Aut de-compiling utility included with it. However it would not handle the exe file I was working with.

Reading through the posts as well as extensively through the comments, I learned I might have a chance if I could find a hard-to-locate custom AutoIT package that could reverse decompile versions that Exe2Aut wouldn’t process.

The Google hunt began.

And I eventually hit pay-dirt:

This showed me I needed to add UPX: the Ultimate Packer for eXecutables to my toolkit so I could uncompress my AutoIt exe file. Quoting from obijuan’s post:

…first use the upx.exe provided by autoit (\autoit-v3.2.0.1\Aut2Exe\upx.exe) to decompress the .exe:

  • Open a command prompt and use upx.exe with the “-d” switch:
  • EXAMPLE: upx -d mytestfile.exe
  • More about UPX can be found here: http://en.wikipedia.org/wiki/UPX

This will decompress “mytestfile.exe” – it almost doubles in size.

But the links provid by obijuan to the AutoIt cracker were not working.  I did, however, now have a potential utility name “myAut2Exe” to aid in my search.

That led me to myAut2Exe, The Open Source AutoIT Script Decompiler 2.7 B.112 over at BootLand forums where there was a working RapidShare download link to the program package.  I also found and earlier version (2.4) /AutoIt Decompiler – Brad’s Projects – Trac.  And then at this location MyAut2Exe - Collaborative RCE Tool Library over at woodmann.com.  (I’m providing several in case any download links go dead.)

I unpacked the download, ran the main “myAut2Exe” AutoIt executable, dragged my (now upx –d ‘ed) suspect AutoIt exe file onto it and let it rip.

It managed to bypass the password protection, and unpack the file and offer a (semi) readable au3 file.  I say semi-readable as it didn’t seem to be a “pure” AutoIt script, but it had enough of the function calls I could inspect it in detail.

I opened it up with Notepad++ and began hunting.  It was very long and the content was very dense.  So I thought I might get lucky and ran a search for “pagingfile.sys”.

Sure enough, the suspect’s DNA was now hiding in plain sight under one of the function calls:

RegWrite("HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management", "PagingFiles", "REG_SZ", "C:\pagingfile.sys 0 0")

Chapter Eight: “The Trap is Sprung”

Every good Agatha Christie mystery usually ends up with the detective assembling all the participants and re-creating the crime to prove the solution.

So, for the final time, I reprepped the system, set Process Monitor to both enable boot event logging (as well as to not use the page file for logging but rather a custom file on the root so I didn’t lose data when the page-file got renamed, ensured the system was fully prepped and armed, including the post-config wizard folder/files, sysprepped the system and shut it down.  Then I off-line Win PE booted the system, mounted the SYSTEM hive off-line and reset the key back to “pagefile.sys” and deleted the paging files.

I then brought the system back up and walked through the setup process again, all the way through.

Finally, when it was completely done, including the post-boot, wizard-config process stage, I fired up Process Monitor, stopped the logging, and loaded up the captured log file.

I set the filter for the process name of the wizard, and then only for Registry Set values.

Bingo. Confirmed.  There was the wizard’s process changing the registry key to “pagingfile.sys”.

Turns out there actually was quite a bit more registry tweaking going on behind the scenes with the wizard and not just the visible environment/system/network entries the technicians were inputting.  Pretty interesting to find that out.

So, when a system is imaged, Sysprep does its usual pre-configuration system setups, the system reboots, and then sysprep hands the wrap-up off to the wizard, which does its things as well as changing the name of the Windows page file to a non-standard file name.

Mystery solved.

Epilogue

Some things still remain currently unresolved.

I’m not responsible for creating the wizard.  I’ve got a request for information from the group that might be responsible for it inquiring as to the purpose and/or reason for renaming the page file from the Windows default of “pagefile.sys”.  Not sure if it is a typo or some security feature.  Doesn’t make much sense from my perspective at the field level and the size is being left for the system to set automatically still.  However it is so unexpected a configuration “tweak” either it was accidentally copied into (retained) in the wizard if our custom AutoIt developer copied someone else’s original script elements into theirs while building it and didn’t realize (or notice) the impact, or it is for some really clever benefit.

Also, while the wizard changes the page file name, it doesn’t do anything to clean up or delete the old “pagefile.sys” one.  This results as noted in a now unused pagefile taking up over 1 GB of space on the drive.

The wizard does have great benefit and needs to remain in the system setup process.  However this particular behavior adds some headaches at the moment for my imaging building work.

I have to reset the Registry key (well maybe not but am doing so anyway) as well as off-line delete the “pagingfile.sys” file before doing my image captures to keep the image file down by that 1.6 GB size.

ImageX does skip certain files during the image capture process by default, pagefile.sys being one of them.

\$ntfs.log

\hiberfil.sys

\pagefile.sys

"\System Volume Information"

\RECYCLER

\Windows\CSC

I could take advantage of the ImageX Command-Line Options and create an ImageX Configuration File which will skip the “pagingfile.sys” one as well.

And this was cool to learn:

If you rename your Configuration_list.ini file to Wimscript.ini and store it in your ImageX directory (where the ImageX.exe file is located), it will automatically run when you run the /capture option, without requiring you to use the /config option.

Sweet.

However, while this helps me with the ImageX imaging, it doesn’t help with Clonezilla which would continue to capture that file.

So, until I can get clarification from the folks who wrote the custom AutoIt wizard on why it’s present (and maybe get it removed if not critical), looks like I will be manually removing the custom page file off-line before image capture unless I can figure out a way to include automatically cleaning/deleting it during the pre-shutdown sysprep process.  I’m less concerned with the registry correction as it doesn’t seem to hurt anything.

And I’m going to have to decide if I need to confuse our technicians even more by requesting them to delete the (now-unused) pagefile.sys file as part of their post-system image deployment setup steps (to gain back the 1.6 GB lost) or if it is more trouble than it is worth and just let the sleeping dog lay.

Now that was a puzzle worth investigating!

--Claus V.

Post Script:

While researching this issue I uncovered a tip on an annoying sysprep behavior I have noticed.  On some systems when I run sysprep, the process takes a few minutes or less to complete and shut down the system.  Other times sysprep will appear to run for over thirty-minutes before shutdown occurs.  I’ve not been able to find any rhyme or reason to this experience…nor apparently have many others.

Some say it has to do with the mass-storage controller section of the sysprep.ini file. Others say it has to do with rebuilding the driver stores on some systems.

While not really logical, at least one person suggested it is a known (but unacknowledged) bug in sysprep that can be prevented by moving the sysprep dialog boxes a bit while it is running:

Deployment Server Sysprep hanging – Egghead Cafe

Unfortunately, Windows XP has very little to no syprep logging.  I know of one bug only with sysprep that requires you to move, the sysprep dialog box and the hang will clear.

Amazingly, I’ve had success with this technique.  Again, it clearly doesn’t seem to be logical, and maybe I’m doing a rain-dance on a cloudy day, but hey, thought I would share…

--cv

Update: Never thought I would see the day…

Update: saw a link on the 10.18.2009 – TechBlog linkpost that validates what I was seeing in this post:

Looks like Thursday will be the day to hit the big-box stores for the very latest in computer system hardware/software thrills and chills. -- cv

On a beautifully cool and refreshing fall Saturday, Alvis and I ran some errands around town.

She needed to cobble together the components to make her Halloween costume; a Tour Guide Barbie (from Toy Story 2).  Her other high-school friends are also adopting the characters of Toy Story.  Ought to be fun.

Since we were in the neighborhood and neither of us was quite ready to head home, we dropped in to Best Buy to look for an Enchanted DVD (found), and see what other goodies were to be had.

While I was there price comparing a 2.5” IDE 350 GB laptop drive ($110) against the NewEgg one I’ve been eyeing ($90) for a pre-Win7 upgrade, I was really blown away by what I didn’t see.

Almost no laptops or PC’s were available or on display.

Normally when I have been in Best Buy, just about every horizontal surface in the computer area is covered by laptops or desktop.  Today?  Only four netbooks and two full size laptops.  I saw only two desktop systems.

When we dropped in to Office Depot it was even worse.  Two laptops and no desktops on display.

Is the economy that bad that no-one even wants to carry this merchandise on the shelves?  It was really funny seeing how BB was trying to hide the missing items with all kinds of mice, USB sticks and other small-box products artfully arranged on all the surfaces.

No.

I suspect that all these (Vista) systems have been pulled and I am only observing the lull before the Windows 7 release storm.

I imagine that by next weekend the shelves will be full again, but this time with newly deployed laptops and desktops armed with Windows 7 operating systems.  They are probably already in the back waiting deployment.  No doubt the crack team of sales specialists have been hard at work training on the finer selling points for Windows 7.

It was just kinda freaky to walk into these spaces and not see hardly any computers this go round.

Really freaky.

--Claus V.

Sunday, October 11, 2009

Mostly for the Forensics Crew: A rapid-fire linkfest

Here’s a collection of links that is 90% + aimed at the Windows forensics crowd.

Got to drop off daughter for pre-Sunday PM service activities and get some groceries bought before it gets too late!

  • ChromeForensics v1.0.1 : woanware. Mark Woan had kindly supported my fumbling foray into ChromeForensics a few weeks ago.  We ended up working my knucklehead-ed-ness out and he ended up updating this with a nice Help file.  I checked it closely and made a suggestion that he added in.  He graciously gave me unneeded credit for the tip.
  • Windows Incident Response: Linkity-Link. Harlan Carvey’s nice well-rounded linkpost regarding some assorted forensic topics including a tease on WiFi geolocation in forensics.
  • Forensics from the sausage factory: Windows Photo Gallery. – In that post, Harlan pointed to this fascinating post by DC1473 on forensics clues from Windows Photo Gallery usage.
  • Windows Incident Response: Where was Waldo?. Then Harlan later came back with an amazingly neat follow-up post on WiFi geolocation and forensic bits extracted from the Registry.  This is really cool stuff and even sysadmins may find it useful.  Suppose you have a policy against WiFi usage of work laptops/systems.  During a system audit you could use RegRipper to discover WiFi connections as well as possible connection point history.  Using Harlan’s technique, you might also be able to discover where it was used at (home, work, public library, etc.).  Not only does this provide great data for the analysis, but it could provide context for system activity observed as well expand the information available for the response.  Really neat stuff.
  • CDP - What Switch Am I Connected To? and Monitoring Traffic with Span Ports – SynJunkie.  Two really great posts out of series of ones touching on network monitoring, and Cisco switch/router configuration techniques.  I’m singling these out in particular as they are of interest to sysadmin troubleshooting on the network as well as traffic captures.
  • Forensically interesting spots in the Windows 7, Vista and XP file system and registry (and anti-forensics). IronGeek.  Useful list of Registry locations worth taking a look into, as well as some background info on them.  Though not nearly as complete as Windows Forensic Analysis DVD Toolkit, Second Edition.
  • JADsoftware’s Internet Evidence Finder. Updated to version 2.0.4.  Change Log
  • Windows Incident Response: Hakin9 articles.  Harlan goes on in this new post to discuss some timeline creation and analysis thoughts.  This is an ongoing theme on his WindowsIR blog.  I recently had to construct just such a thing and am coming to appreciate the issues facing those needing to present highly detailed technical information on incident response in a manner that doesn’t cause non-technical managers’ eyes to glaze over and miss the impact of the information presented.
  • Disk2vhd. Sysinternals has just released a new freeware tool.  This utility could be of great benefit to both sysadmins as well as forensics folks.  I use Virtual PC as my preferred platform for virtualization and while there are many tools that will convert a system image to VMWare machine, this could be a great tool for doing a similar thing for VPC. From the description: “Disk2vhd is a utility that creates VHD (Virtual Hard Disk - Microsoft’s Virtual Machine disk format) versions of physical disks for use in Microsoft Virtual PC or Microsoft Hyper-V virtual machines (VMs). The difference between Disk2vhd and other physical-to-virtual tools is that you can run Disk2vhd on a system that’s online. Disk2vhd uses Windows’ Volume Snapshot capability, introduced in Windows XP, to create consistent point-in-time snapshots of the volumes you want to include in a conversion. You can even have Disk2vhd create the VHDs on local volumes, even ones being converted (though performance is better when the VHD is on a disk different than ones being converted).”
  • 8 bits: Lab FTK Imager: file carving using the MFT.  Neat technique.  I know I’ve got a few utilities that can locate the file location based on a section of sector info, but I need to dig those up again for a refresher.
  • Windd 1.3 Final! (x86 and x64) - Matthieu Suiche’s blog !.  Get it!
  • Beta version of NirLauncher package is available to download. Nir Sofer has released a new tool of his that allows downloading and launching of his tools as a package-manager.  Really cool and neat.  Similar to KLS SOFT’s - WSCC - Windows System Control Center.
  • This Is a Photoshop and It Blew My Mind - Photosketch - Gizmodo.   Not forensics related but clever.  Do a stick-figure sketch of a image scene and feed it to Photosketch.  It will then find the different images and mash them up into a single image it creates/renders.  How cool is this!
  • .PhotoFilmStrip.  A freeware utility that allows you to create “Ken Burns” style panning in/out/across of still images into a video format.  Really neat.  Spotted and reviewed over at this freewaregenius post.

Finally, I really like using Universal Extractor to unpack setup files and examine them for no-install operation.  However, from time to time I encounter some packers that it can’t handle.  Usually related to newer versions of the compression software used.  I recently ran into just that issue with an Inno Setup package.  Fortunately, I just had to go over to innounp, the Inno Setup Unpacker and download this newer version, copying the files into the Universal Extractor folder and overwriting the older ones.  Unpacking working perfectly again.

Special hat-tip of gratitude to Miles over at his TinyApps.Org Blog.  He has been kindly encouraging me behind the scenes on the back-channels in my recent un-plugged state and also tossing me some of the links noted above that I might have missed with the several hundreds of RSS feeds that had accumulated in my feed-reader that I had to cull through to get caught up.  He has had a number of interesting posts of his own lately including Unixy goodness: command compendiums, dd, acronym origins, and a shell stopwatch, dd block size, Wipe MBR / Track 0, and SFK.  If you aren’t RSS feeding TinyApps blog, you need to be or you are missing out! 

Cheers!

Claus V.

First Fatal KSOD on Vista

To summarize:

XP Home on our SFF Shuttle desktop system. Vista Home Premium (x32) on both our laptops. Win 7 RC Ultimate (x64) (VHD booted) on both of those same laptops. Win 7 RC Ultimate on Alvis’s laptop (native install).

The XP system has seen a few BSOD’s over the years, but always I have been able to recover the system and get past it.

The Vista systems have seen more than a few BSOD over the past year.  Almost exclusively they have been related to video-driver updates.  There have also been a handful of times that I thought I was getting one of Vista’s blacK Screens Of Death (KSOD) after reboots from Windows Updates.  However in the first case, judicious use of the System Restore point or “Last Known Good Configuration” has always saved my bacon.  In the latter case, patience was the key and after leaving it on the black screen with the cursor on reboot eventually (up to 30 min sometimes) the system eventually completed it’s updating and progressed on normally.

I’ve never seen any KSOD or BSOD on either of the Win 7 systems during booting or as a crash. Ever.

So when little bro texted me earlier last week saying he was rolling in from the Red Baton with his Dell XPS system stuck on a KSOD I couldn’t help but get a little excited.

Yesterday, from 3:30 to 8:00 PM we watched football, had guy-talk, ate Sonic, and had a grand ole time as I attempted to get the system and his data safely back in operation.  Oh yeah, Mom and her four-legged white-shag carpet hung out as well enjoying her sons’ banter.

Digger had already done all the right things up to that point to attempt to restore the Vista system to good graces.  Unfortunately it just wasn’t cooperative.

The patient was a still-young Dell XPS system, quad Intel cores, 4 GB RAM, with two appx 500 GB HDD’s.

It was last seen functioning fine on Wed. when he returned it was locked up and rebooting into Vista brought up the green loading bard which progressed to a black screen with no cursor movement.  Leaving in this state for hours brought no changes.

So I sat down and evaluated the situation with him.

It wasn’t a boot-loader issue as he could successfully get to the Vista loading process displays.  And, since I had set it up to dual-boot Win 7 RC (x64) via VHD earlier in the year, that system was still loading with no complaints..allowing him to function just fine (though his user-data was still in the Vista system).

It saw that Vista system as a “D” drive but was “inaccessible” due to not setting security permissions to allow him access to it via Win 7.  Certainly we could have done that (as I have done on our own dual-boot laptops) but I was still in the initial troubleshooting stages and didn’t want to complicate things.

I had brought over with me my custom Win PE 3.0 bootable USB stick so I booted the system with it, and then just copied his Vista “user” folder over to his secondary hard-drive to “rescue” his personal files, music, etc. Just over 130 GB in data.  Sheesh!

With that safely tucked away, were were now good to proceed with attempting to get the main system going again.

Based on his trouble description and what I was seeing, I was suspecting that he was encountering some kind of driver loading error.  Maybe an update was recently applied that killed it during the loading process, or some kind of service was failing to load.

I tried to use Nir Sofer’s Drop-Dead-Quick Blue Screen of Death Diagnosis Utility but unfortunately, though bro is a geek, he hadn’t configured his system to save crash-dumps automatically.  So even though he had briefly seen a BSOD flash once during one reboot attempt, we couldn’t access any clues from it.

The various safe-mode boot options didn’t help, they also died on the KSOD with an unmovable cursor.

Last Known Good Configuration didn’t help at all either.  Same thing.  Nor did booting to a command-prompt.

I had a Vista recovery disk burned (as did bro.) so I re-did those options.  (See also Creating a Windows Vista Recovery CD - TechRepublic.com.)  The “Startup Repair” option seemed helpful but I learned (and he had earlier found out) no problems were detected.  “System Restore” was unhelpful as he had also not set his system to enable automatic/periodic System Restore Point creations.  Bummer.

So running out of our “usual” options for repair, I turned next to the tips in this post shared by BackRoomTech Julie.

That post (and the subsequently updated source post, REVISED: How to fix the Vista KSOD (blacK Screen Of Death) | LogBlog) required some geek-fu.  Again turning to my Win PE 3.0 boot stick, I mounted the Registry Hive, checked the recommended key, but alas, it was set to what it should have been.

In the comments I did find several other neat tips, such as pressing the left-shift key five or so times quickly to enable the “sticky-keys” dialog to come up.

- After boot up, wait a few minutes at the KSOD until disk activity stops
- Shift key five times, or hold shift for 8 seconds, until Sticky/Filter keys pops up
- Say ‘Yes’
- Configuration menu for sticky/filter keys comes up. Under the Help menu, choose “Is this version of Windows legal”
- Internet Explorer is launched. In the address bar, enter C:\Windows\explorer.exe

From there (if you can get there) you can then attempt to run msconfig to disable your startup options.

Some folks got lucky with this approach, but in our case either Sticky Keys was disabled or the system just wouldn’t get to a point where we could activate it.

There were also a few comments that said if you renamed/deleted the Logs folder (’Go to C:\Windows\System32\Winevt\ and rename your Logs folder to Logs_Bad, and make a new Logs folder’) it might help resolve a KSOD error.  No go.

Most of these tips are lined out in detail here in this very fine post:

Truth be told, there are many, many reasons why a Vista system can KSOD. In some cases restoration is possible, but in most (particularly with no previous System Restore Points available) success may be limited.

So with all known avenues exhausted, it was time to flip the switch and reload Vista.

Bro had brought along his Dell OEM System Restore disk.  We popped it in and began the “Fresh Install” process.  About 45 minutes later it was completed.

The following were known or unexpected treats I discovered:

  1. Because the previous Windows system was still present, it automatically found and applied his original OEM Vista key.  No reactivation was required.
  2. The old Windows folder was renamed to Windows.OLD. This will be beneficial later.
  3. The custom bootmgr wasn’t modified by this process. (referring to our dual-boot with VHD mod). After the new Vista system was installed, we could still boot to the Win 7 VHD system with no impact! Nice.

When we got his Vista system through the initial profile/system setup, we looked for updated and found almost 70+ were required.  I also found that the OEM disk set it up as SP1 level.  Instead of doing all those updates, then putting SP2 on the system, I tried to just download SP2 directly.

And found that I am spoiled.

Mom’s home (where we were at) is equipped with DSL broadband.  I have cable broadband at home.  A Vista SP2 download would normally take me anywhere from five to ten minutes at home.  At mom’s, the download was reporting a seven-hour download time.  Egads!

I had brought a lot of files with me but not that one.  On the other hand, the job wouldn’t be done (in my book) until the system was fully patched and updated.  Even though bro. could take care of that himself back in the Red Baton.

However, just maybe…just maybe…we might be set.

Knowing that Windows almost always keeps a copy of the downloaded update packages on file, I did a quick search for the Vista SP2 setup file in the Windows.OLD folders using the wildcard *KB948465*.  Sure enough, there it was!  I copied it to a fresh “temp” folder and in no-time flat had his Vista system updated to SP2.  A few more update checks got the remaining hardware/software updates Windows needed.

Done.

We may never know what caused the KSOD on his system.  He still has a few key programs to reload back at his place, but only a few as he is now committed to waiting for the public release of Windows 7 in the next few weeks.  Then he can pave Vista and get a fresh load of Windows 7…which he seems to be much more impressed with anyway.

And no, Windows 7 is not immune from the KSOD either.

While researching a solution to the Vista issue, I stumbled on this announcement that came out related to Windows 7 just last week.

In hind-sight, the only other thing I should have done was to toss Harlan Carvey’s RegRipper or Mark Woan’s RegExtract at the hive files.  They weren’t corrupted and I was able to “off-line mount” each of them in regedit while troubleshooting. While this certainly wasn’t a forensics response, the system-data that they could rip out of the registry might have provided me the needed clues to determine what had been going on at the point of initial system failure.  So sysadmins, don’t forget the benefit of these tools for system troubleshooting and troubleshooting information gathering. Be familiar with them.  I’m going to ask bro. to extract these original hive files for me (or do it over a remote-wire connection) and analyze the results…maybe later this week.

Here’s looking at you, KSOD.

Claus V.

Saturday, October 10, 2009

Fixing a fun little problem: FF to IE bookmark import

For a number of not so complex reasons, I decided to try rolling back to the dark-side at work and migrate back from Firefox to Internet Explorer as my primary web-browser there..

While I did use IE (version 8) periodically and kept a smattering of “Favorites” bookmarks, it wasn’t my daily browser.  That was reserved for Firefox.

In Firefox I had amassed quite a large and sophisticated collection of bookmarks in a highly organized folder structure.

So naturally I wanted to bring them over into IE.

So from the file-menu in Firefox I just did the “Bookmarks” –> “Organize Bookmarks” –> “Import and Backup” –> “Export HTML” routine.

That generated a “bookmarks.html” file.  Easy peasy.

At first I “cheated” and just opened that file in IE and make a Favorites link to it on my favorites bar.  When clicked, it opened the list as a single HTML formatted page and I would just scroll down to find the link I was looking for. This worked well enough but there were a lot of links and a lot of scrolling.

I eventually decided it was time to import them directly into IE as “real” folders/favorites.

So in IE I created a new “FF Imported” folder and then went through the “File” –> “Import and Export” –> “Import from a File” –> “Favorites” –> and selected the bookmarks.html file Firefox exported.

In the past I’ve always had a 100% success rate and didn’t expect any issues.

However this time it ran for a few seconds, then generated a failure error.  What?  Never had that issue before.

Looking in my importation folder I found that some of the bookmarks/folders had imported but only about 10% or so.  Strange.

So I went to delete the folder and start again.

Only one Favorites folder wouldn’t budge.  No matter which of multiple techniques I attempted to ply.

It was named “Holding..” and had nothing (really, nothing) inside it.  It was actually a subfolder but I couldn’t delete the parent folder either.

Eventually I decided that the double-periods might be mucking things up and somehow making it be an “invalid” folder name thus giving the system fits dealing with it.  I suppose I could have just left it but that seemed messy.

Luckily I remembered an old GSD post where I had mentioned Delete FXP Files (free-edition) -

This fantastically clever utility from JRTwine Software continues to amaze me. It isn't something most sysadmins will regularly need. See, every now and then you may come across a file or directory that somehow got named something that Windows just won't let you delete. It's not that it is "locked" per-se in the normal sense, but that the name itself makes Windows balk and your deletion request. I highly recommend this tool and suggest you keep it handy, just in case.

So I popped over there and downloaded the old version, installed it, and in seconds, had the offending folder cleanly and decisively removed. No fuss.  Delete FXP Files remains a fantastic tool that is probably almost never needed; but when you do, you really will be glad to have it handy.  Note, I also found that there is a much enhanced Delete FXP Files 2009 version out now as well.  Check it out!  Well worth the value purchase price…

Anyway, that fixed my initial headache, but didn’t bring me any closer to figuring out the importation failure.

But I did have a clue.  Somehow in the importation process, a folder had been created with an “invalid” folder name.  Did maybe other shenanigans lurk in my bookmarks.html file?

Not wanting to repeat the invalid folder name fiasco on my next attempt, I opened up the bookmarks.html file in Notepad++ and looking through the HTML code, quickly found the code entry that created the “Holding..” folder name.

Despite the mean “DO NOT EDIT!” comment at the top of the file text, I felt fairly confident changing the hyperlink text name reference so that the folder was now named “Holding” without the double-periods.

I also knew I had a number of other bookmarks in there that didn’t carry any text at all.  In Firefox they were on my bookmarks toolbar identified as “favicons” to cut down on space.  So I edited them as well to have a brief text name now.

The more I got to looking through the remaining exported Firefox bookmark code, the more bookmarks I found in HTML code that had names with potentially/really invalid file/folder name characters, or special characters, etc.  So I ended up editing them all to have more simplified (and allowed) names.  Took quite a while.

When I was finally done, I went back and attempted to import the now-modified bookmarks.html file into IE.

This time it imported perfectly with no errors!

Hurrah!

So the problem and solution was quite clear…but why might this have occurred?

I think it very simple.

Version 3.x builds of Firefox use JSON/SQLite based databases to manage the new Places bookmarking structure.  As such, it really doesn’t seem to care what you name a bookmark or what characters you use to name it.  Spaces, slashes, combos of symbols, it all seems pretty much valid.  Even no-named bookmarks (icon only) are allowed.

The “problem” lies in the IE method of “Favorites” bookmark management.  Instead of using a database format to store them, pop into the user account folder and look for the “Favorites” folder.  Yep. Same old Windows file/folder structure.  Bookmark folders are really folders, and the bookmarks are actually files with “name.url” format.  The Registry helps control how these are ordered/displayed in the IE browser, but while great to know, that is not directly related to this particular issue.  (more details of the IE Bookmarks/Registry connection here: CodeProject: Internet Explorer Favorites) from Jean-Paul Mikkers.)

Sidenote: Could this provide a possible forensics angle?   Might be good for registry analysis folks…I’m curious if the registry entries might temporarily survive if say someone attempted to delete their user profile “Favorites” folder/contents to hide bookmarks or used a IE “washer” application.  Would these Registry items survive as remnants if IE hadn’t been re-launched and the keys/values hadn’t had a chance to be rebuilt?  Would they remain for a while in a backup registry file or restore point file?

Anyway, as such, during the importation process, it reads the bookmark.html file, strips out just the required name/link bits to recreate the folders and bookmark structure.  If any of those folders/links are names something invalid, it might successfully create a folder so named (but unable to be renamed/moved/deleted by the system) or outright crash/error-out during the importation process.

There are a few third-party tools that offer to export/import bookmarks between different browsers (Transmute pops first into my mind).  I guess I could have tried one in this application, but it seemed at first blush to have been a very simple thing to do.

Now I know better and hopefully you will as well.

Like I said…a fun exercise.

(And rest assured, Firefox remains my preferred/recommended daily web-browser.  Like I said, this was just a work-thing exercise.)

Cheers!

Claus V.

Monday, October 05, 2009

Keep Alive Ping

Yep. Still here. All is well.

I came down with quite a cold-rebound two weeks ago.  Lavie hauled me in to our family doctor and I ended up getting loaded down with a mess of antibiotics, nasal sprays, etc. Did a number on my system to get it fixed up again.  Go figure.

I was pretty drained and ended up unplugging and getting some much needed rest.

Been spending extra time visiting with the extended family and watching cartoons and college football on the weekends with Lavie and Alvis.

I think it fell at a relatively quiet time on the Webs as when I finally got around to popping my RSS feeds I didn’t see too much of remarkable interest.

Rest assured, I still have some stuff in the blog hopper.  I might have time this weekend to get one or two posts up.

Cheers!

Claus V.