Sunday, March 13, 2011

Quick Tip: Fatal Error C0000034 installing Windows 7 SP1

Got to the church-house early this morning to bring up the systems we use to run the presentation and lighting-control software.

Booted the Windows 7 x64 system up.

Was surprised to see this error during the boot up process:

Fatal Error C0000034 applying update operation 282 of 117183….

Oh Noes…Really?  A few hours before before services? Nice.

Head fogged from DST “Spring Forward” madness, I set to work with the rest of the technical crew watching (and Mr. D kindly bringing me a fresh styro of Joe).

A reboot didn’t help.

I brought up the sister-system (also Win 7 x64), crossing my fingers and hoping for the best.  Fortunately it booted fine.

Some quick Google work off of it quickly found a lot of additional material on the webs about others encountering this issue.

Funny thing.  I’ve upgraded both our home Win 7 x64 systems as well as our Win 7 x32 system with SP1 and had no issues.  Nor have I heard (in the tech news) of any major issues with the Win 7 SP1 upgrade…but suddenly I felt like I turned over a rock and discovered a major creepy-crawly!

Once I had done some research and felt I had a good plan of solution, I set to work:

  1. Not having my USB-based “off-line” boot drive with me left me at a disadvantage.  I dashed out the house too quickly this morning and left it on the mantle.  Bother.  What I did have was a working Win 7 system hooked to our sound-board system.  In other words a functioning system with a optical media burner and a ton of blank CD/DVD media. Score.
  2. I hopped over to NeoSmart and their Download Windows 7 System Recovery Discs page.  I then downloaded the x64 version of the Win 7 Recovery disk ISO file and burned it to CD.
  3. I rebooted the borked system with the disk and dropped into the CMD line option.
  4. Following this Windows Servicing Guy post by Joseph Conway (Senior Support Escalation Engineer Microsoft Enterprise Platforms Support), I manually loaded the main system’s off-line “System” registry hive file, dug down to the indicated reg-key and cleared it as instructed.   Unloaded the hive and rebooted.
  5. System booted up (after rolling back the SP1 install) OK with no apparent damage done.
  6. Immediately upon deciding the system was operating stabling, created a manual System Restore point on all our machines (even the working one!).

The services went off without a hitch and no-one but us “back-desk-pew geeks” knew this mornings pre-service preparations were much more exciting than normal!

Oorah!

Important Notes and Observations:

Post Update #1: Found an amazing post from Günter Born that goes into awesome detail with various solution options, outstanding details and helpful screen-grabs, and even some technical root cause analysis thoughts.  Only problem is that his blog/post is in German so probably, what, 99% of the US may not ever discover this Günter’s amazing work and help with this issues (lots of supporting Links also!).  Too bad.  Google Translate version here and it handles it pretty well. Windows IT guys and gals shouldn’t have any issue following it despite a few auto-translation oddities.  Original page: SP1-Installation hängt, Error C0000034/C000009A - Born’s Windows IT Blog.   Maybe also useful from Günter: Buglist's collateral damage by Service Pack 1 (Google Translate version offered) original language page link.   Actually, Günter’s site is very amazing with his detail in trouble issues noted. I’m going to be keeping an eye on this blog for a while to come! Born’s Windows IT Blog.  Additional recent helpful tips/notes from Günter below (all linked via Google Translate service):

Post Update #2: Back at the church-house again this afternoon. Original system was running fine so turned my attention to the second one.  When it shut down this morning it did apply 4 pending updates. Apparently Win 7 SP1 was indeed one of them.  When I brought it up again this afternoon, it also failed with the exact same Fatal Error C0000034 applying update operation 282 of… issue.  Hmmm. Interesting.  What’s more, the manually created system-restore I specifically did this morning on it was no-where to be found. That’s serious.  I again had to revert to the same solution I previously mentioned.  Worked fine.  System recovered and it claimed (as did the first) to have rolled back the SP 1.  I rebooted and it came up fine.  I then had downloaded the SP 1 package file and tried to put it on that one.  Curiously, it would not install saying there were missing components. I tried again but no dice. When I went to the System window via Control Panel, it does claim to be running at SP 1 level.  However when I check “Programs and Features” and check the Microsoft updates listing carefully, I don’t see it listed anywhere.  So now I am left in a conundrum.  They system thinks SP 1 is installed, I don’t find it actually listed as installed, and a manual download and install attempt of SP 1 fails as it is missing required components.   This is looking a bit more dire.  I really, really hope MS gets to the bottom of these issues very soon and offers some kind of roll-back/repair cleanup fix.  I’m really not looking forward to rebuilding these systems.

First things first.  I don’t ever do updates on our key production systems before services for just this reason.  However, I came to find out that the update was pushed via AD settings earlier this past week and the person on the system at the time just walked away from it at day’s end without validating it took on the reboot.

I really should have remembered to grab my USB-boot “offline" drive.  I rarely leave home without it for just this reason.  I just lucked out that the 2nd system didn’t also crash and I had both internet access and a CD-burner to make the rescue disk in the pinch.  The disk is now safely taped to the side of the case for future reference.

Fatal Error C0000034 applying update operation was a KB article under Vista. Now it has had Win 7 added to it as well; MS KB Article ID: 975484  - Your computer may freeze or restart to a black screen that has a "0xc0000034" error message after you install Windows 7 Service Pack 1 or a Windows Vista service pack

I’m still not sure I fully understand the root-cause of this error.  There is a lot of speculation in the forums at the moment.  I did discover I am not the only one who had the error happen on update operation “282”; Fatal Error C0000034 installing Windows 7 SP1 - Gary Davis’ Blog.   Coincidence?  Our two production-systems are high-end Dell Inspiron desktops; one took the SP 1 fine and the other did not.

x64 -bit Win7 systems seem to be succumbing the most to the issue, but there could be x-32-bit Win7 systems also impacted.

There are at least three primary solutions I uncovered that other smarter folks have previously worked out.  I reviewed them all carefully before implementing one.

Joseph Conway of Microsoft offers two as does the MS KB Article 975484 I linked earlier.

Error 0xC0000034 during Service Pack 1 installations for Windows 7 and Windows 2008 R2 - The Windows Servicing Guy

The first is to simply attempt to roll back to a previous “System Restore” point.  That’s usually a safe bet, however in my case, although System Restore was set to “on”, there were no System Restore points found on the impacted system.  Some others also report finding this to be true also.

The second is to (via CLI or GUI) remove a specific registry key value from the PC’s SYSTEM hive.  That worked for me.

Critical Tip!  If you follow Jeff or the MS KB’s steps after having “off-line” booted the impacted system with a Win 7 System Recovery disk (like I did rather than directly off the ailing Win 7 System Restore boot process) you have to get your drive letter bearings in your brain set first.  It will use a RAM-drive “X” for the running recovery system.  However (in my case at least) the C: was actually referring to the Rescue disk and the D: was actually my “real” system’s “C:” volume.  Confused?  I was at first.

See the instructions (Joseph’s are clearer) talk about navigating/loading items from the C: (your local system volume).  But if you are off-line booting, then that may not necessarily be correct.  In my particular case, I ended up having to navigate and load the SYSTEM hive from the D:\Windows\System32\config\ location.

If neither of those work, I also found mention of a third solution in a Windows TechNet forum: Windows 7 Ultimate SP1 installation fails with error code c0000034 posted by “thiswoot”.

Basically it involves restarting the system, waiting for it to time-out on the fail and go into a system-recovery routine.  Log in and hunt down a very specific pending.xml file, finding and cleaning some specific lines out, resaving the file, then restarting again.

It is clever and appears to be a home-brewed solution before it was clear that MS had a preferred KB solution and the MS blog guys started posting their solutions. 

I was going to do it first until I kept reading the follow-on thread posts and eventually found Jeff Hughes’ Ask the Core Team blog re-post of Joseph Conway’s solutions.

Joseph Conway then then did a follow-on post Why you don’t want to edit your pending.xml to resolve 0xC0000034 issues - The Windows Servicing Guy blog.  You may want to read it first before proceeding with that method.  It’s not that Joseph isn’t saying it won’t work and don’t do it under any circumstance. He is just adding additional background info so you can know the consequences of getting your system going with that solution pathway.

That said, if the first two “official” solutions don’t work, and you (like many other Win 7 admins and users) are desperate to get the system up and going, it does appear to have a high success ratio.

There seems to be some regularity in various comment threads that the issue could be linked to using WSUS to push out the Windows 7 SP1 to systems, coupled with the end-user choosing the “install downloaded-updates/Shutdown” option when they log off.  That’s not a certain thing.  I suspect Joseph Conway and the MS guys are still working on the true root-cause identification.  I’d recommend keeping an eye in the rolling comments of this post of his as he is responding to comments very kindly and actively.

As for getting Windows 7 SP 1 successfully on the system post-C0000043 failure?  I'm going to (mid-week) download the Win7 SP1 standalone installer file and give it a try: Windows 7 and Windows Server 2008 R2 Service Pack 1 (KB976932).  Most reports are that this seems to work OK on a re-load.

Cheers.

--Claus V.

Posted by Claus at 3 comments
Labels: , , ,

Saturday, March 05, 2011

Self-Installing Xplico in Ubuntu - Virtual Edition

image

Above Image…the Xplico baby is delivered and working perfectly!

In my previous Xplico post, I mentioned how I had been using the VirtualBox images of Xplico.  And how suddenly they had stopped working.

Having been using this tool for a while, the sudden loss of this resource was frustrating.

In the end I sought to create my own self-built version so I could have a running version in my own VirtualBox session/image.

Plan A - Good Theory, Difficult Implementation

My original plan was quite simple.  (Warning: Linux-noobie stumblings ahead!)

  1. Create a  8 GB dynamic VirtualBox vmdk file.
  2. Find a Debian-based LiveCD that included a local installer.
  3. Load the vmdk file using the LiveCD to boot it.
  4. Install the Debian OS.
  5. Install Xplico
  6. Celebrate.

In theory this should have worked fine.

I had no challenges making the vmdk file.

I picked out PureOS and Linux Mint Debian LiveCD’s as my platform sources. Downloaded both and went with Mint.

I booted the vmdk file and installed Mint. No issues besides having to do some gparted work on the volume and some formatting of the partition. No biggie.

Then I set about doing the Xplico installation.  The Xplico developers have done a great job with providing the documentation on their Xplico-Wiki:

Install Xplico

So it should have been a piece of cake. Right?

Unfortunately, despite all my Step by Step attempts, I couldn’t apt-get a version of libmysqlclient16-dev.  And even though I continued on bravely anyway, stuff just started falling apart.

So after a few hours of work last night struggling through--and at least another hour of research--I found an alternative Xplico-installation method offered and decided to get some zzz’s and start fresh in the morning.

Plan B - Can it be this easy?

My new plan was realistically simple.

  1. Create a  8 GB dynamic VirtualBox vmdk file.
  2. Download Ubuntu Desktop Version 10.10 (it has a local installer).
  3. Load the vmdk file using the LiveCD to boot it.
  4. Install Ubuntu.
  5. Install Xplico via a pre-crafted script I had discovered in a forum.
  6. Celebrate.

And it worked!

The GSD Xplico Recipe

Here’s the Haps!

After much research from the night before, and realizing that the “official” Xplico VirtualBox images were based on Ubuntu, that seemed the way to go rather than my first choices.

Note this assumes some moderate familiarity with VirtualBox and Linux.  I’m leaving some of the details out that seem straight-forward (to me)…YMMV.

  1. Download VirtualBox if you haven’t already done so.  At the time of this post I used 4.0.4. Install accordingly.
  2. Launch and create a new virtual machine using the wizard.  Give it a  name, for the OS type pick “Linux” and for version pick “Ubuntu”. Pick your base memory size.  For my host system I’ve got lots of RAM so I went with 1024MB but you could use the default 512MB.  I kept the Boot Hard Disk option checked and allowed it to create a new hard disk at 8 GB. Since space is still a premium, even with a 500GB local hard drive, I went with the Dynamically expanding storage disk option. I took the default location, confirmed the size and hit “Finish”.  Done.
  3. Next I downloaded Download Ubuntu Desktop Edition 10.10 x32 bit version of the LiveCD.
  4. Once done I modified by virtual machine storage settings for the CD to point to the ISO I just downloaded and then launched the virtual machine.
  5. Once Ubuntu booted I just clicked the large “Install Ubuntu” button offered.
  6. I decided to go with all the defaults, including downloading of updates while installing as well as installing all third-party software packages offered. I took the default to let the installer erase and use the entire disk automatically (look ma! No manual gparted work!).
  7. While the installation went on in the background I continued with the localization setup and profile setup.  I decided to name my build GSD-Xplico and use “xplico” for both the name and password (to mirror the default account in the Xplico app) for simplicity.
  8. Hang out and chill for a while (or get started make an Old Bay Gulf-Coast pot-boil for dinner) as the installation/updating process completes. Yummers.
  9. When done, reboot as requested by the installer (don’t forget to disassociate the attached ISO LiveCD/Installer first!).
  10. Log in using the credentials you created in step 7.
  11. Optional but recommended.  Go ahead and install the VirtualBox Guest Additions.  I’m assuming most folks still here should be able to handle knowing how to do that. This will help a number of things but most of all will allow you a few more screen resolution size options.
  12. Optional but recommended. When prompted by the Update Manager, go ahead and install all available updates offered. At the time of this post, I found 275 updates offered.
  13. When done, reboot.
  14. Log in again and open up Firefox.
  15. Now for the secret sauce.
  16. Browse to http://5ff1cwepqm.tal.ki/20101216/wicd-xplico-261923/
  17. In that GnackTrack forum, commenter blaksark posted the following Xplico Script installation by Nsark.  All honor and credit ascribed accordingly.

    sudo apt-get update && sudo apt-get install -y gdebi sed && wget http://sourceforge.net/projects/xplico/files/Xplico%20versions/version%200.6.1/xplico_0.6.1_i386.deb && sudo gdebi -n xplico* && sudo find /etc/php5/apache2/php.ini -exec sed -i.bak 's/post_max_size = 8M/post_max_size = 800M/g; s/upload_max_filesize = 2M/upload_max_filesize = 400M/g' {} \; && sudo service apache2 restart && sudo service xplico restart && firefox localhost:9876
  18. Copy that script to the clipboard.
  19. Open “Applications” --> “Terminal” from the top menu bar.
  20. Paste the copied script.
  21. Press “Enter”
  22. Provide the prompt your password.
  23. Watch Nsark’s magic run for a bit. Basically it is getting all the dependencies, all the packages, installing them, then adjusting the apache settings to allow for larger PCAP file size uploads, restarting apache and the xplico service, and finally launching Firefox to the Xplico web-page.  Brilliant!
  24. When completed, close the terminal window.
  25. Behold, a wonderfully installed version of Xplico!
  26. You may want to set the Xplico Web Interface page as your Firefox homepage.  http://localhost:9876/users/login
  27. Default Username = xplico
  28. Default Password = xplico
  29. Admin Username = admin
  30. Admin Password = xplico
  31. Tips…you will want to use the default sets above for general PCAP work and Analysis. Use the Admin account to change some variables, user accounts, and configuration settings.  Most mere mortals probably won’t need to fiddle with these at all. 
  32. Adjust Ubuntu theme/wallpaper accordingly for attitude and coolness factor as needed.  I personally kept the default “Ambiance” theme but changed the wallpaper to the included orange feather on the grey background.  Seemed to match the Xplico Web-page interface colors nicely.  If you have already resized the virtual screen size to as large as you can but still feel a bit jammed up in the Xplico web-interface, you can also adjust the zoom size in Firefox to be a bit smaller to get more on without having to fiddle with the scroll bars.

That’s pretty much it!  You’ve just built your own lab for processing PCAP files.  Sure it doesn’t have all the extra cool pen/sec/for tools and apps that DEFT LiveCD comes with, but hey! it works and you built it yourself! And with some more work, you can download additional network/security packages as needed.

If you can’t wait, download, unpack, and upload Sample captures from the Xplico Wiki site.

I’ll go into more detail on those and the wonders of Xplico PCAP session reassembly in the next post.

Please also note…if you shut down Xplico and the Ubuntu system, then before you re-launch Xplico the next time you need to run the following command in a terminal session before launching Firefox and logging into the Xplico web interface:

sudo /etc/init.d/xplico start

I suspect in the DEFT 6 LiveCD, that when you run the Xplico icon and the terminal window opens but doesn’t close out it is trying to do the following but failing for some reason.

sudo /etc/init.d/xplico start http://localhost:9876/users/login

I haven’t had time to see if a manual-launch of Xplico in the DEFT 6 Live CD will work better that way.  Xplico appears to work but fails on uploading of PCAP files in my experience.

Post Script #1 - Useful Xplico-building Resources

Before I eventually dug up blaksark’s Nsark script, I did uncover a few more installation recipes from other Xplico tinkerers.

I'm listing them below as together they provide a great overview of other installation techniques on a few other platforms.  They might be found helpful by others all assembled in one place:

Post Script #2 - Pre-Loaded Xplico Distros (Installable)

For whatever reason, to the best of my knowledge, the DEFT builders haven’t included an installer for the LiveCD to allow installation directly onto a local drive (real or virtual).

Only after all this exercise, and some leads in the resources mentioned above I’ve found (so far) two LiveCD distros that do include “pre-built” versions of Xplico in them, and can be fully installed in a real/virtual system.  This may be another option for folks who don’t want to cook your own version as I’ve shown earlier.

If you are aware of any other LiveCD’s (with installer support) that include pre-added builds of Xplico, please drop the information in the comments and I’ll keep this post updated.

updated 03/06/2011 to include Security Onion LiveDVD suggested by Doug Berks.

Hope someone finds this useful.

Next stop…putting Xplico through the paces on PCAP processing and traffic reassembly.

Cheers!

Claus V.

Posted by Claus at 3 comments
Labels: , , , , , , , ,

New and Improved Material

Just a pause to empty out the handy-dandy “to-be-blogged” folder.

Windows Tips and Tricks

Firefox Bits

I’ve been running the Nightly x64 builds of Firefox 4.0b13pre for some time now and it has been very stable.  The fact that there is a developer release version of FlashPlayer for x64 bits helps a lot in the usability factor.  Right now I am still split between using Chromium nightlies for “fun” surfing and Firefox 4.x for my daily web-work commuting.

One of the major challenges with both platforms is finding compatible extensions/add-ons for the new versions.  I’ve had to walk away from some old favorites as they just don’t work at all in the new Firefox 4 world.

That said, I’ve been successful enough to keep productivity and security intact and will post updated extension/add-on lists for both browsers in the near future.

New Utilities

Piriform is one of a handful of software providers who I want “one-of-everything” when I visit.  Counted up there with Sysinternals and NirSoft as my go-to source for awesome software.  They haven’t been resting and have been hard at work on updates to what should be Windows-required utilities for all sysadmins.

Tip: In case you haven’t figured it out yet, once you go to the product download page, just scroll to the very bottom of the page at the Builds header and find the tiny “builds page” link to find the jump to the portable (zip) versions.

Cheers!

Claus V.

Posted by Claus at 2 comments
Labels: , , , , , , ,

Xplico & VirtualBox Headaches - Part II

Yes.  I know.  I really know.

I’ve promised a post on the wondermous Network Forensic Analysis Tool (NFAT) Xplico.

When it’s working, it is an outstanding tool, particularly when you have to take some of your PCAP files from the analysis bench into the boardroom and present findings in a way decision makers can relate to after an incident or network analysis review.

I started out cutting my teeth by using the 0.5.x builds directly in the DEFT Linux LiveCD builds.  Then I started playing around with the Xplico-provided VirtualBox Image builds including the new 0.6.x versions.

I was all set to start writing a post…when I was surprised at work to suddenly be getting no-boot errors on the VirtualBox vmdk drives I had some cases going on on my XP system.  Attempts to reload VirtualBox (from the 3.2.x version to the latest 4.0 versions) and/or redownload and deploy the various Xplico-provided vmdk images were unsuccessful…despite all the MD5 download hashes matching…even on different XP systems.

Fortunately, I was still going strong on my home system’s VirtualBox vmdk images for Xplico where I had some community-provided PCAP files to use for the post.

Only last weekend, when I launched them, they too experienced the same error.

image

Above: The killer-diller error.  Brand new, first-launch of Xplico’s latest VirtualBox 0.6.1 image/appliance.  Note that right after setting the system clock and activating the swap file fsck does a forced check saying the drive hasn’t been checked in over 249 days… Same thing in both VirtualBox 3.2.x builds as well as the latest 4.0.x releases; XP/Win7..doesn’t matter.

image

Above: After the original error, the damage has been done and now I get this every Xplico VirtualBox Image boot.

So now I was left with trying to use Xplico directly off the DEFT LiveCD builds.  Only the previous version of Xplico in the DEFT 5 was an older version and didn’t seem to render the images in the rebuilt web-page sessions, nor Xplico in DEFT 6 which seems to run, but for some reason all attempts to upload PCAPS failed (I think it is an apache issue as the terminal window never closes like it does on the DEFT 5 LiveCD build).

Double Bummer!  Particularly after feeling a bit better having overcome this DEFT 6 and VirtualBox: Maybe it’s just me? issue a few months ago.

Now, while I got started in the early days of LiveCD building by hand-building custom Knoppix (Damn Small Linux) boot CD’s, I’m just a few levels above “noobie” when it comes to Linux building, working, and troubleshooting.

As the images presented earlier capture, the whole issue seems to be that when I ran any of the VirtualBox vdmk images, during the boot process a diskcheck (fsck) was/is triggered due to some kind of date/clock-time stamp.  It claims I haven’t used these in over 258 days…thus triggering the fsck.  Only if I do run a manual fsck as suggested, it claims to find a bunch of stuff “bad” and “fixes” it all.  Only upon reboot the system is hosed.

I know there are ways to Skip or Bypass a Fsck but despite my best attempts, I couldn’t get grub to cooperate with me.

So now I was really frustrated.  I was/am still unable to get the (really nice when running) VirtualBox images directly from Xplico working.  And the versions in the LiveCd’s from DEFT, while nice, aren’t really a convenient environment for real and persistent NFA case work.  Based on previous work with Xplico I know that it can deliver and deliver very well…only I felt like I was running lame with any of these current solutions.

So that meant I had one last possibility (at least as far as I knew at the time)…roll my own “installed” Linux build on a fresh vmdk file in VirtualBox, and then manually install Xplico into it.

I’m cool with that, I needed a fully working Xplico build, and maybe it would be a good exercise before going into Xplico proper.  How hard could it be?

The answer?

Really, really frustrating…then stupidly simple.  Seriously simple.  Even Alvis could do it.

image

Above Image…the Xplico baby is delivered and working perfectly!

It can be done, and now I have a fully functional Xplico application running in an installed/hdd based configuration (still virtualized in a VirtualBox vmdk file) so I can save and revisit all my PCAP uploads.  Sweet Success!

So that post is coming up next…maybe even later today.  I now need to reproduce/test it on my work XP system…just to be 100% certain the process works.

In the meantime, this humble Linux padawan would deeply value any feedback from the Linux/VirtualBox Jedi Masters on why out of the blue the fsck started complaining about the time since last boot right after setting the system clock (certainly not 249 days!) on these vmdk images…and any solutions for fixing this issue. Now that I can roll my own I’m not really going back, however other users/testers might be curious and run into the same thing. 

From the Google work I was able to do, there may be an issue with the way the VirtualBox BIOS is reporting the actual time/date (or that it can’t get it from the hardware system) to pass on correctly to the virtual system.  Am I the only person running into this issue with the Xplico VirtualBox images?  Surely not as it replicated on different XP hardware systems as well as (finally) my Windows 7 system as well…and despite many installs/uninstalls/reinstalls/fresh-system installs, I have since been unable to get one running again.

I believe that by default, fsck is set to run automatically after x/days or y/boots.  However, I’m curious why that now always appears, even after a fresh reimport of either Xplico VB appliance.

Cheers!

Claus V.

Posted by Claus at 1 comments
Labels: , , , , , , , ,
Subscribe to: Posts (Atom)