So Dad calls me up last weekend because his home pc is freezing up after boot.
Symptoms: Once he logs onto his desktop it seems ok, he is able to launch a program or two. He goes to check his email or browse the web and ----bammo! The Great XP Freeze. Pressing Ctrl-Alt-Del is unresponsive. Finally after what could be many minutes of waiting the system is flooded with pop-up Task-Manager windows matching the number of times he has tried the Microsoft Three-Finger Tango. Internet is good again and mail comes flowing.
Hmmmm. I'm on the phone and Dad. I could set up a TightVNC remote session, but I'm not sure how well that would run on his chilly system.
Troubleshooting begins:1) He mentioned a new update of ZoneAlarm got installed recently. Firewall problem? Could be. Since he doesn't run a hardware based router/firewall and is on cable modem, I have him disconnect his CAT-V cable. ZA gets uninstalled.
2) He is also running a Nortons Security A/V "suite" product. OK. I've had trouble with those. Uninstall that pile of programming also. Reboot.3) System comes up quick and fine. Once on his desktop he is able to open programs with no sign of the dreaded XP Freeze. Great!
4) Before we drop him back on the Net, let's get Windows XP built-in firewall enabled. I lead him through the steps. We find it and eventually get it activated. Great! Reboot.
5) Hook the pc back to the Net.
6) Boots up and gets to the desktop....bammo! Lockup! The Microsoft Three-Finger Tango fails to get Task Manager launching. Hmmmm. Malware?
-At this point we are approaching the point where more advanced troubleshooting will need to kick in. I suggest Dad brings the box down here to me on his weekly drive-in to Houston for his consulting job. Plans are made.
(Few days later)
7) I fire up the box on our dining-room table and set to work, Dad watching and taking notes. Class is in session. XP boots and runs great. Dad is shaking his head. I really need to replicate the freezing behavior! The box isn't cooperating. (I think it knows a techie is in front of it. I swear they can sense our presence!) It's still off the Net for now.
8) First thing I do is to drop in my utility-tools CD and fire off HiJack This. I find a number of old "auto-run" references and some "quick-launch" references we don't need. I clean all those out, but no signs of malware. Lots of dead startup links for Symantec left by the Nortons uninstall. Thanks guys... I confirm the HiJack This findings with Sysinternal's AutoRuns. Yep. Looks clean now.
9) Next I run a SpyBot S&D scan. Some cookies and mundane stuff. Nothing interesting here. I don't even bother with a 2nd scan with AdAwareSE.
10) Check the Add/Remove Program List. Even though Norton's was uninstalled, there are still four Norton's related items still hanging on in there. Uninstalls fail since the product is gone. I'll come back later to strip them out of the registry manually. I find Microsoft Anti-Spyware Beta (expired) is still on the system. That gets removed.
11) Reboot. XP still humming along well. Well.....a quick check with Sysinternal's Process Explorer doesn't show any unusual Processes or threads.
12) I check his system properties. Yep XP SP2, 512MB RAM, 800ishMHz processor. (OEM HP system build.) The processor speed is pretty low (in my humble opinion) for running XP, but RAM is OK and it is working. With Process Explorer still going I launch various programs and watch the cpu load shoot up on some processes but then drop down once the application is going. Looks like the processor is having a bit of a challenge keeping up. Could be bad memory, but I doubt it since this pc hasn't had any BSOD events. I tell Dad he should consider upgrading to a better processor but it isn't critical just yet as all they do on the box is some word-processing, light picture cropping and email/Net surfing.
13) OK. Well, let's hook it to the Net and get a new firewall and A/V solution loaded that will be more kind on his processor than ZA and Nortons.
14) Reboot and hook to the net behind my firewall/router. Once on the desktop....BAMMMO!!!! TOTAL SYSTEM LOCKUP!!!!
15) I'm actually very excited to see this! I've got the problem replicated now! No Net = good pc. Net = bad pc. Now I've got a target to focus on.
16) The Three-Finger Tango fails. So I reboot in safe-mode, system is fine...and add Process Manager to the Startup Group. Hopefully this will launch it prior the the lock-up stalling so I can see what bad-boy is causing it. Whatever it is, it is clear to me that Something is getting to the net and either hogging up all the bandwith from the pc or is executing a process that the lowly cpu cannot keep up with.
17) Once rebooted the system comes back and Process Explorer comes to life. Fantastic! I'm watching the cpu process loads and.....wait for it.....YES! Got it! Well how about 'dat. The G-man set the trap, waited on stake-out, and collared the kriminal!
18) What did Claus nab? Who was playing "Mr. Freeze" like in a bad Batman movie? Yep. Microsoft Windows Update (wuauclt.exe). (Note: this is the "good" one and not a virus/trojan faker.)
Let's pause a momement in this "thrilling bat-tastic blog-drama" and sort out what we learned so far. Dad's pc has sufficient RAM, an overtaxed processor (it was upgraded from WindowsME to XP) and bit the dirt when it was connected to the Net. No malware on the system. Through careful troubleshooting, we can now explain the lockup event. When connected to the Net, XP (having Automatic Update check enabled) goes on-line immediately and searches for availiability of any updates to download. Unfortunately, the wuauclt.exe process take prioriy over the system and runs the cpu cycles up to 90+%. Bad. Nothing else works until it completes it's check (however long that takes--usually a LONG time as of late) then closes the process and sufficeint cpu resources become available for the system (and Dad) again. All the things Dad tried to launch are released from the buffer and flood the desktop. Nice.
19) Time to fix. Well. That's being generous. I would be more accurate to say--time to "work around" the issue. The (user-side) "fix" would probably be a beefier processor (more on that later...).
20) I go into Control Panel and completely disable the Windows Automatic Updates feature. This will prevent the wuauclt.exe from launching at startup and hogging the processor. XP hates this and tells me about it in no uncertain terms. It places a scary red shield in the system tray and provides balloon text warning about the serious consequences of that decision. Great. 'Preciate that, Microsoft.
21) After a minute of Googling, I locate a page that reminds me how to disable the "Security Alert Icon" notification feature. Gone!
22) I place a shortcut to Windows Update web-page on the desktop and advise Dad to manually check for critical updates the weekend after the 1st Tuesday of each month.
23) I download (via my handy PC First-Aid Kit blog link) and install Sunbelt Software's Kerio Firewall. Dad wants the "simple" installation option that enables inbound firewall protection only. He is too overwhelmed with deciphering the outbound firewall prompt messages and is afraid he might block something important (which has occurred). I agree. Inbound blocking only. Done.
24) I download and install AVG-Free anti-virus. I give a quick tour and configure it to automatically download and install the updates and run a scan in the late afternoon. AVG cooperates nicely with Keiro and picks up his Outlook email client and plugs right into it with no additional configurations needed.
25) Reboot. The final test.
26) Armed with Process Explorer we watch as the system hits the desktop and the process thread cycles look normal. AVG and Kerio are running lightly on the system and no lockup and no wuauclt.exe launching. Mischief Managed!
27) Just for kicks I fire up IE 6 and we run the Web-based Windows Updates. BAMMO! there goes the lockup and the process load is at 90% again. I explain to Dad that based on the cpu loads I'm watching, it looks like it is working, just VERY slowly. I advise him to just start the Updates, and then walk away from the pc for a while (like before he goes jogging). When he comes back it (hopefully) will have finished up.
I didn't install the latest Windows Defender Beta version, as I didn't want to add any additional processing demands on his taked system.
Dad is happy. The system is breathing new life and all is well. Another modern-day father-son bonding moment. Instead of working under the hood of the MG-TD (kit-car) convertible we built together (Classic Roadsters, LTD. - the Duchess model) way-back when, now it is over a keyboard and system software. Ahhh. Nothing like the hum of a hard-drive and cooling fan to set the bonding mood.....unfortunately, no beer was consumed in the process...maybe why I was able to figure it out this time....
Post-script.
I'd never noticed any problems myself. I have Automatic Updates set on my machines to auto download and notify (but not install so I can check them first). So this morning, I ran a "custom" Web-based Windows Update scan on my own pc: 2.0GHz AMD processor, 1GB RAM. I fired up Process Explorer before I began and watched. Sure enough, the wuauclt.exe process peaked around 74ish% of my cpu cycles. Wow. But because of the beefy processor, it just peaked and dropped up and down very quickly until it was done. So I had the system headroom to not notice any lockups or anything.
Open Questions that remain for Microsoft (or anyone else who knows...):
1) Why does wuauclt.exe need to claim that many cpu cycles to do it's job? Is it that intensive a process to check for updates on the Web and compare that to the locally cached catalog? Or is the code just poorly written?
2) Why does it take (what seems like) FOREVER for the query to Microsoft's update servers to return a response? Are they being hammered? Do they have enough? Is it a bandwidth issue in Redmond?
3) I'm quite OK with it taking a while to download the updates to the system, but it just doesn't seem to me to require that long a delay in displaying the list of needed system updates. This happens on our Win98 (yes, I know...) systems, Windows 2000 SP4 systems, and XP (Pro/Home) systems. Can't this be more efficient?
4) A google search on this subject suggests that many others are having similar issues and Microsoft is aware of this "issue."
Others who are having (unrelated but interesting) Windows Freezing issues....
System freezing up? Check your hardware - Ed Bott's Windows Expertise
Tech Support With Thomas Hawk -- Thomas Hawk's Digital Connection
The Case of the Process Startup Delays - Mark Russinovich's TechNet Blog (formerly of Sysinternals)
Note to self: Damn. I SOOOO wish I could have Mark's l33t troubleshooting skills! I've GOT to buy his books. He's taught me so much!
See you in the skies, and happy Microsoft defrosting....
--Claus
9 comments:
Claus - Does the Windows update process set itself to realtime priority? If not, I don't see how it could lock up the system. Even in high priority, other processes shouldl get time to run while the update process waits on I/O.
If a unix system were locking up like this, I would suspect that something was blocking in the kernel -- probably a device driver bug. Just a suggestion, but have you checked the NIC drivers to make sure they're up to date?
Good thought, Jim. I'm not a process level specialist, but I agree it should let the other things run. What was amazing was that it was taking up that level of CPU usage all by itself. There just didn't seem to be any extra room for anything else to run since it stayed pegged at 90+ percent.
I didn't check to confirm the process priority of wuauclt.exe on my Dad's pc. On mine, it is set to "normal".
He was just so happy to have it working again.
That's why I was led to the feeling that his CPU just couldn't keep up with the processing load of whatever instructions wuauclt.exe was feeding it.
Another possibility (currently left unexplored) is that his wuauclt.exe has some type of file corruption or is a "bad" version. I didn't look into that either. I guess I should have checked out the file's version number and info. If in doubt, the Windows Updates files could have to be reinstalled or I could have just reinstalled IE 6 again as well.
Next time I get my hands on it, that's what I'll be checking out.
The drivers seemed ok on the NIC and once I got that Windows Update process manually managed, the rest of his net connections flew pretty quickly, with no system slowdowns observed.
--It was quite an interesting night.
A couple of interesting observations here:
(1) The lockups that your father experienced look a lot like some that I experience, but in my cases Sunbelt/Kerio Personal Firewall (both free and paid functionality) has been part of the problem. Sunbelt released an updated version that fixed part of the problem so it no longer completely hangs Windows, but part still remains unfixed. Sometimes Sunbelt doesn't seem to be part of the problem but sometimes it clearly is. When Sunbelt's log file on disk ends with the previous day's network accesses and doesn't include network accesses that succeeded today before the hang, I think Sunbelt still has work to do. But you have no problem with the same firewall and you do have the problem only with the infamous wuauclt.exe. I wonder how to reconcile this.
(2) You seem to like Japan a lot. Did you notice how Sunbelt/Kerio Personal Firewall displays some of its strings? This is a comparatively minor issue because it doesn't affect the firewall's functionality, but there have been occasions when I wished I knew which network interface it was warning me about.
Hi Norman, thanks for stopping in.
Yes, there is much of Japanese culture that I appreciate and like. And lots that I find just as crazy and mixed up as our own here in the States. I haven't had a chance to visit there yet, but maybe one day...Guess watching the NBC mini-series Shogun imprinted itself on my psyche somehow as a kid...
I did a second post to update some info in this one Thawing an XP System - Update. Don't know if you saw it. Dad says he isn't having any more lockups in the weeks since I performed this service. He is very happy with it...but has provided some more details that are useful.
1) He says that he is able to manually run the Windows Updates, it takes forever, when finally done, presents him with a list (every one?) of critical updates. He downloads and installs them. Next time he checks again they are all back.
2) That suggests to me that his Windows Update software/files are corrupted somehow and I need to go back and rebuild/reinstall those. I remember now that he had told me about this problem before but hadn't mentioned again until now. That is quite likely why he is having the issues with wuauclt.exe as bad as he is.
3) After speaking to my "little" brother, I learn now that he had upgraded Dad's pc to XP by overlaying WinME. I never upgrade operating systems like that. I always prefer to do a fresh install. That could be where the Windows Update components got corrupted.
4) I haven't run into any issues with Sunbelt's Kerio firewall yet. Well, sometimes Lavie finds that she has missed a "permission" box and before we know it, the task bar is filled with twenty plus Kerio icons! We have to kill it and restart it. Other than that, it seems fine. Dad liked it so much on his system, he bought it.
5) Since we run a hardware-based firewall via our router, I don't worry or look at our software based firewall logs at all. I know they are there, but since I'm just using them to warn us of any "leaks" out that we don't expect (trojans/malware/etc.) I don't expect to find many/any inbound logs present. I guess I should go back and take a closer look from time to time. That's a very good observation you make! I wasn't aware it didn't report on which interface it was warning about. That definitely would be good info to have. I'll be looking at the Sunbelt logs this weekend and see what I can see.
Sounds like you "generally" like Sunbelt enough to stick with it for now. I'd recommend trying Jetico, but even for a sysadmin like myself, it remains chatty as heck after all this time and is a real pain in the rump to configure/adjust/etc. until you get past the steep learning curve. I can only comfortably recommend it to those who need super leakage protection as well as the knowledge to handle its constant feedback requests. That's not meant as a complaint--that is one of the features that makes it so strong--but just my observation in daily usage. I haven't taken off my desktop system yet, but may eventually swap it out with Sunbelt's Kerio as well.
In the meantime, you might want to try (if you haven't already) running Sysinternal's Process Explorer, FileMon, and Diskmon apps. SysInternals apps concurrently . The logging function of these might give you some indication where/when Sunbelt is hanging up or having issues. That would be a fun task sorting through all that data however.
Sorry I left something out of my previous comment:
> You seem to like Japan a lot.
> Did you notice how Sunbelt/Kerio
> Personal Firewall displays some
> of its strings ...
... under Japanese Windows XP?
The program gets some strings from Windows APIs and then shoves the strings through an 8-bit font instead of using Windows' standard working fonts. The result is unreadable in any language.
Since I run Virtual PC in the same machine and set up a Microsoft loopback adapter (not 127.0.0.1 but an additional simulated adapter which can be used for communication with emulated PCs), Kerio takes charge of more than one network adapter. For a while when Kerio was popping up message boxes with mojibake I had to try to guess which network cards it was talking about.
Thank you for pointing me to your update and I read it just now. I will check the priority of wuauclt.exe in the future. But I still have the impression that Kerio is still part of the problem, at least in my case.
I paid for Sunbelt/Kerio after several months of having no crashes in a previous version. I felt they'd finally earned it. But then problems resumed. I also paid for AVG for myself, after seeing the free version work well enough on my wife's PC. I also have been using a hardware router since getting an ADSL line, but Kerio was invaluable during days of dialup and W98 or W2000 or XP prior to SP2 (outweighing the trouble caused by its crashes).
Have you upgraded from windows update to microsoft update?
Microsoft update automatically updates windows plus a number of other microsoft programs like office. Unfortunately the system scanning code of MS update is flawed and ends up eating close to 100% CPU on slow machines freezing them for minutes. Until MS fix it you can revert to the simpler and working windows update and just update office manually via the office update website whenever you remember to do it (btw but who actually worries about updating office, apart from me?).
I hope it helps.
Carlo
Carlo,
No, I haven't taken the plunge to Microsoft Update. I've heard a mixed bag of results, so call me "old-school" but I'm passing for the present time...but you are correct and that's one alternative option I should have considered.
In the end, the problems ended up being a corrupted Updates Catalog.
Mostly Mozilla Madness... (look near the bottom of the post)
I nuked it and rebuilt it. That seemed to fix most of the issues. He is back to getting updates fine again, but still likes doing them manually.
And, BTW, I'm notorious for updating Microsoft Office as well! Too many people forget about it, and it isn't a big focus at work either so my IT teammates really get annoyed with me for bringing it up constantly with them!
Thanks for stopping by!
You father-son bonding made good reading.
I am having the same problem with my computer. The wuauclt.exe runs under svchost.exe and periodically that svchost.exe starts consuming 99% of cpu and locks up my computer.
Then it gets complicated.
First, I have turned off Auto Update but wuauclt.exe still fires up.
Second, This is my work pc. I encounter this problem only when I am in the office. So I suspect there is something on our networking scripts that cause this behavior.
Any ideas.
You didn't say what type of OS?
XP Professional?
If you have Auto Updates turned off, it is possible that group-policy settings exists that are over-ruling the local settings.
Try going to Start-->Run and type gpedit.msc
Browse down the left hand side and go down through Local Computer --> Computer Configuration --> Administrative Templates --> Windows Components --> select "Windows Updates".
On the right hand window, go through each item and adjust the settings accordingly.
If you are running Windows Defender, that might also call to the wuauclt.exe file to perform the definition updates.
Another idea: my pal jimthompson recommended in the first comment post here to look at the service's priority setting (use something like Process Explorer for Windows v10.21 to see what it is set at when you catch it running. You might try bumping the priority level down to see if that helps.
Finally, I think the (real) problem with Dad's pc was that he had some corruption in his local Windows Updates catalog. That was causing the update process to hang as it tried to sort it out. Check out the bottom of this follow-up post as well...Mostly Mozilla Madness...
1) Manually downloaded the Windows Installer 3.1 Redistributable (v2) and installed it.
2) Manually downloaded the Update for Background Intelligent Transfer Service (BITS) 2.0 and WinHTTP 5.1 (KB842773) and installed it.
3) Rebooted the system.
4) Dumped the contents of the local Automatic Updates DataStore and Download folders,
5) Reregisterd the DLL files that are associated with Cryptographic Services (Method 4) ..
6) Rebooted the system.
7) Ran a check for Windows Updates. Found them, downloaded them, installed them.
8) Rebooted and checked again. They were all applied and none were now found needing installation.
Perfect!
It's frustrating. Good Luck.
Post a Comment