Thursday, July 17, 2008

Windows CPU throttling techniques

At work we have an old DOS 16-bit application that is central to our customer’s daily tasks.  It must be run and used by the majority of staff.

Alas, it is still being run on Windows 2000 and XP Professional systems.

This means that those (NT-based) 32-bit OS systems cannot run the application natively.

To get it going, what Microsoft does is to call to a particular environment subsystem; in this particular case, ntvdm.exe.

According to what I have found, it fires up a Virtual DOS Machine “application environment” which this particular DOS 16-bit application runs in.

No problem, except in our case, that ntvdm.exe session gobbles up 98% of the CPU cycles almost full time.  This renders other 32-bit applications such as Outlook, Word, IE, or other applications the user is also likely to have open and running almost useless as they stutter and “hang” waiting for CPU cycles to be released.  Task-switching is a nightmare.

Because that DOS application is network-based (makes calls and database updates to local servers), it can not be suspended while in the background, else the network connection might drop and the application crash horribly.

In the past we have thought it was a memory issue with DOS grabbing it all up.  That is not the case.  Extensive monitoring of the systems with it running using Process Explorer made it clear it is a CPU thing.

Changing the priority the process ran under wasn’t effective as a solution. No matter what it was set on, the CPU % utilization stayed pegged at 98%.

So, what now?

CPU Utilization Control #1 – Process Throttling

Now having some idea what we were dealing with, I wondered if there would be a way to restrict the ntvdm.exe process so it throttled back to a more system-friendly utilization rate.

Additional searching on the web (now that I new what I was looking for) turned up a number of third-party applications that looked like they might help.

Thread Master - (free for personal and corporate use) – This application runs as a Windows service when “installed".  There is no GUI interface. Nothing to see that it is working.  It is completely transparent to the user.  Tweaks and configurations are handled in the registry keys the program makes.  Download the files to the system, run the install.cmd and you are good to go. To uninstall download the uninst.cmd script and it will clean it up.  The program page says it is for Windows 2000 and Server 2003 but my tests on XP Pro seem to show it works just fine.

BES, Battle Encoder Shirase 1.3.7-beta - (freeware) – Download and run the bes.exe file.  You will get a management window.  Click “Target” and select the (already running) process you wish to throttle back. Then click “Limit”. It will control that (and any other selected processes) as long as it is resident. If you wish to throttle a process the next time it launches, select the limit/watch so it re-controls it on relaunch.  You can watch/throttle up to three processes but only “watch” for one.  If you want to watch just one, you could make a startup script that calls BES from a command-line. This might make it “invisible” to users. Launching in this mode invokes the watch/limit mode for the process listed.

Process Lasso - (free for personal use/ $ for business use) – A really top-notch process control application. By default, it has been set up to monitor and lower the priorities of processes that use too many CPU cycles. Works on both single/dual core processors.  Highly customizable and can be launched in a non-GUI format so it is “invisible” to end users. Runs under Windows 2000, XP, 2003, Vista, and 2008 in both x32 and x64 bit flavors. Wow!  This could be one of the most—if not the best—fine-tuned and polished applications of this sort out there currently.  It can also be invoked from highly optioned command-line arguments and application processes can be added to “exclude” lists to bypass throttling if needed.  I like it so much I am using it on my home systems.

ProcessTamer - (free, registration recommended) – Tiny application that monitors CPU usage of processes. It will throttle back the CPU process depending on set threshold limits.  You can toggle the application on/off as needed (optical media burning, perhaps), as well as set very detailed rules for process exclusions and priority adjustments.  Certainly a nice little tool as well.

Priority Master 2008 - (demo/$) – Application that not only will control running processes but will also allow uses to permanently set the priority value for an application, as well as terminate any program you specify should not be allowed to run. Great with dealing with malware processes.  This looks to be a very extensively designed utility tool.

Prio - Priority Saver - (free for personal use) – Adds some extra tabs to Windows Task Manager as well as keeps a record of your priority setting changes and when the process is relaunched (with Prio running) will open it under the preferred priority setting.

To figure out which one might work best for our ntvdm.exe control needs I fired up a benchmarking application (PassMark’s PerformanceTest) with the system not running the application, with the application running, then with it running under each of the first-four listed CPU managers.

In the end I found very close positive stats for both Thread Master and Process Lasso. 

Because Thread Master was free for corporate use I went with this application.  I did have to spend much additional work in the registry along with Process Explorer to get it tweaked just right.  If I set the CPU throttling too low, then the DOS application would hang or appear to lock-up.  If I set it too high, no benefit was seen. It looks like the sweet-spot between functionality and throttling control is around the 75% to 85% range.  Because it runs as a service and has no GUI interface, it is 100% transparent to end-users so they can’t get in to it and muck up the settings.  That is a nice feature.  It did seem to improve responsiveness of the Win 32-bit applications running at the same time as ntvdm.exe with our particular DOS 16-bit application.

Now I am going to have to find a few production machines to test it out in real-world DOS-16 bit application utilization loads.  Still got a lot of work to do before we can bless it officially as a “solution” and push it out but it looks like we might be on the right track.

CPU Utilization Control #2 – Processor Affinity

Turns out that we have a blend of systems in our desktop computing environment.  Not that I didn’t know that but I never really appreciated the nuances until now.  We have at least four different models that are single-core processors. Our newest desktop systems are now dual-core systems.

Believe it or not, that makes a world of difference.

When I went to see if the ntvdm.exe DOS-16 application impact was as severe on our dual-core systems I found it was not the case.  32-bit applications didn’t lock up when the DOS-16 application was running. In fact, it was almost like there was no impact alone.

Additional monitoring with Process Explorer set to show muti-core loads found that the ntvdm.exe process was being load-balanced across both cores.  This resulted in the CPU utilization being more manageable for other applications, also sharing across cores.

So in the end it appears that we might only need to deal with CPU throttling solutions on single-core systems. Nice.

There are a number of ways you can actually control the cpu processor affinity for an application on a dual-core system.

  • Quick and Easy:  If you want to make a temporary change, just open Windows Task Manager.  Find the running process you want to control, right-click and select “Set Affinity” from the menu options. For a dual-core assign it to either CPU 0 (the first one) or CPU 1 (the second one). Done.

  • Quick and Detailed:  If you want to make a temporary change, just download and run Process Explorer. Find the running process, right-click and select “affinity” from the menu options. For a dual-core assign it to either CPU 0 (the first one) or CPU 1 (the second one). Done.

  • Free and Convenient:  Use the freeware application THG Task Assignment Manager. Tom’s Hardware Guide provides this free utility to help you quickly manage processor affinity settings on the fly with their light and handy tool.  For full read and download link; Getting More Bang Out of Your Dual Processing Buck – Tom’s Hardware

You can also change an application to run exclusively under a single CPU on a dual/multi-core processor, but it takes a bit more work.

In my case it specifically addressed the ntdvm.exe file tweaking that I was looking at, but could apply to almost any process.

Troubleshooting an MS-DOS application which hangs the NTVDM subsystem in Windows XP and Windows Server 2003 -  markwilson.it

Read the post but it comes down to using a Microsoft Windows 2000 Resource kit tool, imagecfg.exe with some detailed command-line arguments.

Related posts on this technique:

As I said, on our dual-core systems, the impact of NTVDM.EXE is so marginal it isn’t worth the effort to tweak it out and modify the file on them.  Also, since ntvdm is a core system file, there is the chance that a modification will be erased in a later Microsoft patch or service pack update.

However it was certainly fascinating to learn about and is a good bit of info to keep for future reference.

Curiously, my dual-core explorations on our desktop systems led me to an incredible discovery that was a headache waiting to happen…and to a solution that was elegant and painless.

Wait for that post very soon!  I promise it will be a doozie!

For the Very Curious

For an excellent treatise on optimizing applications and the NT application architecture/subsystems and strategies, this 9-page whitepaper by Sean Daily is tops; Optimizing Applications – Windows IT Library.  I learned LOTS from reading it.

It also has some great tips to help customize applications running in a ntvdm.exe session so they report their name instead of the generic ntvdm.exe process name, issues related to the wowexec.exe thread (Win6 on Win32), running Win16 apps in different memory spaces, process thread priorities, and a stack of desktop environment optimization tricks.

One of the coolest was a trick on page 8 showing how to run the Explorer process, the Desktop process and the Taskbar all in separate processes (by default they run under a single session). This might minimize the impact that process crash in one area might wipe out the others.

Page 9 finally has some great (and unintended) retro-tool links, including views of the then cutting-edge Process Monitor (PMON.EXE), Process Viewer (PVIEWER.EXE), and Process Explode (PVIEW.EXE—same file name, different program); all dug out of the WIndows NT Resource Kit.  All early hints of what would come out as Process Explorer and Process Monitor tools from (Microsoft) Sysinternals camp.

It’s all good stuff to know, even if you don’t make the tweaks.

--Claus

8 comments:

Joe said...

Claus,

You may want to also try DOSBox. It's a x86 emulator that is very stable from what I've read about it over time. The developers have a focus on getting legacy DOS games to work, however running your old DOS software on it may solve your problems. I'd guess that using DOSBox would be much more friendly than using various process manipulators, and far less resource-hungry than a full-blown DOS VM from VirtualBox or whatnot. May take a little bit to configure it to your liking, but then you can just dupe the same config on all the PCs.

This link shows their efforts in emulation:
http://www.dosbox.com/status.php?show_status=1

Claus Valca said...

@ Joe - You know, it's funny you mention DOSBox.

I had tried that technique but it couldn't support the specialized printing needs that the application required.

I also tried running it in it's "original" state when we ran it in pure DOS environments (before Win 3.11) using VirtualPC and VirtualBox. I was much more successful that way.

Virtual PC and DOS tips

Virtual removable media drive utilities

However, on the older single-core systems we run this on, we are only able to provide between 256 MB and 512 MB system RAM. Now I'm running into true memory issues. And we can't spend the $ to upgrade all these systems to the 1GB they really need in that case.

One day the replacement application for this DOS app will be deployed....

Good suggestion!

DougCuk said...

I found an interesting solution to running an old custom written DOS Network Database application on XP recently. And it was free!

The main problem I had to solve was how to allow the application direct access to LPT1 - but the solution should work for any system port. The solution was a custom I/O device driver called PortTalk. Worth a look if you have not seen it before.
See http://www.beyondlogic.org/porttalk/porttalk.htm

Previously posted in the wrong thread - apologies

Claus Valca said...

@ DougCuk - Thanks for the additional PortTalk link!

I'll have to take a closer look at it to see if it could be a solution. This DOS app that has been such a headache maps to LPT1, LPT2, and LPT3 for various components and output. In most circumstances, we can funnel all the output to LPT1 and work just fine.

It also requires the printer handle PPDS emulation modes. That limits the types of printers we can deploy.

Ed Qualls said...

WOW! Thank you for taking the time to write this up. I've had a 16-BIT application driving my customers nuts with 99% CPU utilization for some time now. I ended up using BES instead of Thread Mster, as I didn't want to try to get my users to be comfortable with registry keys.

This is going to make my customers very happy.

Thanks again,

Ed

Claus Valca said...

@ Ed - Thank you for the kind words! I know what it is like to have customers who are frustrated and no matter what you do, a workaround just seems out of reach.

Like you point out, each of the solutions takes slightly different approaches in their interfaces and methods. That's why I wanted to take a look at mulitiple offerings, and provide the full list for readers.

I'm glad you found the effort useful!

Cheers!

-Claus V.

DougCuk said...

Regarding the PortTalk driver - my problem was with a dongle protected application that required direct access to the LPT1 port address - to read and write to the security dongle. Using this driver creates a shell with priveledged port access levels for the specified DOS application, and closes once the application terminates.

I have a heavily used 8 station network running a multiuser database, which has been running for over 18 months now, with no problems. The DOS application was written back in 1991 and worked fine on versions of Windows up to 98se. It expects to be able to access LPT and COM ports by direct IO port addresses both for communication and dongle security checks.

The DOS application has a limited printer command set and bypasses Windows printer drivers - attempting to send data drirect to the LPT port. I was concerned about problems over the long term but this fix has been trouble free.

Anonymous said...

While the solution you found worked - it's a brute force approach not addressing the real cause.

You should try tamedos from www.tamedos.com. The source of your problem is probably that the dos 16bit app is not using bios calls for keyboard input but looping for keyboard input. This sucks a cpu dry which never mattered in stand alone dos days.

Once you put the tamedos app on your machine then you will find the dos app will use almost no cpu. We've used this technique with our own hybrid vertical market application that still has some legacy 16 bit code in it and it allows us to run 50 users fine on a single terminal server. Without it one user will eat one cpu core.