Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago my wife complained that her Vista desktop was not responding to her typing or mouse clicks. Given the importance of the customer, I immediately sat down at the system to troubleshoot. It wasn’t completely hung, but extremely sluggish. For example, the mouse moved and when I clicked on the start button the start menu opened after about 30 seconds. I suspected that something was hogging the CPU and likely could have resolved the problem simply by logging off or rebooting, but knew that if I didn’t determine the root cause and address it, she’d likely be calling on my technical support services again in the near future. In any case, stooping to that kind of troubleshooting hack is beneath my dignity. I therefore set out to investigate.
My first step was to run Process Explorer to see which process was using the CPU. After a few minutes Process Explorer finally appeared and showed that not one, but two processes were involved, each consuming 50% of the CPU: Iexplore.exe and Dllhost.exe. Iexplore is Internet Explorer (IE) and I suspected that IE itself wasn’t the problem, but that it was a browser helper object (BHO), ActiveX control, or some other plugin loaded into IE. Similarly, Dllhost.exe is the host process for out-of-process COM server DLLs, so it was probably not at fault, but the COM server loaded into it. Both required digging deeper and I decided to tackle IE first.
In order to try and get some CPU headroom in which to operate, I suspended the Dllhost process by selecting it in Process Explorer, right-clicking to open the process context menu, and selecting the Suspend entry:
That put the Dllhost process to sleep and, as I expected, that freed up 50% of the CPU. That’s because the computer was a dual-core system and so to consume 100% of the available CPU cycles a process would have to have two threads, each hogging one of the cores. Most bugs I've seen that result in the CPU being pegged are caused by a single thread.
Processes don’t execute code, threads do, so I needed to look inside the IE process to see what thread or threads were running. I double-clicked on Iexplore.exe in Process Explorer to open its process properties dialog and switched to the Threads page. Several threads were running, but one was dominating the CPU:
From past experience I knew that Ieframe.dll was part of IE, but to be sure I clicked on the modules button on the Threads tab of the Properties dialog and switched to the Details page of the resulting Shell properties dialog:
The description didn't give me a clue as the thread's specific purpose, so I moved to the second clue about the thread, its start function. Because I had configured Process Explorer to retrieve symbols for Windows images from the Microsoft symbol server in Options->Configure Symbols, Process Explorer showed the name of the function where each thread began executing. Sometimes the DLL or function where a thread starts executing is enough to identify the thread’s purpose or the software causing a problem. In this case, the thread began in a function named CTablWindow::_TabWindowThreadProc. The function name hints that it’s the one in which the main thread of a tab starts running, but that still wasn’t enough to tell me why the thread was running so much; I needed to dig even deeper and look inside the thread to see where it was executing.
To look at what the thread was up to, I double-clicked on it in the Threads list to open the Thread Stack dialog, which shows the functions on the thread’s stack. A stack is essentially an execution history, where each function listed called the one above it on the list and the function at the top of the list is the one most recently executed by the thread at the time of Process Explorer looks at the stack. I scrolled through the list, looking for frames that referenced 3rd-party DLLs or Microsoft IE plugins, since they would be far more likely to have a bug than IE’s own code. Sure enough, I found frames pointing at a popular 3rd-party ActiveX control, Adobe Flash:
Just to be sure that I hadn’t happened to catch Flash running when a different component was using most of the CPU time, I closed and reopened the stack dialog several times, but all of them pointed at Flash.
The first thing I do when I suspect that some software is causing a problem is to check the vendor’s web site to make sure that I have the latest version. I opened the Process Explorer DLL view and looked at Flash.ocx’s version, went to Adobe’s site and looked at the version of the current Flash download, and they were the same.
I was at a dead end. I couldn’t know for sure if Flash had a bug or, more likely, there was a Flash application that had a bug, nor could I be sure that the problem wouldn’t recur. I tried to determine which site was hosting the Flash content by closing tabs one by one, but when I had close them all the thread was still running.
At this point the only options I had were to uninstall Flash and leave my wife with a degraded web experience, or terminate IE to stop the current CPU usage and hope that it wouldn’t happen again. I chose the latter and the case remains open. Since investigating this I’ve seen the same Flash behavior again on my wife’s system and on my own, so have been vigilantly watching the Adobe site for a new version just in case its due to a bug in Flash itself. I was disappointed that there was no actionable result of the investigation, but at least I knew what had caused the CPU usage.
I now turned my attention the Dllhost problem with the hope that I'd meet with better success. Process Explorer lists in a tooltip the component or components loaded into hosting processes like Svchost.exe (the Windows service host process), Rundll32 (the Control Panel applet hosting process), Taskeng.exe (the scheduled task hosting process on Vista and Server 2008), and Dllhost.exe. I moved the mouse over Dllhost.exe to see what COM server it was running:
It was running the Thumbnail Cache COM server, whose job it is to create Explorer thumbnails for image and media files. It is part of Windows, so once again I had to look inside the process for more clues. I resumed the Dllhost process I had suspended earlier and opened the process properties threads page:
The thread consuming the most CPU in this case started in Quartz.dll’s ObjectThread function. I looked at its properties and saw that it was another Windows DLL, the DirectShow Runtime, with a generic function name:
Next, I double-clicked to look at the thread stack:
The first few frames were in User32.dll and Ntdll.dll, core Windows system DLLs, but frames 4-7 are in the Sonicmp4demux.ax (".ax" is an extension commonly used for DirectShow filters), a 3rd-party component. The function names for those frames were the same and didn't make sense because the Microsoft symbol server only stores symbols for software included in Windows. Several more stack snapshots confirmed that it was the code causing the CPU usage.
Now that I had my suspect, the next step was to check for a newer version. But first I had to figure out what software the DLL came with, which was harder than it seemed. I opened the DLL view to take a closer look at the version information, but the description didn't reveal anything:
There were no folders in the Start menu or items in the Add/Remove Programs list with Sonic in the name. I Windows-Live-searched (I expect that word to be added to Webster's any day now) for Sonic and found that it's part of the Roxio's CD and DVD authoring software suites. I looked in the start menu and sure enough, found a Roxio folder:
I ran the Roxio software to check its version number and discovered that the Creator application includes a built-in facility to check for updates. I ran it, but it came up empty:
I checked the Roxio web site just to be sure and it turned out there was a newer version that the built-in updater hadn't offered, perhaps because the update, according to the page, didn't offer anything new:
I downloaded it anyway (all 640MB of it!) and waited the 15 or so minutes for it to install. Then I checked the version information of Sonicmp4demux.ax to see if it was newer, but its version number, 1.4.402.60802, was the same as the one I'd seen in the DLL view and the file was two years old:
I could have uninstalled the software, which would ensure that the problem wouldn't return, but I wanted to keep Roxio for its DVD authoring functionality. I didn't care if I didn't get thumbnails for Roxio-specific image formats - I wasn't even sure there were any I'd ever see in Explorer - so I set out to see if I could disable just the Sonic demultiplexer. I could have searched the Registry for the DLL name, which is surely where it was registered, but that's a brute-force approach and if there were indirect or multiple references I could easily end up disabling more than just its thumbnail generation and possibly breaking something in Windows.
Process Monitor was the perfect tool for the job. Because I didn't know when the problem might reoccur - it might takes days to reproduce - I didn't want to just run it and let it consume all available virtual memory or disk space, so I set the History Depth in the Options menu to have Process Monitor retain only the most recent 1 million events:
I also set an Include filter for paths matching C:\Windows\System32\Dllhost.exe, minimized it, and let my wife have the system back.
The next day I came home from work, sat down at the computer and saw from Process Explorer that Dllhost.exe was back at it, consuming 50% of the CPU. I suspect that because it's a dual-core system, the problem had been showing up regularly, but my wife hadn't noticed it because the remaining CPU capacity was enough to mask it (another good reason to buy multi-core processors!). I brought Process Monitor to the foreground and noted it had seen 114,000 Dllhost operations, which was obviously way too many to scan through individually. I searched for "sonicmp4" and found a reference in a Registry query near the end of the trace:
The query is of a COM object registration for the demultiplexer. Because the COM object is a 3rd-party DLL, I was certain that that COM Class ID (CLSID) isn't hard-coded into Windows, so I went back to the first entry in the trace and searched for "A7DD215", the first few characters of the CLSID. The search found a match a few thousand operations earlier:
The CLSID was in the name of a Registry key under another COM object registration. I Windows-Live-searched (that just rolls off the tongue, doesn't it?) for the parent CLSID and found this KB article that explains that the registry key is where DirectShow filters register: http://msdn.microsoft.com/en-us/library/ms787560(VS.85).aspx I took a look at the stack for the particular query to confirm that's the reason Dllhost was reading from there:
I was now confident that I could simply rename the Sonic filter registration key to prevent its use. I never delete registry keys when performing this kind of troubleshooting just in case the change disables important functionality or somehow breaks something else. I had seen from the traces that the thumbnail cache generator had come across an AVI file that caused it to load the Sonic demultiplexer, a format Windows is obviously able to handle on its own, so I was pretty sure things would continue to work. After terminating the Dllhost and making the change, I browsed to the same folder, deleted the thumbnails, and confirmed that there was no reduced functionality as far as I could tell. I then used Roxio to successfully burn a DVD with a number of AVI files. This case was closed.
My wife's system was now usable again, and though I wasn't able to close the Flash-related part of the case, at least I knew the cause and could keep an eye out for updates. More importantly, by solving the Dllhost part of the case, even if Flash went crazy again, her system would still be usable and she wouldn't be filing a critical support incident for it with me - thanks to Process Explorer and Process Monitor.
Danno hit the nail on the head. I was going to submit a similar post but he pretty much covered the multi-thread OS concept. No "robust" OS should let ANY process run away with the CPU the way, at least, Microsoft OSs have permitted and apparently still do (due to a faulty and obsolete scheduler implementation) so that no thread gets starved of the CPU for 30 seconds. If a thread hasn't finished its work within its timeslice, it must wait its turn for another timeslice to resume work, barring some reasonable priority scheme (not 1/2 minute).
With the speed of modern CPUs, the user would typically not notice the CPU hog since they would get a timely response from the OS. These concepts have been around for years (before Windows & even DOS) so Windows has no excuse for not having better task management.
I hate it when I click on something like the Start Menu and nothing happens. So you click it again, and wait...and wait, and then, the Start Menu pops-up and then disappears as it executes both clicks, although I've never had to wait 30 seconds. And yes, I have even learned to click it a third time so that the menu will be visible when the OS finally gets around to me.
@Dr. House, Hughe, maybe Denise Miller, or more likely J@CKA$$:
Before ripping into somebody for making a mistake, make sure you know what you are talking about. Exasperates and exacerbates have practically the same definitions and are synonyms, so Yahoo! (I had to throw them in there for fairness/balance) synonyms and apologize.
Also, don't lie and say you don't care. You cared enough to post didn't you?
BTW, (<--acronym around before electronics so I'll assume it somehow isn't considered English language castration by you) there is no such thing as "high-water pants" anymore. The kids now call these shorts:)
"Exasperates and exacerbates have practically the same definitions and are synonyms"
This is exactly the sort of exasperating mis-"information" that exacerbates the growing problem with careless and imprecise use of language. :)
Exacerbate: to make a situation worse; more violent, bitter, or severe
Exasperate: to annoy or provoke anger
Agree with you, though, about "Hughe". His diatribe is now immortalized in my collection.
...Great article, Mark.
I've been a big fan of Mark's since I discovered Winterals years ago. I love the logical organization of his apps in areas which MS has always needed major help with. (though I've never been able to get your access group dumping app to work in my domains <cringe in frustration>).
After reading a number of these comments after Mark's story I think we have to say that Windows's memory management needs help. Not only should we be able to run a memory gatekeeper app which has a set of rules that looks for issues like runaway CPU hogging processes which has self identification (loops of 1000's of cycles that never end), you can drill through various categories like the stack flow, dll cpu process levels, etc, when they're happening. But also, a way to identify memory allocation to components to identify excess, a 100% detailed and accurate breakdown of how much memory Windows is allocating at start and after apps are run, so you can see why windows needs 800 MB on a fresh startup, but after an hour why is it at 1400 MB, or whatever the issue.
I wish there were companies that would produce memory management subsystems which we could swap in as a superset to correct these problems or help granularly diagnose them.
As far as I'm concerned Mark has done a great job at unveiling some of the mystery with Windows subsystems with his apps. Maybe someone at MS will get smart and make him a project manager of some of these subsystems so the next version of Windows can finally improve it's efficiency rather than just cosmetically.
My computer has been horribly slow for the past few days. The mouse was sluggish and, as noted above, pressing buttons resulted in an extended delay before anything happened. The keyboard kept missing letters when I typed.
I usually keep the Task Manager open, so I can keep an eye on things, and am therefore familiar with the processes which "should" be running under normal circumstances. Over the past few days I noticed the appearance of the aforementioned "dllhost", which I don't ever remember seeing previously.
Noticing the high CPU usage of Firefox, I decided to change the priority to Below Normal. However, I was surprised to find the priority already set to High. I changed it to Low. Checked again, and it was back at High. Tried several more times, with the same result. WTF??? Nothing like this happened before.
So what changed a few days ago?
I installed Silverlight.
Could this be the answer? Bet yer house it could!
After uninstalling Silverlight, everything is back to its normal speed. Firefox no longer insists on running at Above Normal priority, the mouse zooms around the screen as it should, and the keyboard is functioning correctly.
What the hell are MS playing at with this nonsense?
This is really a great info for professionals. Applications like Flash, Roxio, AOL etc are the culprits for Windows bad behavior. Unfortunately that is not what others see it. So, I wish Windows team would properly handle such scenario and a threshold alert say 70% CPU. If it goes beyond 70%, let Windows report the info you collected using all your knowledge and the tools to the end user. This helps a lot.
Even in linux (Ubuntu, Fedora ...) I am seeing flash taking 100% cpu time.
Why blame windows for it?
Hope you're still monitoring this. This is kind of an appropriate place to post it. There is some kind of a bug or "feature" (clearly not working right) that makes CPU usage on my laptop (and other people's machines) to spike heavily thanks to explorer.exe (windows explorer process) when there is My Web Sites on MSN shortcut present in the system. I don't know where it's from, and once moved to recycle bin the usage of explorer.exe goes from 50% on my new Vista x64 installation to something like 1-3%.
Call stack shows all MS dlls
I am sorry for english.
after all years, A giant like microsoft is developing an OS which is Windows and it is getting slower or crashing because of stupid web pages those high loaded with flash files, or the registry is getting corrupted because of bad un/installations,etc.
What the hack thousands of Microsoft OS developer are doing ? are they know thinking ? or designing ? or microsoft management is sleeping ?
It is shame , because that BIG VISTA is getting responceless because of stupid Flash files in the web pages ?
I am not anti windowser or lover of mac or lover of linux.
I just want please, drink a black coffee and think and design a working OS, not the Kinder Garden OS.
Interesting, I'm now running the Windows 7 Beta and i'm experiencing a similar perf hit in explorer when viewing folders with AVI's in them. Seems like there is a rogue filter, which seems like it is also affecting Media Player and Media Centres performance enumerating files.
If you are running Flash 10 and still experiencing the problem, then drop back to Flash player 184.108.40.206.
Regarding the 'Sonic' application in the article, this is not actually part of the Roxio software package.
I believe Roxio bought them out or something, I dunno exactly the deal there.
You got the Sonic software most likely as part of the software with the computer, I'm guessing you had a HP or Compaq.
If you look for updates to Sonic it just kicks you to the Roxio page, my assumption is that Sonic is legacy product & they'd prefer you just remove Sonic & install Roxio.
You might be able to get an update by going to HP/Compaq and doing a software update instead of trying to do the app by itself.
Surely the correct way to fix IE is to install Firefox or its variants, and flashblock? Then you can ONLY allow flash you WANT to run, saving your DL as well.
Wow. Slow Vista System? Let's go through twenty steps no normal human could ever accomplish, then decide that simply solving half of the issue is good enough because you've got enough raw hardware that the user won't notice.
Why not just get a quad core? Then you can go another two years without fixing a Windows issue.
i run flash in firefox on linux and periodically have seen similar problems stalling the browser. in a fit of youtube click-itis i may see several tabs with flash. this has not been so much a problem since getting the 64bit native flash plugin that they finally released. previously i had been using nspluginwrapper to run the 32bit version. the usage still maxes some times and the browser becomes unresponisve, but the other issues i had are gone. with the 32bit plug-in, some times after a while the tabs will not load the flash without a few reloads, and other times all the flash content dies leaving grey boxes in their place on every tab that had some.
sure wish flash were not closed source ;)