Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago my wife complained that her Vista desktop was not responding to her typing or mouse clicks. Given the importance of the customer, I immediately sat down at the system to troubleshoot. It wasn’t completely hung, but extremely sluggish. For example, the mouse moved and when I clicked on the start button the start menu opened after about 30 seconds. I suspected that something was hogging the CPU and likely could have resolved the problem simply by logging off or rebooting, but knew that if I didn’t determine the root cause and address it, she’d likely be calling on my technical support services again in the near future. In any case, stooping to that kind of troubleshooting hack is beneath my dignity. I therefore set out to investigate.
My first step was to run Process Explorer to see which process was using the CPU. After a few minutes Process Explorer finally appeared and showed that not one, but two processes were involved, each consuming 50% of the CPU: Iexplore.exe and Dllhost.exe. Iexplore is Internet Explorer (IE) and I suspected that IE itself wasn’t the problem, but that it was a browser helper object (BHO), ActiveX control, or some other plugin loaded into IE. Similarly, Dllhost.exe is the host process for out-of-process COM server DLLs, so it was probably not at fault, but the COM server loaded into it. Both required digging deeper and I decided to tackle IE first.
In order to try and get some CPU headroom in which to operate, I suspended the Dllhost process by selecting it in Process Explorer, right-clicking to open the process context menu, and selecting the Suspend entry:
That put the Dllhost process to sleep and, as I expected, that freed up 50% of the CPU. That’s because the computer was a dual-core system and so to consume 100% of the available CPU cycles a process would have to have two threads, each hogging one of the cores. Most bugs I've seen that result in the CPU being pegged are caused by a single thread.
Processes don’t execute code, threads do, so I needed to look inside the IE process to see what thread or threads were running. I double-clicked on Iexplore.exe in Process Explorer to open its process properties dialog and switched to the Threads page. Several threads were running, but one was dominating the CPU:
From past experience I knew that Ieframe.dll was part of IE, but to be sure I clicked on the modules button on the Threads tab of the Properties dialog and switched to the Details page of the resulting Shell properties dialog:
The description didn't give me a clue as the thread's specific purpose, so I moved to the second clue about the thread, its start function. Because I had configured Process Explorer to retrieve symbols for Windows images from the Microsoft symbol server in Options->Configure Symbols, Process Explorer showed the name of the function where each thread began executing. Sometimes the DLL or function where a thread starts executing is enough to identify the thread’s purpose or the software causing a problem. In this case, the thread began in a function named CTablWindow::_TabWindowThreadProc. The function name hints that it’s the one in which the main thread of a tab starts running, but that still wasn’t enough to tell me why the thread was running so much; I needed to dig even deeper and look inside the thread to see where it was executing.
To look at what the thread was up to, I double-clicked on it in the Threads list to open the Thread Stack dialog, which shows the functions on the thread’s stack. A stack is essentially an execution history, where each function listed called the one above it on the list and the function at the top of the list is the one most recently executed by the thread at the time of Process Explorer looks at the stack. I scrolled through the list, looking for frames that referenced 3rd-party DLLs or Microsoft IE plugins, since they would be far more likely to have a bug than IE’s own code. Sure enough, I found frames pointing at a popular 3rd-party ActiveX control, Adobe Flash:
Just to be sure that I hadn’t happened to catch Flash running when a different component was using most of the CPU time, I closed and reopened the stack dialog several times, but all of them pointed at Flash.
The first thing I do when I suspect that some software is causing a problem is to check the vendor’s web site to make sure that I have the latest version. I opened the Process Explorer DLL view and looked at Flash.ocx’s version, went to Adobe’s site and looked at the version of the current Flash download, and they were the same.
I was at a dead end. I couldn’t know for sure if Flash had a bug or, more likely, there was a Flash application that had a bug, nor could I be sure that the problem wouldn’t recur. I tried to determine which site was hosting the Flash content by closing tabs one by one, but when I had close them all the thread was still running.
At this point the only options I had were to uninstall Flash and leave my wife with a degraded web experience, or terminate IE to stop the current CPU usage and hope that it wouldn’t happen again. I chose the latter and the case remains open. Since investigating this I’ve seen the same Flash behavior again on my wife’s system and on my own, so have been vigilantly watching the Adobe site for a new version just in case its due to a bug in Flash itself. I was disappointed that there was no actionable result of the investigation, but at least I knew what had caused the CPU usage.
I now turned my attention the Dllhost problem with the hope that I'd meet with better success. Process Explorer lists in a tooltip the component or components loaded into hosting processes like Svchost.exe (the Windows service host process), Rundll32 (the Control Panel applet hosting process), Taskeng.exe (the scheduled task hosting process on Vista and Server 2008), and Dllhost.exe. I moved the mouse over Dllhost.exe to see what COM server it was running:
It was running the Thumbnail Cache COM server, whose job it is to create Explorer thumbnails for image and media files. It is part of Windows, so once again I had to look inside the process for more clues. I resumed the Dllhost process I had suspended earlier and opened the process properties threads page:
The thread consuming the most CPU in this case started in Quartz.dll’s ObjectThread function. I looked at its properties and saw that it was another Windows DLL, the DirectShow Runtime, with a generic function name:
Next, I double-clicked to look at the thread stack:
The first few frames were in User32.dll and Ntdll.dll, core Windows system DLLs, but frames 4-7 are in the Sonicmp4demux.ax (".ax" is an extension commonly used for DirectShow filters), a 3rd-party component. The function names for those frames were the same and didn't make sense because the Microsoft symbol server only stores symbols for software included in Windows. Several more stack snapshots confirmed that it was the code causing the CPU usage.
Now that I had my suspect, the next step was to check for a newer version. But first I had to figure out what software the DLL came with, which was harder than it seemed. I opened the DLL view to take a closer look at the version information, but the description didn't reveal anything:
There were no folders in the Start menu or items in the Add/Remove Programs list with Sonic in the name. I Windows-Live-searched (I expect that word to be added to Webster's any day now) for Sonic and found that it's part of the Roxio's CD and DVD authoring software suites. I looked in the start menu and sure enough, found a Roxio folder:
I ran the Roxio software to check its version number and discovered that the Creator application includes a built-in facility to check for updates. I ran it, but it came up empty:
I checked the Roxio web site just to be sure and it turned out there was a newer version that the built-in updater hadn't offered, perhaps because the update, according to the page, didn't offer anything new:
I downloaded it anyway (all 640MB of it!) and waited the 15 or so minutes for it to install. Then I checked the version information of Sonicmp4demux.ax to see if it was newer, but its version number, 1.4.402.60802, was the same as the one I'd seen in the DLL view and the file was two years old:
I could have uninstalled the software, which would ensure that the problem wouldn't return, but I wanted to keep Roxio for its DVD authoring functionality. I didn't care if I didn't get thumbnails for Roxio-specific image formats - I wasn't even sure there were any I'd ever see in Explorer - so I set out to see if I could disable just the Sonic demultiplexer. I could have searched the Registry for the DLL name, which is surely where it was registered, but that's a brute-force approach and if there were indirect or multiple references I could easily end up disabling more than just its thumbnail generation and possibly breaking something in Windows.
Process Monitor was the perfect tool for the job. Because I didn't know when the problem might reoccur - it might takes days to reproduce - I didn't want to just run it and let it consume all available virtual memory or disk space, so I set the History Depth in the Options menu to have Process Monitor retain only the most recent 1 million events:
I also set an Include filter for paths matching C:\Windows\System32\Dllhost.exe, minimized it, and let my wife have the system back.
The next day I came home from work, sat down at the computer and saw from Process Explorer that Dllhost.exe was back at it, consuming 50% of the CPU. I suspect that because it's a dual-core system, the problem had been showing up regularly, but my wife hadn't noticed it because the remaining CPU capacity was enough to mask it (another good reason to buy multi-core processors!). I brought Process Monitor to the foreground and noted it had seen 114,000 Dllhost operations, which was obviously way too many to scan through individually. I searched for "sonicmp4" and found a reference in a Registry query near the end of the trace:
The query is of a COM object registration for the demultiplexer. Because the COM object is a 3rd-party DLL, I was certain that that COM Class ID (CLSID) isn't hard-coded into Windows, so I went back to the first entry in the trace and searched for "A7DD215", the first few characters of the CLSID. The search found a match a few thousand operations earlier:
The CLSID was in the name of a Registry key under another COM object registration. I Windows-Live-searched (that just rolls off the tongue, doesn't it?) for the parent CLSID and found this KB article that explains that the registry key is where DirectShow filters register: http://msdn.microsoft.com/en-us/library/ms787560(VS.85).aspx I took a look at the stack for the particular query to confirm that's the reason Dllhost was reading from there:
I was now confident that I could simply rename the Sonic filter registration key to prevent its use. I never delete registry keys when performing this kind of troubleshooting just in case the change disables important functionality or somehow breaks something else. I had seen from the traces that the thumbnail cache generator had come across an AVI file that caused it to load the Sonic demultiplexer, a format Windows is obviously able to handle on its own, so I was pretty sure things would continue to work. After terminating the Dllhost and making the change, I browsed to the same folder, deleted the thumbnails, and confirmed that there was no reduced functionality as far as I could tell. I then used Roxio to successfully burn a DVD with a number of AVI files. This case was closed.
My wife's system was now usable again, and though I wasn't able to close the Flash-related part of the case, at least I knew the cause and could keep an eye out for updates. More importantly, by solving the Dllhost part of the case, even if Flash went crazy again, her system would still be usable and she wouldn't be filing a critical support incident for it with me - thanks to Process Explorer and Process Monitor.
I experience the same behavior with Flash frequently.
You're Insanely smart as Larry Osterman said and dear hubby :D.
I'm also digging into a sluggish performance problem with my nVdia 8800 card. It's using the latest Display Driver available for Windows XP. The hardware seems to be working fine with Vista (with it's default driver).
Anyway thanks a lot for your posts on trouble shooting and debugging.
Re. Flash: It could be because some lame Flash programmer wrote a polling loop in ActiveScript. No amount of version updates will get you out of it.
Re. sonicMP4Demux.dll: I presume you could just run "regsvr32 /u sonicMP4Demux.dll", and have it deregistered.
You should be able to unregister any DirectShow filter using regsvr32 /u.
regsvr32 /u Sonicmp4demux.ax
The name sounds like it's a demuxer for .MP4 format files, which is basically the same as Apple's QuickTime .mov format.
"I Windows-Live-searched (that just rolls off the tongue, doesn't it?)"
I prefer to call in learching. Since the live team didn't choose a name which could be used as a verb (aka google-ing) we need to get creative. Thus comes learching: Live sEARCH-ing. I prenounce (probably incorrectly) as lurching.
Mark Why is it that this flash thing does not suprise me?
Flash always hangs my machine. causes my IE to crash and etc. thanks god IE8 has tab recovery feature.
Given the importance of the customer... Ha!
Anyway, did anyone else do a double-take during the opening credits of the new Smallville.
Does Mark have just a passing resemblance to the Green Arrow?
Will read your article now...
Would it be possible to change the thread scheduling algorithm in order to improve the system's responsiveness in such circumstances (high cpu load)?
I was waiting for one of these articles to appear again! It was as entertaining and educational as always. Thanks a lot, Mark.
I was having alot of issues with high cpu loads on my core2duo system, and whenever i would look deeper into the problem that was causing it, it was almost always related to either a audio/video application hanging, or related to the "Multimedia Class Scheduler" service, thats new with vista, and came with SP1 i believe.
The biggest issues i seem to have are all related to the fact that i typically have ALOT of Directshow filters and plugins installed at any one time, because i do alot of video production, and have to take video from alot of different formats, and hack them to pieces and put them back together.
Its mostly due to the fact that i have to import alot of video that is not native to the program i am using, or I have to use a media splitter/demuxer to fool the application into importing the video at all.
I pretty much solved all my CPU hogging issues by doing this -
uninstall quicktime and anything that relies on quicktime, such as itunes
install K-lite codec package
install Quicktime Alternative
install the Quicktime-itunes addon, so that you can still run itunes and any other cranky quicktime dependant application without having quicktime actually installed.
you can get all those programs from www.codecguide.com
it is an absolute godsend for anyone who is needing to edit, process, or format video for many different devices, and seems to help alot with keeping media apps that are dependant on media splitting, demuxing, or importing through directshow from hanging and just burning up cpu cycles
So the real question, or at least the one I find most interesting, is this: when will the various bits of Windows/IE that so glibly host third-party code detect and reign in such excesses?
Great article Mark! What's the average customer to do however? Inevitably they will take their system back to where they bought it, the technicians won't find the problem but will reinstall the operating system. Everything will be fine until the customer re-installs the offending software and then the whole thing starts all over again.
Thanks for some insight on how to utilize Process Explorer better. I was always frustrated with finding out what addin causes IE to crash.
Mark, I think this case is interesting because would really really like to understand why those 2 processes were able to make the entire system almost unusable. Can you shed any light on the following questions:
Why does it take 30 seconds to open the start menu ? If there are 2 processes running, each of them taking up almost 100% of the CPU cycles of one core, why is the Windows scheduler not able to allocate some reasonable fraction of the CPU cycles to other processes, especially Explorer.exe ?
It seems utterly bizarre to me that one process running in a tight loop can make the whole machine almost unusable. Is it not reasonable to expect the scheduler to evenly split the CPU between processes or threads ?
I remember using a Sun-3 machine (Motorola 68020, running some version of Sun's Unix) back in 1989 and if there was a runaway process (or two), everything would slow down proportionally. So if you had 2 processes spinning in a tight loop, and you wanted to do something else, that "something else" would get 33% of the CPU cycles. That seems much better than giving almost all CPU cycles to those 2 processes and almost none to Explorer, so that the start menu takes 30 seconds to respond to a click.
I would really love to find out why Windows allocates the CPU cycles the way it does.
My wife had the same problem (the Flash problem). I switched her to Chrome because the Chrome task manager makes it very easy for her to identify a page with a wacky out of control Flash widget and kill it, or, alternatively, kill Flash.
This kind of problem is only going to increase as random people write poorly tested scripts that go into infinite loops. *Sigh*.