Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago my wife complained that her Vista desktop was not responding to her typing or mouse clicks. Given the importance of the customer, I immediately sat down at the system to troubleshoot. It wasn’t completely hung, but extremely sluggish. For example, the mouse moved and when I clicked on the start button the start menu opened after about 30 seconds. I suspected that something was hogging the CPU and likely could have resolved the problem simply by logging off or rebooting, but knew that if I didn’t determine the root cause and address it, she’d likely be calling on my technical support services again in the near future. In any case, stooping to that kind of troubleshooting hack is beneath my dignity. I therefore set out to investigate.
My first step was to run Process Explorer to see which process was using the CPU. After a few minutes Process Explorer finally appeared and showed that not one, but two processes were involved, each consuming 50% of the CPU: Iexplore.exe and Dllhost.exe. Iexplore is Internet Explorer (IE) and I suspected that IE itself wasn’t the problem, but that it was a browser helper object (BHO), ActiveX control, or some other plugin loaded into IE. Similarly, Dllhost.exe is the host process for out-of-process COM server DLLs, so it was probably not at fault, but the COM server loaded into it. Both required digging deeper and I decided to tackle IE first.
In order to try and get some CPU headroom in which to operate, I suspended the Dllhost process by selecting it in Process Explorer, right-clicking to open the process context menu, and selecting the Suspend entry:
That put the Dllhost process to sleep and, as I expected, that freed up 50% of the CPU. That’s because the computer was a dual-core system and so to consume 100% of the available CPU cycles a process would have to have two threads, each hogging one of the cores. Most bugs I've seen that result in the CPU being pegged are caused by a single thread.
Processes don’t execute code, threads do, so I needed to look inside the IE process to see what thread or threads were running. I double-clicked on Iexplore.exe in Process Explorer to open its process properties dialog and switched to the Threads page. Several threads were running, but one was dominating the CPU:
From past experience I knew that Ieframe.dll was part of IE, but to be sure I clicked on the modules button on the Threads tab of the Properties dialog and switched to the Details page of the resulting Shell properties dialog:
The description didn't give me a clue as the thread's specific purpose, so I moved to the second clue about the thread, its start function. Because I had configured Process Explorer to retrieve symbols for Windows images from the Microsoft symbol server in Options->Configure Symbols, Process Explorer showed the name of the function where each thread began executing. Sometimes the DLL or function where a thread starts executing is enough to identify the thread’s purpose or the software causing a problem. In this case, the thread began in a function named CTablWindow::_TabWindowThreadProc. The function name hints that it’s the one in which the main thread of a tab starts running, but that still wasn’t enough to tell me why the thread was running so much; I needed to dig even deeper and look inside the thread to see where it was executing.
To look at what the thread was up to, I double-clicked on it in the Threads list to open the Thread Stack dialog, which shows the functions on the thread’s stack. A stack is essentially an execution history, where each function listed called the one above it on the list and the function at the top of the list is the one most recently executed by the thread at the time of Process Explorer looks at the stack. I scrolled through the list, looking for frames that referenced 3rd-party DLLs or Microsoft IE plugins, since they would be far more likely to have a bug than IE’s own code. Sure enough, I found frames pointing at a popular 3rd-party ActiveX control, Adobe Flash:
Just to be sure that I hadn’t happened to catch Flash running when a different component was using most of the CPU time, I closed and reopened the stack dialog several times, but all of them pointed at Flash.
The first thing I do when I suspect that some software is causing a problem is to check the vendor’s web site to make sure that I have the latest version. I opened the Process Explorer DLL view and looked at Flash.ocx’s version, went to Adobe’s site and looked at the version of the current Flash download, and they were the same.
I was at a dead end. I couldn’t know for sure if Flash had a bug or, more likely, there was a Flash application that had a bug, nor could I be sure that the problem wouldn’t recur. I tried to determine which site was hosting the Flash content by closing tabs one by one, but when I had close them all the thread was still running.
At this point the only options I had were to uninstall Flash and leave my wife with a degraded web experience, or terminate IE to stop the current CPU usage and hope that it wouldn’t happen again. I chose the latter and the case remains open. Since investigating this I’ve seen the same Flash behavior again on my wife’s system and on my own, so have been vigilantly watching the Adobe site for a new version just in case its due to a bug in Flash itself. I was disappointed that there was no actionable result of the investigation, but at least I knew what had caused the CPU usage.
I now turned my attention the Dllhost problem with the hope that I'd meet with better success. Process Explorer lists in a tooltip the component or components loaded into hosting processes like Svchost.exe (the Windows service host process), Rundll32 (the Control Panel applet hosting process), Taskeng.exe (the scheduled task hosting process on Vista and Server 2008), and Dllhost.exe. I moved the mouse over Dllhost.exe to see what COM server it was running:
It was running the Thumbnail Cache COM server, whose job it is to create Explorer thumbnails for image and media files. It is part of Windows, so once again I had to look inside the process for more clues. I resumed the Dllhost process I had suspended earlier and opened the process properties threads page:
The thread consuming the most CPU in this case started in Quartz.dll’s ObjectThread function. I looked at its properties and saw that it was another Windows DLL, the DirectShow Runtime, with a generic function name:
Next, I double-clicked to look at the thread stack:
The first few frames were in User32.dll and Ntdll.dll, core Windows system DLLs, but frames 4-7 are in the Sonicmp4demux.ax (".ax" is an extension commonly used for DirectShow filters), a 3rd-party component. The function names for those frames were the same and didn't make sense because the Microsoft symbol server only stores symbols for software included in Windows. Several more stack snapshots confirmed that it was the code causing the CPU usage.
Now that I had my suspect, the next step was to check for a newer version. But first I had to figure out what software the DLL came with, which was harder than it seemed. I opened the DLL view to take a closer look at the version information, but the description didn't reveal anything:
There were no folders in the Start menu or items in the Add/Remove Programs list with Sonic in the name. I Windows-Live-searched (I expect that word to be added to Webster's any day now) for Sonic and found that it's part of the Roxio's CD and DVD authoring software suites. I looked in the start menu and sure enough, found a Roxio folder:
I ran the Roxio software to check its version number and discovered that the Creator application includes a built-in facility to check for updates. I ran it, but it came up empty:
I checked the Roxio web site just to be sure and it turned out there was a newer version that the built-in updater hadn't offered, perhaps because the update, according to the page, didn't offer anything new:
I downloaded it anyway (all 640MB of it!) and waited the 15 or so minutes for it to install. Then I checked the version information of Sonicmp4demux.ax to see if it was newer, but its version number, 1.4.402.60802, was the same as the one I'd seen in the DLL view and the file was two years old:
I could have uninstalled the software, which would ensure that the problem wouldn't return, but I wanted to keep Roxio for its DVD authoring functionality. I didn't care if I didn't get thumbnails for Roxio-specific image formats - I wasn't even sure there were any I'd ever see in Explorer - so I set out to see if I could disable just the Sonic demultiplexer. I could have searched the Registry for the DLL name, which is surely where it was registered, but that's a brute-force approach and if there were indirect or multiple references I could easily end up disabling more than just its thumbnail generation and possibly breaking something in Windows.
Process Monitor was the perfect tool for the job. Because I didn't know when the problem might reoccur - it might takes days to reproduce - I didn't want to just run it and let it consume all available virtual memory or disk space, so I set the History Depth in the Options menu to have Process Monitor retain only the most recent 1 million events:
I also set an Include filter for paths matching C:\Windows\System32\Dllhost.exe, minimized it, and let my wife have the system back.
The next day I came home from work, sat down at the computer and saw from Process Explorer that Dllhost.exe was back at it, consuming 50% of the CPU. I suspect that because it's a dual-core system, the problem had been showing up regularly, but my wife hadn't noticed it because the remaining CPU capacity was enough to mask it (another good reason to buy multi-core processors!). I brought Process Monitor to the foreground and noted it had seen 114,000 Dllhost operations, which was obviously way too many to scan through individually. I searched for "sonicmp4" and found a reference in a Registry query near the end of the trace:
The query is of a COM object registration for the demultiplexer. Because the COM object is a 3rd-party DLL, I was certain that that COM Class ID (CLSID) isn't hard-coded into Windows, so I went back to the first entry in the trace and searched for "A7DD215", the first few characters of the CLSID. The search found a match a few thousand operations earlier:
The CLSID was in the name of a Registry key under another COM object registration. I Windows-Live-searched (that just rolls off the tongue, doesn't it?) for the parent CLSID and found this KB article that explains that the registry key is where DirectShow filters register: http://msdn.microsoft.com/en-us/library/ms787560(VS.85).aspx I took a look at the stack for the particular query to confirm that's the reason Dllhost was reading from there:
I was now confident that I could simply rename the Sonic filter registration key to prevent its use. I never delete registry keys when performing this kind of troubleshooting just in case the change disables important functionality or somehow breaks something else. I had seen from the traces that the thumbnail cache generator had come across an AVI file that caused it to load the Sonic demultiplexer, a format Windows is obviously able to handle on its own, so I was pretty sure things would continue to work. After terminating the Dllhost and making the change, I browsed to the same folder, deleted the thumbnails, and confirmed that there was no reduced functionality as far as I could tell. I then used Roxio to successfully burn a DVD with a number of AVI files. This case was closed.
My wife's system was now usable again, and though I wasn't able to close the Flash-related part of the case, at least I knew the cause and could keep an eye out for updates. More importantly, by solving the Dllhost part of the case, even if Flash went crazy again, her system would still be usable and she wouldn't be filing a critical support incident for it with me - thanks to Process Explorer and Process Monitor.
Great, thorough and amusing article,as ever, but what I find can work better with a CPU hogging process is to set its base priority to "Low" in Task Manager, or "Idle" in Process Explorer, so that its threads keep running but if any other thread becomes ready to run, it should get on a processor in preference to the Low priority threads of the wayward task since they will almost certainly be at a higher base priority (typically "Normal").
The reason that the system became unusable in this situation is due to the fundamental way that the NT scheduler has worked since NT4, for good or for bad. You can read the gory details in my book "Windows Internals", but I'll give a brief explaination here.
By default, threads run at priority 8, but get priority boosts for short periods when they wake up from sleeping or waiting and when they get a windows message and they are their window is the foreground.
What happened on my system was that the flash and dllhost threads had priorities set to "above normal", which is 9 for threads of processes running in the default process priority class.
When I unlocked the screen, clicked on the start menu, and interacted with other processes running at the default priority of 8, their main window thread got a short boost to 12 and then got dropped back to 8. At that point it was below the priorities of the active threads. Windows includes an anti-starvation priority boost that kicks in every several seconds and temporarily boosts the starved thread's priority to 15.
The long delay to see a response to input was therefore the result of the window's thread finally getting enough of those short priority 15 boosts to display it fully. In the case of the start menu, that was around 30 seconds.
Yes, it's not great, but it's been that way for 13 years across many OS releases - NT4, Win2K, XP, Server 2003, Vista and Server 2008 - and you'll only see this kind of problem in special circumstances - like I did. That said, we're always looking at things like this to see how we can improve it while keeping in mind the problems of application compatibility and avoiding unintendend side effects.
I wish it was always that easy :)
I'm still trying to figure out why my machine gets slower over time, without any obvious culprits in process explorer. (I.e. total CPU usage < 10%, plenty of phys. mem. available, and yet, sluggishness)
So I'll just go ahead and ask if you've got any ideas what to look for. (I mean, if *you* don't know, who would? ;)
Mark, you state that "only see this kind of problem in special circumstances", but with Flash and 3rd party codecs becoming so prevalent, this is something that I see several times per month on different systems. Both on XP and on Vista. For servers I agree: it generally doesn't happen all that often. But for the less controlled environment of a personal workstation, it happens very often.
This is a great example of how Vista has failed to address the sheer and growing complexity of the Windows environment.
Most of us geeks would have failed to get as far as you did, so what is the non-geek down the street to do?
Rebooting and encountering the same problems over and over again is hardly a customer experience that will engender loyalty to the platform.
I love your posts, man! Process Explorer is a wonderful tool!
Mark, you're too kind. I would have kicked Roxio off my systems for good soon as I realized I need to hack the registry to fix things.
Thanks for the additional explanation that makes it clear why you experienced the behavior you described. Do you know why these threads were running with priority "Above Normal." I tried running a webpage with Flash (youtube) on my Win XP system and the base priority for the IE process was 8. Is there something different in Vista that raises the base priority for multimedia applications? Also when I was checking this I noticed that Process Explorer had a base priority of 13. Is this something you added to help keep Process Explorer responsive when troubleshooting these kinds of problems?
>Do you know why these threads were running with priority "Above Normal."
No, but it's not uncommon for programmers to program their threads with higher priorities with the intention of making their operation more responsive. In most cases programmers should leave their threads at the default priority.
Yes, Process Explorer sets its priority to "High" so that it can operate even when threads like the ones in this case are in CPU-bound loops.
Thanks for taking the time to explain this with the screenshots. Much appreciated. A great post.
So Vista was slow. No news there.
Q: Doctor doctor, my Vista is slow.
A: Don't use it then.
The take home point here would be that if you really insist suffering Vista, you'd better marry someone bright bloke like Russinovich to trouble-shoot it every other day.
Really good and detailed investigation as always Mark.
Thank you so much for sharing your experience with everyone.
>No, but it's not uncommon for programmers to >program their threads with higher priorities >with the intention of making their operation >more responsive. In most cases programmers >should leave their threads at the default >priority.
So this particully problem (dllhost.exe) was actually cause by two different bugs.
1. The 3. party codec could hang in an endless loop.
2. The priority was incorrectly "above normal", especially for a background thumbnail updater.
But is bug 2 actully a problem in windows ?
Nothing in this case was Vista-specific.
Vista is great if you have the hardware, but troubleshooting it is as difficult as XP. I am just a Windows experienced user and I doubt I would be capable of using PE like Mark. If I would have found such a problem and if nothing was logged in event viewer, I would start excluding third-party entries with msconfig (or with msconfig in combination with autoruns), and once I have found the problematic add-in, I would have updated/downgraded the program or simply discarded it, replacing it for another.