Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago my wife complained that her Vista desktop was not responding to her typing or mouse clicks. Given the importance of the customer, I immediately sat down at the system to troubleshoot. It wasn’t completely hung, but extremely sluggish. For example, the mouse moved and when I clicked on the start button the start menu opened after about 30 seconds. I suspected that something was hogging the CPU and likely could have resolved the problem simply by logging off or rebooting, but knew that if I didn’t determine the root cause and address it, she’d likely be calling on my technical support services again in the near future. In any case, stooping to that kind of troubleshooting hack is beneath my dignity. I therefore set out to investigate.
My first step was to run Process Explorer to see which process was using the CPU. After a few minutes Process Explorer finally appeared and showed that not one, but two processes were involved, each consuming 50% of the CPU: Iexplore.exe and Dllhost.exe. Iexplore is Internet Explorer (IE) and I suspected that IE itself wasn’t the problem, but that it was a browser helper object (BHO), ActiveX control, or some other plugin loaded into IE. Similarly, Dllhost.exe is the host process for out-of-process COM server DLLs, so it was probably not at fault, but the COM server loaded into it. Both required digging deeper and I decided to tackle IE first.
In order to try and get some CPU headroom in which to operate, I suspended the Dllhost process by selecting it in Process Explorer, right-clicking to open the process context menu, and selecting the Suspend entry:
That put the Dllhost process to sleep and, as I expected, that freed up 50% of the CPU. That’s because the computer was a dual-core system and so to consume 100% of the available CPU cycles a process would have to have two threads, each hogging one of the cores. Most bugs I've seen that result in the CPU being pegged are caused by a single thread.
Processes don’t execute code, threads do, so I needed to look inside the IE process to see what thread or threads were running. I double-clicked on Iexplore.exe in Process Explorer to open its process properties dialog and switched to the Threads page. Several threads were running, but one was dominating the CPU:
From past experience I knew that Ieframe.dll was part of IE, but to be sure I clicked on the modules button on the Threads tab of the Properties dialog and switched to the Details page of the resulting Shell properties dialog:
The description didn't give me a clue as the thread's specific purpose, so I moved to the second clue about the thread, its start function. Because I had configured Process Explorer to retrieve symbols for Windows images from the Microsoft symbol server in Options->Configure Symbols, Process Explorer showed the name of the function where each thread began executing. Sometimes the DLL or function where a thread starts executing is enough to identify the thread’s purpose or the software causing a problem. In this case, the thread began in a function named CTablWindow::_TabWindowThreadProc. The function name hints that it’s the one in which the main thread of a tab starts running, but that still wasn’t enough to tell me why the thread was running so much; I needed to dig even deeper and look inside the thread to see where it was executing.
To look at what the thread was up to, I double-clicked on it in the Threads list to open the Thread Stack dialog, which shows the functions on the thread’s stack. A stack is essentially an execution history, where each function listed called the one above it on the list and the function at the top of the list is the one most recently executed by the thread at the time of Process Explorer looks at the stack. I scrolled through the list, looking for frames that referenced 3rd-party DLLs or Microsoft IE plugins, since they would be far more likely to have a bug than IE’s own code. Sure enough, I found frames pointing at a popular 3rd-party ActiveX control, Adobe Flash:
Just to be sure that I hadn’t happened to catch Flash running when a different component was using most of the CPU time, I closed and reopened the stack dialog several times, but all of them pointed at Flash.
The first thing I do when I suspect that some software is causing a problem is to check the vendor’s web site to make sure that I have the latest version. I opened the Process Explorer DLL view and looked at Flash.ocx’s version, went to Adobe’s site and looked at the version of the current Flash download, and they were the same.
I was at a dead end. I couldn’t know for sure if Flash had a bug or, more likely, there was a Flash application that had a bug, nor could I be sure that the problem wouldn’t recur. I tried to determine which site was hosting the Flash content by closing tabs one by one, but when I had close them all the thread was still running.
At this point the only options I had were to uninstall Flash and leave my wife with a degraded web experience, or terminate IE to stop the current CPU usage and hope that it wouldn’t happen again. I chose the latter and the case remains open. Since investigating this I’ve seen the same Flash behavior again on my wife’s system and on my own, so have been vigilantly watching the Adobe site for a new version just in case its due to a bug in Flash itself. I was disappointed that there was no actionable result of the investigation, but at least I knew what had caused the CPU usage.
I now turned my attention the Dllhost problem with the hope that I'd meet with better success. Process Explorer lists in a tooltip the component or components loaded into hosting processes like Svchost.exe (the Windows service host process), Rundll32 (the Control Panel applet hosting process), Taskeng.exe (the scheduled task hosting process on Vista and Server 2008), and Dllhost.exe. I moved the mouse over Dllhost.exe to see what COM server it was running:
It was running the Thumbnail Cache COM server, whose job it is to create Explorer thumbnails for image and media files. It is part of Windows, so once again I had to look inside the process for more clues. I resumed the Dllhost process I had suspended earlier and opened the process properties threads page:
The thread consuming the most CPU in this case started in Quartz.dll’s ObjectThread function. I looked at its properties and saw that it was another Windows DLL, the DirectShow Runtime, with a generic function name:
Next, I double-clicked to look at the thread stack:
The first few frames were in User32.dll and Ntdll.dll, core Windows system DLLs, but frames 4-7 are in the Sonicmp4demux.ax (".ax" is an extension commonly used for DirectShow filters), a 3rd-party component. The function names for those frames were the same and didn't make sense because the Microsoft symbol server only stores symbols for software included in Windows. Several more stack snapshots confirmed that it was the code causing the CPU usage.
Now that I had my suspect, the next step was to check for a newer version. But first I had to figure out what software the DLL came with, which was harder than it seemed. I opened the DLL view to take a closer look at the version information, but the description didn't reveal anything:
There were no folders in the Start menu or items in the Add/Remove Programs list with Sonic in the name. I Windows-Live-searched (I expect that word to be added to Webster's any day now) for Sonic and found that it's part of the Roxio's CD and DVD authoring software suites. I looked in the start menu and sure enough, found a Roxio folder:
I ran the Roxio software to check its version number and discovered that the Creator application includes a built-in facility to check for updates. I ran it, but it came up empty:
I checked the Roxio web site just to be sure and it turned out there was a newer version that the built-in updater hadn't offered, perhaps because the update, according to the page, didn't offer anything new:
I downloaded it anyway (all 640MB of it!) and waited the 15 or so minutes for it to install. Then I checked the version information of Sonicmp4demux.ax to see if it was newer, but its version number, 1.4.402.60802, was the same as the one I'd seen in the DLL view and the file was two years old:
I could have uninstalled the software, which would ensure that the problem wouldn't return, but I wanted to keep Roxio for its DVD authoring functionality. I didn't care if I didn't get thumbnails for Roxio-specific image formats - I wasn't even sure there were any I'd ever see in Explorer - so I set out to see if I could disable just the Sonic demultiplexer. I could have searched the Registry for the DLL name, which is surely where it was registered, but that's a brute-force approach and if there were indirect or multiple references I could easily end up disabling more than just its thumbnail generation and possibly breaking something in Windows.
Process Monitor was the perfect tool for the job. Because I didn't know when the problem might reoccur - it might takes days to reproduce - I didn't want to just run it and let it consume all available virtual memory or disk space, so I set the History Depth in the Options menu to have Process Monitor retain only the most recent 1 million events:
I also set an Include filter for paths matching C:\Windows\System32\Dllhost.exe, minimized it, and let my wife have the system back.
The next day I came home from work, sat down at the computer and saw from Process Explorer that Dllhost.exe was back at it, consuming 50% of the CPU. I suspect that because it's a dual-core system, the problem had been showing up regularly, but my wife hadn't noticed it because the remaining CPU capacity was enough to mask it (another good reason to buy multi-core processors!). I brought Process Monitor to the foreground and noted it had seen 114,000 Dllhost operations, which was obviously way too many to scan through individually. I searched for "sonicmp4" and found a reference in a Registry query near the end of the trace:
The query is of a COM object registration for the demultiplexer. Because the COM object is a 3rd-party DLL, I was certain that that COM Class ID (CLSID) isn't hard-coded into Windows, so I went back to the first entry in the trace and searched for "A7DD215", the first few characters of the CLSID. The search found a match a few thousand operations earlier:
The CLSID was in the name of a Registry key under another COM object registration. I Windows-Live-searched (that just rolls off the tongue, doesn't it?) for the parent CLSID and found this KB article that explains that the registry key is where DirectShow filters register: http://msdn.microsoft.com/en-us/library/ms787560(VS.85).aspx I took a look at the stack for the particular query to confirm that's the reason Dllhost was reading from there:
I was now confident that I could simply rename the Sonic filter registration key to prevent its use. I never delete registry keys when performing this kind of troubleshooting just in case the change disables important functionality or somehow breaks something else. I had seen from the traces that the thumbnail cache generator had come across an AVI file that caused it to load the Sonic demultiplexer, a format Windows is obviously able to handle on its own, so I was pretty sure things would continue to work. After terminating the Dllhost and making the change, I browsed to the same folder, deleted the thumbnails, and confirmed that there was no reduced functionality as far as I could tell. I then used Roxio to successfully burn a DVD with a number of AVI files. This case was closed.
My wife's system was now usable again, and though I wasn't able to close the Flash-related part of the case, at least I knew the cause and could keep an eye out for updates. More importantly, by solving the Dllhost part of the case, even if Flash went crazy again, her system would still be usable and she wouldn't be filing a critical support incident for it with me - thanks to Process Explorer and Process Monitor.
As an IT tech, I've encountered this behavior from windows quite frequently. Third party applications acting haywire. That was half-expected on 98, ME, even XP. But...
Considering how long Vista was in development...
Considering the cost of the OS...
Considering the intended audience that this flash-bang gee-whizz OS was created for...
Considering all that? I'm sorry Mark. Vista should be good enough to prevent this from occurring. Vista should be able to manage this so the end-user...a home customer...could manage any such problems without being married to one of the leading software debugging techs in the world.
Thank you for a very informative blog entry. I'm sure it'll be shared with anyone coming to be for recommendations on Windows Vista.
I have experienced similar slowness for the past year or more. However, my diagnostics skills do not match yours.
I finally gave up and got a Mac. After 18 years of Windows, I was ready for a change. I'm glad I did. It's been a long time since I was this happy with a computer. Sure I've had a few issues with my Mac, but far fewer in the months since I've bought it than I get in one week on Windows.
Sorry to sound like a Mac fanboy. I had no intention of leaving Windows...until it left me (with a crappy system).
> So if you had 2 processes spinning in a tight loop, and you wanted to do something else, that "something else" would get 33% of the CPU cycles. That seems much better than giving almost all CPU cycles to those 2 processes and almost none to Explorer, so that the start menu takes 30 seconds to respond to a click.
TimeSlice of Windows 9X was reasonable. Also Explorer.exe was faster - it didn't do all the useless stuff the newest versions do.
i used process explorer to track a client's dual core performance problem, too. it was taking 2 minutes to run an excel app i worte that runs in less than 40 seconds on any other pc they had. fired up process explorer, ran the code and what showed up thousands of times? google desktop search. uninstalled it and the code ran in about 30 seconds. thanks.
and as an aside, i hate flash, refuse to install it on my pc's, because it's the most obtrusive piece of software ever written. well, maybe it's not the software, but the web developers, that think everything has to flash, play a movie and blare sound as soon as a website is visted, when all i wanted to do was to read an article.
Thanks for the interesting blog entry that helps give some more ideas about how to use Process Explorer and Process Monitor for troubleshooting problems with software running on Windows.
As soon as I started reading the problem the first thing that came to my mind was that it was a problem caused by Adobe Flash player. I'm not surprise that you were not able to determine the root cause of that problem beyond tracing it to Adobe.
However, I am curious about why the Windows UI was unresponsive simply because of high CPU utilization. As some of the other readers have noted, the Windows scheduling algorithm should allow the user's actions to receive enough CPU time, to do something like open the Start menu, even if there is another task running that is using all available CPU time.
Usually when I see a system where Flash is using all the CPU time it will cause my browser (Firefox) to be unresponsive, but the Windows UI itself will continue to respond to user input. I would think that in Vista using Windows Desktop Manager there should be even less reason for a misbehaving application to make the Windows UI itself unresponsive. Also I've run various compute intensive programs that use 100% CPU time without causing the Windows UI to become unresponsive the way you described (30 seconds for the Start menu to open).
I don't know if you can comment on this at all or not, but I think it is a question that several people have posed and it would be interesting to know if you have any idea why the Windows UI was unresponsive in the problem you were troubleshooting.
Personally I disable video thumbnailers everywhere it is possible. They are commonly broken since they rely on being able to use third party code for things it wasn't actually designed for. And many DirectShow filters don't really support the full feature set of whatever they are supposed to handle to begin with turning thumbnailing into an automated crash generator. And apart from giving nice pictures for marketing, thumbnails are invariably totally useless. I mean really, what are the odds of a randomly picked image on a video file being anywhere near as informative as, a totally random example (not), the filename. Somewhere below 1% unless you have really bad naming conventions, I'd guess.
As a Firefox user, I have the AdBlock extension installed for the express purpose of being able to disable broken Flash on web pages. Mark is probably stuck using IE but maybe there is something similar available. Not much that can be done about the web pages that refuse to work without broken Flash apart from avoiding them, though.
Sorry if this ended up as a double post, I had a little technical problem.
"Considering all that? I'm sorry Mark. Vista should be good enough to prevent this from occurring."
Until you can answer that simple question, you are just a troll.
"As some of the other readers have noted, the Windows scheduling algorithm should allow the user's actions to receive enough CPU time"
How? By increasing the priority of explorer.exe? What if some plugin in explorer.exe runs away with the CPU? Instead of running at the same priority as other processes, you now have a high priority process eating CPU cycles...
Possible Solution: Always run Task Manager (because TM is usually stable) at a higher priority, thus allowing the user to quickly shut down normal priority processes. (this still does nothing for the average Joe)
Everyone blaming the scheduler: There's no way modern scheduler could have hardcoded process names. If you want to give a process a priority - well, that's what priority is for!
Explorer.exe can't run with elevated priority, because then OS would focus on running the shell more than on running your applications, which is what you buy PC for. If a process requests higher priority and then hogs CPU, it is application author's responsibility, and he is the one to blame - call Adobe and Roxio, instead of blaming MS.
On the other side, if the processes have same priorities, then a reasonable scheduler should give time to both of them even when other app goes berserk.
I have heard so many bad things about roxio's product, and this for me is just another nail in the coffin. I just don't trust them.
Mark, look at the filter drivers they install for your CD/DVD devices. I think they might contain one or two surprises as well... (that you may or may not notice as time goes by)
'A few weeks ago my wife complained that her Vista desktop was not responding to her typing or mouse clicks. Given the importance of the customer, I immediately sat down at the system to troubleshoot. ' Hilarious Mark just hilarious!
I had a question though. So since 'I had seen from the traces that the thumbnail cache generator had come across an AVI file that caused it to load the Sonic demultiplexer' is true does this mean that the component went into an infinite loop trying to create a thumbnail for an avi file?
Interesting. Thx for sharing the troubleshooting process, I learned a lot. But I wouldn't submit that for an "I'm a PC" video :-)
This blog is excellent to assess current real-world support: I doubt Microsoft Support would be able to troubleshoot that problem, and it would be impossible for a 'poor' OEM supporter, like Dell or HP, to troubleshoot it through popular email, chat, phone or VNC support; and if they were able to get machine on workbench, seems to me they would simply format HDD to factory settings and close ticket.
Also, are you wondering why some costumers have VIP support and other's don't ? Get the answer from THE Windows expert !
Thanks, Mark, very interesting...
Until you can answer that simple question, you are just a troll."
Build in to the OS that what a few 3th Party tools already are doing: Automatically and temporally lower the priority of any user process that is causing a High CPU load for more than x seconds, where the time the priority is lowered and the thresholds (% CPU load and duration) are both configurable,
and there also should be a way to exempt processes from this.
There are even tools that do this also for memory , where you can assign a maximum memory limit to each user (in a multi user environment)
It is quite common to use tools like this on multi user environments like Citrix XenApp servers to prevent the users perceived performance to become unacceptable when some user starts a high CPU or memory demanding application, you think flash is bad, you should see what some Business reporting software can do to a XenApp server with 30 - 50 users if you don’t use tools like this