The Case of the System Process CPU Spikes

re: The Case of the System Process CPU Spikes

ac — Mon, 07 Apr 2008 11:31:13 GMT

Love this kind of investigation, thanks Mark!

re: The Case of the System Process CPU Spikes

Arun Philip — Mon, 07 Apr 2008 12:31:58 GMT

> Because I want access to detailed information about system processes, as well as my own, I also specify the /e option on Vista, which causes Windows to present a UAC prompt on logon that allows me to grant Process Explorer administrative rights

Somewhat off-topic, but what's your opinion of using Vista's Task Scheduler to launch applications at startup/login with elevated privileges?

I use this technique to start Task Manager* elevated without facing a UAC prompt.

*: No offence, but I start Process Explorer for the heavy stuff and leave task manager running always.

re: The Case of the System Process CPU Spikes

Pieter — Mon, 07 Apr 2008 14:29:45 GMT

Nice and detailed story again! I was at your demo/presentation in Barcelona and suggest people to download and watch the webcast, really cool stuff ;)

re: The Case of the System Process CPU Spikes

David Eriksson — Mon, 07 Apr 2008 14:45:22 GMT

Imagine that you tried to get this fixed through the support form on the vendor's web site. That way we would have known if tech-savvy people who don't have a "hardware ecosystem team" to turn to also could manage to get something like this resolved in such a delightful way as you did.

Cheers,

David

re: The Case of the System Process CPU Spikes

W4nd3r3r — Mon, 07 Apr 2008 15:34:43 GMT

Mark,

Thanks for these posts on your blog. This is THE best info I have seen for troubleshooting. I love the toolset you created. They are the first tools I go for when there is a problem. They have solved countless problems for me.

Cheers!

re: The Case of the System Process CPU Spikes

AlexM — Mon, 07 Apr 2008 16:11:19 GMT

Mark,

The /mimimize switch for process explorer is actually /t, at least for current version :).

Great post, BTW!

re: The Case of the System Process CPU Spikes

Geoff Van Brunt — Mon, 07 Apr 2008 16:58:23 GMT

Wow, I read this and was supprised to find the exact problems I've been having on my Dell laptop. I had narrowed down the offending problem to the System "process" and was about to dig further. I guess now I'll just wait for the new driver and see if that fixes the problem.

Thanks for the article, not only did it help me fix my problem, it also was very informative in tracking down these kinds of problems.

re: The Case of the System Process CPU Spikes

Tim Bolton — Mon, 07 Apr 2008 17:02:43 GMT

Excellent Article!! Thank's Mark!

re: The Case of the System Process CPU Spikes

GS — Mon, 07 Apr 2008 17:28:02 GMT

Mark,

You said you always have Process Explorer running in background whenever you start Windows. Do you have any idea what kind of additional stress this puts on system? I want to do this as well but corncerned that this will eat away my CPU cycles which will eventually cause slow down of system.

re: The Case of the System Process CPU Spikes

Chris Knight — Mon, 07 Apr 2008 17:52:26 GMT

Thanks for the great article, especially the kernrate and xperf tools.

I quite like ThreadMaster as a tool/workaround for flattening out CPU spikes, but it's not a match for quality debugging.

Now I've just got to work out how I'm going to hook up kernrate/xperf via Powershell to my MRTG graphs...

re: The Case of the System Process CPU Spikes

Nader — Mon, 07 Apr 2008 18:20:48 GMT

>...The System process is a special type of process on Vista called a “protected process”...

Mark, restrictions of running inside protected processes requires that these processes be specially signed.

Ooops...What about malware running as a protected process!!

But does the process need to be signed, or is it the binary image on disk that needs to be signed?

/Nader

the restrictions of running inside a protected process, the operating system requires that these processes be specially signed

re: The Case of the System Process CPU Spikes

Adam — Mon, 07 Apr 2008 18:58:12 GMT

Thanks! Now I know what's causing my problems and was able to repro with the same results as you. Feel like sharing that driver? :)

Dell's support site

Ryan — Mon, 07 Apr 2008 19:00:00 GMT

Thanks, Mark--great article!! Your passion to troubleshoot these issues--and then share in clear, concise detail how you did it--is a fantastic benefit to the community.

I have to echo an earlier comment about your support "backdoor," though--most of us are used to stonewalling and levels of isolation when trying to resolve these types of issues with a vendor (sometimes it feels like communicating with Prince--you have to pass notes to the official correspondent, hope it passes their own filter, pray it's communicated correctly, then perhaps get a response). Then, it seems like it takes a miracle to make it onto the Dell support site--I'm personally waiting on a driver update for an easily reproducible issue with Intel's integrated graphics driver (OptiPlex 745) for Vista. I've reported the issue and followed up several times, but still there's no update. The updated driver has been available from Intel since April 2007, and is even on Windows Update (but not in a way that can be mass-deployed as far as I've tried).

In other words, these articles are brilliant... but unfortunately, even with such skills, it's often exceedingly hard to get a vendor response.

re: The Case of the System Process CPU Spikes

RaFi — Mon, 07 Apr 2008 19:48:50 GMT

What about KernRate for Windows Server 2003 x64 ?

Perhaps you can find it on Microsoft.com

I found http://www.nynaeve.net/?p=132

re: The Case of the System Process CPU Spikes

Andrew Sherman — Mon, 07 Apr 2008 20:11:05 GMT

Great article, but begs the question of how to get these sort of problems resolved if you don't work for Microsoft.

re: The Case of the System Process CPU Spikes

Neil Prestemon — Mon, 07 Apr 2008 20:13:58 GMT

Heh; I'm having a weird issue with my Dell laptop too (like Geoff), narrowed it down to System Process - and my suspicion has been that it's the dang broadcom driver, but I've been unable to dig in any deeper.

(seems to be worst when I'm connected via wireless, and when I'm transferring large files, and especially, when I'm in my living room, where my wireless signal isn't the best; the CPU spikes, then the connection drops - when the system gets into this "state", I can't even shut down gracefully. Won't even ctrl-alt-del, just hangs).

I'm very much looking forward to the new driver, hopefully that fixes it.

I'd like to learn how to set up a Symbol server and load symbols, so I don't run into the troubleshooting "brick wall" for these kernel-mode driver cases.

re: The Case of the System Process CPU Spikes

Out here in the real world — Mon, 07 Apr 2008 23:19:07 GMT

Boy are you living in a special world Mark! Those of us out here in the sunlight would still be trying to get past the Vendors first level "tech" in India.

re: The Case of the System Process CPU Spikes

robert — Tue, 08 Apr 2008 01:57:38 GMT

nice post, good to see a lot of explanation and background info for the concepts you talk about - that helps a lot when trying to understand it all.

re: The Case of the System Process CPU Spikes

Friday — Tue, 08 Apr 2008 11:01:38 GMT

> The main way I keep tabs on things is to configure Process Explorer to run automatically when I log in. Whenever I configure a new computer, I add a shortcut to Process Explorer to my profile’s Start directory...

This is programmer's nirvana. To use daily your own tool...

Keep up the good work!

re: The Case of the System Process CPU Spikes

Peter Ritchie — Tue, 08 Apr 2008 20:31:14 GMT

Brilliant! This problem has been plaguing me for a couple of years. I was hoping you'd blog about it.

re: The Case of the System Process CPU Spikes

aaron — Tue, 08 Apr 2008 22:13:38 GMT

Your articles are always interesting to me. I'm a linux user that supports a lot of corporate windows networks.

Of course the part of the article that annoys me is that DRM is limiting you from easily troubleshooting a problem. For a non-uber-windows-developer like me, at the point it said "Unable to access thread" I would have given up. And I would imagine more than a few Microsoft customers would feel the same way.

re: The Case of the System Process CPU Spikes

Intuit — Wed, 09 Apr 2008 05:37:54 GMT

Unfortunately in the case of a Compaq Presario X1000, X1010 laptop, the intermittent spikes weren't so easily diagnosed. The problematic driver simply never rose to the top of the list. Ended up chasing all kinds of Red Herrings. Regardless, the spikes were severly disruptive and halted mouse, audio, video and drive activity for about a 1/4 sec on each occurance; spikes occuring about 2-3 seconds apart for up to 30 seconds during each "fit". The fits were anywhere from two up to five minutes apart. Forget watching videos. Tried stripping down the machine's drivers and services, disabling devices, new drivers, OEM drivers, old drivers, chipset, audio, video, NIC, WLAN, BlueTooth, CardReader, USB Hubs, new Video ROM, new BIOS, etc, etc... lots of time.

To keep a longer story short, I eventually noticed that the spikes did NOT occur in Safe Mode, but DID occur in Safe Mode WITH Networking. In Safe Mode with Networking, the problematic driver like oil in water indeed rose to the top-three on the list; versus being near the bottom. HAL, W70N51, NTOSKRNL

What was it ?

c:\windows\system32\drivers\w70n51.sys:

Verified: Signed

Signing date: 4:00 PM 7/21/2006

Publisher: Intel« Corporation

Description: Intel« PRO/Wireless LAN Driver

Product: Intel« PRO/Wireless LAN Adapter

Version: 1.0.0.0

File version: 1.2.4.41

Original Name: w70n51.sys

Internal Name: w70n51.sys

Comments: NDIS 5.1 Miniport Driver

File Index: 0x00150000

Links : 1381240

Same problem with the newest drivers from Compaq/HP and newest available from the Intel Website. I'll try the oldest drivers I can find but I've spent enough time on this already.

For others, Start Menu --> RUN --> type "CMD.Exe" --> OK button.

At the command console prompt type:

sc config w70n51 start= disabled

Reboot for the driver to be disabled. (it can't be stopped) USB NICs are available for as little as $20 on eBay.

Can't express enough appreciation for your help Russonivich. Thanks. :-)

-Marc

re: The Case of the System Process CPU Spikes

Brandon Clinger — Wed, 09 Apr 2008 07:16:53 GMT

I wish I didn't have to deal with these problems...I blame drivers for every problem I have, someone should really make WHQL hard to get, so things like this do not happen. Drivers need to be stricter.

re: The Case of the System Process CPU Spikes

Ian — Wed, 09 Apr 2008 14:41:28 GMT

I too have had issues with Broadcom Gb ethernet drivers. We built a Dell Dual Xeon server system purely to host a SQL 2005 server (for use with MS Business Contact Manager) on Win2003. It worked fine during testing, but when deployed suffered from excessive sessions of 100% CPU usage (often for hours), during which time users suffered slow and stuttery Outlook 2007.

Spent ages fiddling with SQL, no effect then one day figured out via the SQL Dashboard, that SQL was spending excessing time on network I/O (about 80% time).

Updating the Broadcom network driver, as you did, and bingo CPU usage is hardly moves, except at 8.30am - 9am as users connect.

re: The Case of the System Process CPU Spikes

Koby Kahane — Wed, 09 Apr 2008 20:07:18 GMT

The inability to extract stack traces from threads in the System process is a significant limitation of Windows Vista. Once again users and developers pay the price for a DRM "feature" without any piracy actually being prevented.

How is a driver developer supposed to troubleshoot issues in Vista drivers without convenient access to System process stack traces? The system could be frozen entirely through the use of kd or windbg for kernel debugging and stack traces could be promptly extracted, but that's hardly as snappy as using Process Explorer.

The fact you've had to use a kernel profiler, which only worked by accident (no official build for Vista is available, as you say) for this task is a sad testimonial for how missing the previously available functionality is.

This, along with Visual Studio's RPC Debugging feature no longer automatically attaching the JIT debugger to a COM/RPC server process upon method Step-In under Windows Vista (x86/x64) and XP x64 together form a pattern of significant regressions in developer support features in recent Windows versions.

re: The Case of the System Process CPU Spikes

Hagen Patzke — Fri, 11 Apr 2008 00:04:57 GMT

Very nice to see a short, real-world example using Kernrate - since using it to check where time was spent in a UMDF vs. WDM driver, I had almost fogotten about it.

Yes, also my Vista laptop is sometimes sluggish (just micro-freezed a second ago again!), and thanks to this post I will use Process Explorer and Kernrate to check out why.

And as I'm already writing: Thank you (both) for the excellent, "slim" and extremely useful Windows system tools you give us admins and developers (for free!) for all these years.

Back in 1997, when working in admin + 2nd level support, tools like filemon and regmon were often essential for finding the root cause of a problem. They were certainly much quicker than the available alternative - debugging and/or trying to profile a foreign application.

One thing has not changed in all these years: the "Sysinternals" tools - today e.g. Process Explorer and Process Monitor - are extremely useful. These days I use them for driver and application maintenance: e.g. with WinObj I could finally *verify* that our "mutants" were active and properly named.

This blog - and the webcast you link to above - make it much easier to grasp their potential by giving real-world examples.

To sum up: Thanks! (A lot!)

(...and sorry for the long post)

re: The Case of the System Process CPU Spikes

Ogawamonr — Sun, 13 Apr 2008 21:11:48 GMT

>> I add a shortcut to Process Explorer to my profile’s Start directory that includes the /t (minimize) switch.

Hmmm... While I too habitually add ProcExp to my profiles' Start directory, I tend to ALSO launch it earlier @ HKLM\~\Winlogon\userinit = ""C:\Program Files\MS+Sysinternals\Process Explorer\procexp.exe" /p:r ,C:\WINDOWS\system32\userinit.exe,"

That reminds me of a small glitch in procexp...

After all of everything has loaded (SysTray, etc...), one of the "CPU Hist" Systray icons is gone (i.e., "Gone!" --not hidden or inactive) So I've got 2 IO Hist and only 1 CPU Hist (belonging to the Start folders' process)

------

BTW, I wish for additions to /e /s /t (minimize) and /p (Priority) CmdLine Opts such as:

[ /z -- Pauze (i.e. space bar)]

[ /x:[kqmpx] --Exit Method

...k = single ESC key... q = with Prompt;

...m = Mouse/Menu only... p = with Prompt;

...x = Exit Disabled (use process kill) ]

re: The Case of the System Process CPU Spikes

Funbit — Tue, 15 Apr 2008 05:55:54 GMT

Nice investigation! As always :)

re: The Case of the System Process CPU Spikes

John Currie — Tue, 15 Apr 2008 11:05:03 GMT

I have the same issue, hopefully the drivers will be posted soon.

re: The Case of the System Process CPU Spikes

anonymous — Wed, 16 Apr 2008 02:10:50 GMT

RaFi, Kernrate is part of the Windows Server 2003 Resource Kit Tools and works very well on x64 platforms.

re: The Case of the System Process CPU Spikes

Peter A — Thu, 17 Apr 2008 08:43:48 GMT

Helpful article - thanks!

I wonder if the tools can help in the same way to show the reason for the mysterious 90-100% CPU Windows Explorer hits that come and go for long periods in Vista. Can these be due to 3rd party code in worker threads too?

re: The Case of the System Process CPU Spikes

Stephen — Thu, 17 Apr 2008 17:37:40 GMT

Mark, thanks for this, just the advice on getting Process Explorer running on boot was invaluable; a 450mHz Pentium III is at full stretch with XP.

For the person who asked earlier about the processor usage of ProEx, it is 1%-2% on my rather elderly machine.

re: The Case of the System Process CPU Spikes

Joe — Thu, 17 Apr 2008 18:46:35 GMT

Great article! But...who is running Vista?

re: The Case of the System Process CPU Spikes

Jose Medeiros — Thu, 17 Apr 2008 19:37:10 GMT

Awesome article! Unfortunately, we are all stretched so thin in IT that we rarely have the time to become such experts in this area. Mark, makes it sound so easy ( That is why he has a Ph.d ), however any Systems administrator knows how difficult such a problem is to isolate, and even more so, it it is intermittent.

Jose Medeiros

San Jose, CA

re: The Case of the System Process CPU Spikes

Danny Olson — Fri, 18 Apr 2008 02:43:26 GMT

I really liked this article but whats killing me is commit history commit section where I cant tell what is taking up all my space. Is there someway to see the memory hogs and kill them like you can the cpu hogs?

re: The Case of the System Process CPU Spikes

Vince Lumbers — Tue, 22 Apr 2008 04:58:52 GMT

I run Process Explorer as a service when the machine boots by using Microsoft's SrvAny.

It not only ensures that Process Explorer is running every time the machine boots it also allows me to logon as a normal User and have Process Explorer running as NT AUTHORITY\SYSTEM.

I also have an AT command set to run it at 7am everyday so if it has exited the previous day its there running again each morning again.

re: The Case of the System Process CPU Spikes

Vern Anderson — Thu, 24 Apr 2008 02:22:17 GMT

Mark, can you ask Broadcom if the changes to the driver had anything to do with TCP/Offload or TOE?

I work for a large hosting company and we have had issues with Broadcom drivers on both DELL and HP servers.

specifically:

[HKLM]\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

EnableTCPChimney

EnableRSS

EnableTCPA

handle.ex v3.31 /accepteula switch glitch to fix

Don Draper (ddraper1@gmail.com) — Tue, 29 Apr 2008 23:14:42 GMT

If \HKEY_USERS\...\Sysinternals\Handle\EulaAccepted = 0 in the Registry, then handle.exe /accepteula does not work correctly.

re: The Case of the System Process CPU Spikes

Peter Ritchie — Wed, 30 Apr 2008 19:34:37 GMT

@Jose: And how hard it is to get support from the NIC vendor (since support is usually through a computer manufacturer whose support, honestly, isn't geared toward troubleshooting NIC driver problems.

it would be nice if we could all look up a dev in a hardware manufacturer database and send them and email...

re: The Case of the System Process CPU Spikes

Arik Yavilevich — Fri, 02 May 2008 16:59:24 GMT

Hey Mark,

Thanks for the interesting post.

I had the same problem and following your article I was able to detect that my b57nd60a.sys driver is causing the same problem. I have a HP laptop, but apparently it has the same network card by Broadcom. I have version 10.10 of the driver and I am now waiting for HP to release a new driver.

Since I have a 64bit Vista, I used xperf. This is what I have done:

1) Installed Windows Performance Tools Kit from the link you provided

2) ran: C:\Program Files\Microsoft Windows Performance Toolkit>xperf -start -on diag

3) ran: C:\Program Files\Microsoft Windows Performance Toolkit>xperf -stop -d c:\temp\result.etl

4) Opened the ETL file in the viewer.

5) Clicked "Summary Table" on a CPU graph.

6) Expanded until I found the problematic item.

Very interesting indeed.

Regards,

Arik.

re: The Case of the System Process CPU Spikes

Neb — Wed, 07 May 2008 16:53:24 GMT

It was a great read! I can just say that maybe, to avoid this kind of investigations, one might just use Linux...

re: The Case of the System Process CPU Spikes

Garlog — Sun, 11 May 2008 03:27:34 GMT

Mark,

I am sending out a prayer.

All I ask is that Windows 7's top priority is speed. It should be faster than XP.

Vista was never even an option for me after I discovered lower benchmarks across the board.

Within my circle of friends, not a single one even considers Vista an option due to its performance problems.

Don't release Windows7 until you've got a codebase that is faster than XP, period.

re: The Case of the System Process CPU Spikes

warl0ck — Mon, 12 May 2008 09:16:33 GMT

Hi Mark!

Great post as usual.. Im a frequent user of the tools that you have written. Process Explorer and Process Monitor have helped me troubleshoot quite some problems.. I have started documenting some of my ordeals too. You have asked us to share our success stories with you, but you haven't mentioned how? Psst.. whats your email address?

re: The Case of the System Process CPU Spikes

Westy — Mon, 19 May 2008 10:02:18 GMT

I found the article very helpful, so thanks Mark. Looking beyond desktop machines, can this be implmeneted to troubleshoot servers as well? Working in an organisation that serves internet web applications, is there any way to run these tools remotely rather than from the environment itself?

Look forward to you next article.

re: Resource Spikes with Layered Security

Alan McRae — Mon, 19 May 2008 21:22:16 GMT

Great article! We happen to be analyzing resource spikes (cpu,ram,hd)in client PC's with layered security built from several best-of-breed security products, so this is very helpful to us. Hopefully we'll be able to tweak layered security performance to eliminate unnecessary resource hogs but still maintain comprehensive security protection.

re: The Case of the System Process CPU Spikes

Josh Miller — Thu, 22 May 2008 21:48:33 GMT

Thank you,

I couldn't figure out what was causeing my father's computer to run so slowly. When I used process explorer it showed that the Interupts were running at >75%. When I looked in detail using kernrate I saw that hal was using almost twice as much time as the kernal itself. When I looked at hal in the detailed view from kernrate it was mostly harddrive related commands.

Further investgation showed that somehow the primary IDE controller was in PIO instead of DMA. Once I uninstalled the device and bounced the box everything worked great. The harddrive was running in DMA5 once more, and idle system processes were back below 4%. I would never have figured out this error without sysinternals and watching your presentation of how to use the tools.

re: The Case of the System Process CPU Spikes

ALexander — Fri, 23 May 2008 13:11:43 GMT

hello. great post! in fact it describes same exact trouble that I have. could you please send me that renewed driver that you got from dell. because they still don't have it on website.

saint_ff@Mail.ru

re: The Case of the System Process CPU Spikes

Mark Holloway — Sun, 25 May 2008 23:28:20 GMT

Mark

I remember seeing a video demonstration on the Microsoft site for using Procexp. Do you happen to kown the URL or how I can access the demo again.

Great tool, I am trying to sort out an issue with Windows media player (The player is halted by high CPU usage from some where for about 0.5 - 1 sec always around 3/4 of the way through the track?)

I have used Procexp to indentify the thread, but cannot get much further.

Any help welcomed

Regards

Mark

re: The Case of the System Process CPU Spikes

Dilan — Tue, 27 May 2008 23:51:14 GMT

I have had the same issue. So far I have been hesitant to install the driver update from Broadcom. But after calling Dell and being told they will not support Vista (as my D820 came with XP orignally) I decided to bite it installed the updated driver from the Broadcom site. The spikes have dissapeared.

http://www.broadcom.com/support/ethernet_nic/netxtreme_desktop.php

The issue is that everything is enabled

Greg — Wed, 04 Jun 2008 17:44:25 GMT

This happens because the hardware vendor will configure every possible driver for the entire machine to be loaded and enabled even for devices you do not use.

As a plea for help, can we get a Windows version with a user friendly way to disable drivers and services for parts that are not needed?

Some examples:

Zero configuration wireless networking service

Windows image acqusition (WIA) service

SSDP discovery service

FAX service

etc.

Our standard desktop is severely slowed down by the extra RAM usage used by the unneeded processes and drivers.

Lastly, can some of the always loaded processes (e.g., print spooler) be put into a hibernate mode where they use minimal memory until a print request comes in from the user? Spooler is using 9mb of RAM on a machine booted 1 hour ago where there has been no print jobs started.

broadcom drivers

me — Thu, 05 Jun 2008 04:06:42 GMT

Yep, I had issues with Brodcomm and the Chimeny stack as well. Once I turned off the chimeny stack all was well, but didn't get a new driver to fix it however since then MS has posted a hotfix to turn the tcp chimeny stack off by default instead of on by default which is much better. So if you were able to get a debug version of the driver and Kernrate to get the functions that a protected process used, then couldn't the same method be used to be able to snoop and get past the DRM and get teh keys anyway? DRM doesn't work, there is always away around it so just get rid of it.

Where can I get working Broadcom drivers?

Daniel S — Mon, 09 Jun 2008 05:34:14 GMT

I have the same exact issue described in this article with my Inspiron 1525. Occasionally the Broadcom driver takes up 50% of both cores of my Core2Duo 2.0Ghz and will not fix itself until the system is restarted. Sending/recieving in Microsoft Outlook 2007 reproduces this error every time for me.

Is there a way I can fix this? I'm at my wits end. Mark makes reference to a 'fixed driver' he recieved but I cannot find it anywhere.

MUCH THANKS FOR ANY HELP!

Where to get Broadcom drivers

Arion Lofton — Fri, 13 Jun 2008 13:09:38 GMT

Updated drivers are available on Broadcom site at http://www.broadcom.com/support/ethernet_nic/netxtreme_desktop.php

Hope this helps :-)

Got new driver: Problem solved!!!

Martin Icking — Thu, 19 Jun 2008 14:24:00 GMT

I installed the newest Vista driver (10.82.0.0a) on my Dell Precison M70 from Broadcom's site menitoned above. Since then all peaks on the system process disappeared. Thanks!!!

re: The Case of the System Process CPU Spikes

Chris K — Mon, 23 Jun 2008 23:55:45 GMT

Just wanted to chime in and offer my thanks as well. After seeing a constantly high CPU after I installed a system monitor, I recalled this blog and took a closer look. Sure enough it was the same driver issue. I downloaded the tools and went through the steps, finding the same issue. I'll be sure to keep these on hand whenever I find something similar. Thanks again!

re: The Case of the System Process CPU Spikes

Norman Rasmussen — Fri, 11 Jul 2008 10:47:46 GMT

I've noticed that the system process has been using 17% of my cpu for some time now. I finally got around to kernrating it, and lo-and-behold it was the same Broadcom driver. Installing the new version has made the system process nice and quite again. Thanks Mark!

P.S. I couldn't get xperf to admit that the Broadcom driver was at fault, perhaps I didn't specify the correct logging option.

re: The Case of the System Process CPU Spikes

Matthias — Thu, 17 Jul 2008 11:24:35 GMT

Had the same problem you described (100% CPU spikes) with my Vista installation on HP Compaq nx8220. Tracked it down as directed and replaced the driver with the one from the Broadcom-Site.

http://www.broadcom.com/support/ethernet_nic/driver-sla.php?driver=570x-Vista32

Everything's back to normal now - thank you so much.

re: The Case of the System Process CPU Spikes

Tom Ballard — Wed, 30 Jul 2008 14:57:55 GMT

Wow, amazing how quickly problems get solved when people share each other's secrets a la open source. Microsoft acts internally more like open source, and expects the public not to. Or perhaps everybody gets PDBs just for the asking.

re: The Case of the System Process CPU Spikes

Ravi — Sat, 02 Aug 2008 20:50:37 GMT

This is a very good presentation : Well Explained, i wanted to know if we would be able to find out drivers / Process under the System Process, On our terminal Servers we see System Process having an Handle on NTUSER.dat file under the profile directory.

How to detect which is the culprit

re: The Case of the System Process CPU Spikes

Akaz Farhsa — Fri, 26 Sep 2008 16:43:41 GMT

Mark,

How about adding something like a "Monitor drivers" feature to Process explorer where it uses the same sequence of debugging steps to maintain a list of non-microsoft device drivers that frequently use a lot of cpu? I know some would always be expected to show up but perhaps these can be marked as known and then unknown ones highlighted? This idea needs refining but might be worth adding,

Cheers,

Akaz

re: The Case of the System Process CPU Spikes

mvadu — Sun, 02 Nov 2008 09:59:52 GMT

Hi Mark,

I have Process Explorer instead of Taskmanager.

One problem I used to face is ProcExp will open with standard user privilage so incase I need to see inside system processes it will fail. Today I noticed that I can add a /e switch in registry itself where ProcExp places a shortcut as debugger. So each time I press Ctrl+Shift+Esc to open ProcExp I get UAC prompt and I can open it in Admin mode. Good learning, thanks to Autoruns which gave me the path in registry.

re: The Case of the System Process CPU Spikes

MrO — Wed, 19 Nov 2008 17:01:34 GMT

Thanks for troubleshooting and in the replies the link to the Broadcom vanilla driver.

From Wireshark, I noticed this seemed to be happening when Outlook 2007 was trying to talk RPC/DCE to the Exchange server. There was a repetitive 4 packet transaction, not sure what either Outlook or the server trying to do or why this affected the BroadCom card.

re: The Case of the System Process CPU Spikes

KT — Sat, 22 Nov 2008 01:36:57 GMT

FYI, the latest version of KernRate that runs on Vista and Windows 2008 is available in Windows 2008 DDK (it's in \tools\other). There're both 32-bit and 64-bit versions. The DDK itself is a free download from Microsoft.

re: The Case of the System Process CPU Spikes

Chris — Wed, 03 Dec 2008 04:20:59 GMT

Marc,

THANKS!

Intuit,

THANKS!

Between the two of you I didn't even need to track the bastard intel 2100 crappy driver down. I first attempted the broadcom 57x driver to no avail. Saw Intuits post and tried that. Bingo!

re: The Case of the System Process CPU Spikes

Chuck Quigley — Tue, 17 Feb 2009 19:21:50 GMT

Thanks to this article I was able to trace the constant CPU usage of about 40% on my system to ACPI.SYS. It appears to be looping after coming out of sleep mode and the only way to stop it is through a Restart. Unfortunately I am stuck now. Vista says the driver is up to date and it won't allow it to be removed or disabled. I've seen a few other posts elsewhere with the same problem but none with a fix.

re: The Case of the System Process CPU Spikes

dhruva — Sun, 08 Mar 2009 14:50:35 GMT

Nice article. Having used Kernrate, a mention of krview would have been nice. I have seen the reports generated by krview is very informative and a nice document to send it across for further analysis. I wonder why Microsoft is not coming up with a x64 version kernrate. Though this is OT, I am using this thread to bring to your attention and hope you can use your contacts inside to do something about this. I have not tried xperf yet as I use XP...

re: The Case of the System Process CPU Spikes

Tue, 24 Mar 2009 14:01:33 GMT

I've found a problematic thread using process explorer, but can't suspend it...get the same error message-"unable to access thread" I tried using kernrate but after I press control-c the window just disappears. Is this because it only shows the results by printing? Or is it because I'm using Vista-64 and need to try xperf? Also, isn't there anyway to disable the security protecting of the file I want to access? Via Dos prompt or what? Any help you guys can give would be helpful, but anyhow, kudos on an excellent post!

re: The Case of the System Process CPU Spikes

Arun — Fri, 12 Jun 2009 16:49:55 GMT

Great article. I had CPU spiking up on the system process and went through your diagnostic process. It turned out to be the same Broadcom driver that was causing the issue. I downloaded the latest driver and the issue is resolved now.

Many Thanks.

re: The Case of the System Process CPU Spikes

Sukon — Sun, 09 Aug 2009 16:10:08 GMT

Again a great article.

I would really like to watch your webcasts but unfortunately the bandwidth in Thailand is too poor. Is there any way to download the video files?

Thanks

Sukon