The Case of the Frozen Clock Gadget

re: The Case of the Frozen Clock Gadget

Darrin Stevens — Tue, 16 Oct 2007 00:48:42 GMT

Was the function checking a version just a victim of the memory leak then?

And when looking at stack traces with lines like:

sidebar!Package::EnsurePackageMinVersion

Is the bit after the ! and before the :: a c++ object? Are there any good resources to read to help understand stack traces?

Great post!!!

re: The Case of the Frozen Clock Gadget

markrussinovich — Tue, 16 Oct 2007 01:11:41 GMT

Yes, the package-version functionality was a victim of the leak.

The way to read the symbol is:

Thanks for the feedback.

re: The Case of the Frozen Clock Gadget

denisbider — Tue, 16 Oct 2007 06:39:08 GMT

In my experience - and probably not just my own - the quality of Windows Sidebar is far, _far_ below par for such a prominent feature in such a major release of the world's most popular operating system.

Like I wrote in March:

"I don't know who signed off on a Sidebar of this quality going into the Vista release, but whoever it was, it must have been the kind of B category employee that Bill Gates supposedly said Microsoft ought not to employ.

"The whole Sidebar thing looks like it's been polished by Microsoft, but developed by students. It's just not good enough. Not by a wide margin."

http://denisbider.blogspot.com/2007/03/doghouse-windows-sidebar.html

re: The Case of the Frozen Clock Gadget

Martin Plante — Tue, 16 Oct 2007 06:42:16 GMT

Great in-depth explanation, as usual.

Could the repeated use of window.SetTimeout within the same function the source of the problem? Would it be easy to convert the code to use window.setInterval instead, and compare? I could try it myself if I knew which gadget it is! q;-)

re: The Case of the Frozen Clock Gadget

Simon — Tue, 16 Oct 2007 08:37:59 GMT

I expected GetFileVersionInfoExW to return an error instead of using a null-pointer. Why not file a bug for that?

re: The Case of the Frozen Clock Gadget

bcthanks — Tue, 16 Oct 2007 09:13:44 GMT

A big thanks for taking the time to debug and file a bug report. Otherwise I'm sure all the blame is falling on the gadget's author, and worse, soon there will be a lot of folklore about "don't call RemoveObjects because there's a leak in sidebar.exe ..."

Two observations:

1. somebody isn't checking for NULL after calling malloc(). sidebar.exe should never crash.

2. Microsoft is isolating their gadgets from everyone elses by running two sidebar.exe? Are their gadets more trustworthy than others? Why not isolate every gadget on its own?

re: The Case of the Frozen Clock Gadget

Mathieu CHATEAU — Tue, 16 Oct 2007 09:19:55 GMT

Your post rocks, as always !

re: The Case of the Frozen Clock Gadget

Mathieu CHATEAU — Tue, 16 Oct 2007 09:34:05 GMT

Do you think that reporting the bug to Microsoft through the Error reporting service would have lead to the same ? Since this error is due to something local (a gadget) that expose a memory leak in a windows API.

re: The Case of the Frozen Clock Gadget

Tommy Svensson — Tue, 16 Oct 2007 09:41:43 GMT

I love these thrilling investigations of yours Mark! Very educative and entertaining!

Thanks a lot!

/Tommy

re: The Case of the Frozen Clock Gadget

Fleet Command — Tue, 16 Oct 2007 10:33:04 GMT

May I ask what did you write when you filed the issue in the Windows Bug database? I'm just curious on how would you file a bug like this.

re: The Case of the Frozen Clock Gadget

Lasse Jari Hansen — Tue, 16 Oct 2007 11:35:14 GMT

I wish someone would notice the celsius bug in built-in Weather gadget.

* If you have it set for showing degrees in celsius, it will only check the weather rougly once a day. You will notice this as your work day progresses and the weather changes outside, but not in the gadget.

* Change back to fahrenheit anytime, and the weather changes to the correct status.

* Set it to degrees again and the wrong weather status (from morning time) returns.

I have tried to locate the source of the bug but seems to come from the dll "wlsrvc.dll" in "c:\Program Files\Windows Sidebar", which the gadget uses for weather services.

Maybe this could be a future case story?

re: The Case of the Frozen Clock Gadget

Jack H — Tue, 16 Oct 2007 13:51:56 GMT

As always great post.

re: The Case of the Frozen Clock Gadget

jody — Tue, 16 Oct 2007 14:09:07 GMT

Man I love this stuff!

Thanks, Mark.

re: The Case of the Frozen Clock Gadget

Santhosh — Tue, 16 Oct 2007 15:00:01 GMT

cannot understand one thing, how come such critical APIs passthrough testing with this kind of Bug !!.

re: The Case of the Frozen Clock Gadget

J.R. Raith — Tue, 16 Oct 2007 18:24:50 GMT

I know -exactly- which gadget ran into this memory leak, heh. Or, at least, I know of -a- gadget that, if left alone for a few days (say, over the weekend) causes sidebar.exe to start nomming up RAM. And then, when you close it, sidebar ramps up.

What I find most amusing, though, is that the ending of sidebar.exe doesn't identify the correct gadget that may have caused the problem...

re: The Case of the Frozen Clock Gadget

dumian — Tue, 16 Oct 2007 18:35:32 GMT

Hi Mark,

I have seen it on my machine also, and the symbols drive me to same leaking scenario. quite interesting stuff.

btw is any public release of the 6.8.0.0.1 windbg ?

Thanks,

cosmin

re: The Case of the Frozen Clock Gadget

Adi R — Tue, 16 Oct 2007 21:55:57 GMT

Mark,

Awesome investigation, as usual.

However, I think you are doing a major dis-service to us IT community by not revealing the name of the gadget. A Rogue gadget like this can kill entire Vista, on PCs with 1GB of RAM and relatively low Virtual Memory space, people will keep running into slowness of their system and not knowing why.

Was the author of the gadget notified at least? To give him/her a chance to maybe come up with a work around for the problem, for time being?

re: The Case of the Frozen Clock Gadget

Neil Prestemon — Tue, 16 Oct 2007 22:36:35 GMT

I wish that the debug symbols were more accessible to non-developers. It seems like you have to have an expensive MSDN membership just to have access to these tools; yet a lot of employers won't justify that for a person who is in a test or systems engineering role, who could find such tools useful for troubleshooting, even though they don't necessarily sling-code as their primary job.

re: The Case of the Frozen Clock Gadget

markrussinovich — Tue, 16 Oct 2007 23:10:59 GMT

The Windbg debugger and debug symbols are available to anybody. See the link to the symbol server in the blog post.

re: The Case of the Frozen Clock Gadget

Jean Gauthier — Wed, 17 Oct 2007 19:55:16 GMT

"Two observations:

1. somebody isn't checking for NULL after calling malloc(). sidebar.exe should never crash.

...

This is false. If after a malloc, you get a NULL pointer, what can you do?

Trying to prevent that makes only the program crash elsewhere and its really tough track back the error in that case, because the stack trace is irrelevent. I suggest only adding debug asserts.

May I had that powerful memory leak revealer tools exist for both Native and Managed code. A trivial leak like this might have been easily found.

re: The Case of the Frozen Clock Gadget

edgar — Thu, 18 Oct 2007 01:12:00 GMT

Strange.

It's not only a bug.

Looks also like a bad design.

This canvas API's should not make any File-Version-Check ?

Or do I miss something ?

re: The Case of the Frozen Clock Gadget

manaya — Thu, 18 Oct 2007 10:51:50 GMT

Amazing post! really gets in depth to the "digging to the bug" process in a very educative and entertaining way.

M.-

re: The Case of the Frozen Clock Gadget

Nader — Thu, 18 Oct 2007 19:46:56 GMT

Hi Mike

Great investigation,

Quick Question please, I'm trying to create a folder named "con" on Windows Vista Desktop, and I'm receiving an error message "The specified device name is invalid"...

Any Explanation!! :(

re: The Case of the Frozen Clock Gadget

Nader — Fri, 19 Oct 2007 12:24:01 GMT

Great,

Thanks Terry.

re: The Case of the Frozen Clock Gadget

markrussinovich — Fri, 19 Oct 2007 16:36:54 GMT

Con is a reserved name in Win32 that represents serial ports.

Thanks for the feedback.

re: The Case of the Frozen Clock Gadget

Nektar — Sat, 20 Oct 2007 02:23:57 GMT

1. If you get NULL after allocating memory then you should inform the user not crash!

2. If "con" is a reserved device name then say so and not give an uninformative message "the device name is invalid"!

The second issue has existed for year in Windows and still not fixed. Also,

3. When you overwrite or delete a file in Windows Explorer which is in use, Windows does not still inform you by which application the file is used so that you may close that application before deleting/overwriting the file.

This issue has also existed for years.

re: The Case of the Frozen Clock Gadget

Nektar — Sat, 20 Oct 2007 02:24:53 GMT

By the way, "con" represents the console (keyboard) I think.

re: The Case of the Frozen Clock Gadget

Frymaster — Sun, 21 Oct 2007 01:58:38 GMT

inform the user how? bearing in mind anything you do can't involve allocating memory...

and if you _do_ inform the user, what then? it's not like you can do anything about it, so you'll have to close anyway.

closing your application and informing the user is, not coincidentally, what happens when an application crashes, with the added advantage that a bug report is generated that might lead to the issue being fixed.

re: The Case of the Frozen Clock Gadget

Terry — Sun, 21 Oct 2007 03:25:21 GMT

I've submitted those automatic crash reports for years, without seeing any change after many updates. I've concluded that if I'm very lucky and the crash is resolved, it will be in the next version of Windows, not this one.

re: The Case of the Frozen Clock Gadget

Erwin Ried — Sun, 21 Oct 2007 07:46:09 GMT

Mark, are you seen the bug in the Calendar Gadget?

http://www.fayerwayer.com/up/2007/10/vista-calendario.png (13th August is twice)

I tried to fix it, but is very tricky because I discovered Javascript was the origin of the error, especially with Date.getDate() and Date.getDay() functions.

re: The Case of the Frozen Clock Gadget

Norman Diamond — Mon, 22 Oct 2007 10:16:16 GMT

Tuesday, October 16, 2007 2:55 PM by Adi R

> However, I think you are doing a major

> dis-service to us IT community by not

> revealing the name of the gadget. A Rogue

> gadget like this can kill entire Vista,

The Rogue application was Vista itself. Maybe you need to read the article again. The gadget invoked APIs correctly, the defective APIs leaked memory, the defective APIs killed the correctly written gadget, and the defective APIs killed the sidebar.

Once upon a time I would be amazed that Mr. Russinovich was able to submit a bug report without paying a fee, and/or that the answer didn't come back "not reproducible inside of Microsoft". Of course now those days are gone, since Microsoft bought him.

Wednesday, October 17, 2007 12:55 PM by Jean Gauthier

>> "1. somebody isn't checking for NULL after

>> calling malloc(). sidebar.exe should never

>> crash."

> This is false. If after a malloc, you get a

> NULL pointer, what can you do?

Report an error. If doing something less trivial than this case, try to give the user a chance to save their work, even though you can't make any further changes to it. Try to send a crash dump to the developer.

> Trying to prevent that makes only the program

> crash elsewhere and its really tough track

Bingo. That's exactly what happened. That's exactly why you need to check the result of malloc instead of barging on ahead like Vista does.

re: The Case of the Frozen Clock Gadget

jgurtz — Tue, 23 Oct 2007 22:52:00 GMT

I find it most amusing (after coming all this way to a protected address space with NT, and leaving behind the crashy world of DOS/Win3.xx and Win9x!) that we are seeing a problem where one gadget's misfortune is affecting all the others.

Come on now. Haven't we all agreed that reliability trumps efficient RAM usage? .NET/Java and Vista its self seem to be fairly compelling supporting arguments to that in an age where most new machines will be outfitted with 2GB of RAM.

re: The Case of the Frozen Clock Gadget

ShawnNa — Fri, 26 Oct 2007 02:36:18 GMT

Mark, please consider a related but much more sinister leak:

function leakTimer() {

window.setInterval("leakTimer();", 10000);

}

This is one of the rare problems that I was not able to isolate using sysinternal or microsoft tools! The actual problem had a much larger interval so would lock up the machine if left overnight. But besides a slight increase in memory usage and high kernel mode cpu, there's little else to go on. Because it affects the windows message dispatch loop, all UI threads start showing high kernel mode cpu, making it very difficult to even isolate down to the process which caused the problem. Tools that did work (non-UI) like kernrate also showed nothing. All I could tell from instrumenting my (completely unrelated) app was that PeekMessage and GetMessage was taking an unusually long time. In hindsight, this is expected since that looks to be the place WM_TIMER messages are shed or generated. Worse of all, if this was accomplished over a period of time in a native app which did the equivalent of:

for(int i=0; i<30000; i++) {

SetTimer(NULL, i, 0, NULL);

}

you would see no excessive memory or cpu usage in the offending app! It's disappointig that Microsoft doesn't provide some protection from a single badly written app. Any suggetions on how something like this could be tracked down or how to gain insight into the GetMessage/DispatchMessage loop? In the end, I was only able to stumble upon the solution by luck using bear.exe:

http://run.to/sz

and noticing the high number of Timer object in the Systems Process (don't know what that means either as it's shown with pid 0). Sorry for the long post but I thought you might find this interesting and would appreciate any feedback.

re: The Case of the Frozen Clock Gadget

Norman Diamond — Fri, 26 Oct 2007 03:55:33 GMT

"(after coming all this way to a protected address space with NT [...]) that we are seeing a problem where one gadget's misfortune is affecting all the others."

No, we are seeing a problem where the OS's misfortune is affecting all of the applets.

In some ways it's like getting a BSOD, though not as serious. In Windows 2000 if you used Windows Media Player and got a BSOD, then the misfortune affected all running applications (and affected disk files if any had been open for writing at the time). But Windows Media Player wasn't the reason, the video driver was the reason. Microsoft did better testing on video drivers that they put into XP. Now maybe they'll do better testing on user mode libraries that they put into the successor of Vista.

re: The Case of the Frozen Clock Gadget

Woo — Mon, 29 Oct 2007 01:35:25 GMT

Hello Mark,

Your article is really helpful. Thanks.

BTW, I have a question regarding Process Explorer. (Sorry for posting this question here. Your blog is the only place that I can ask a question.) Sometimes ProcExp doesn't show some properties in the properties box's Performance tab like following picture.

http://www.debuglab.com/board/upload/procexp_abnormality.jpg

What happened to the process?

Thanks in advance.

re: The Case of the Frozen Clock Gadget

David Solomon — Tue, 30 Oct 2007 17:15:59 GMT

the "n/a" you are seeing is because you don't have rights to access that information for the LSM process. Try running Process Explorer elevated and you'll see that information.

re: The Case of the Frozen Clock Gadget

John Wiltshire — Mon, 05 Nov 2007 05:08:46 GMT

> This is false. If after a malloc, you get a

> NULL pointer, what can you do?

>Report an error. If doing something less trivial

>than this case, try to give the user a chance to

>save their work, even though you can't make any

>further changes to it. Try to send a crash dump

>to the developer.

Have you ever tried to do any of this when you know you can't allocate anything on the heap?

"Report an error" is going to require heap memory to display a dialog box. Bad luck on that one then.

"Let the user save their work" is definitely going to require heap memory to manage the save file, as well as the dialog boxes and the like needed to present useful information to the user.

"Sending a crash dump" - again, needs heap memory to write the file. Remember, even something as simple as fopen() doesn't work because it ultimately calls malloc().

If malloc() returns NULL then your program is dead in the water. The absolute best you can usually do without having an "emergency space" preallocated to work in (as well as a few file handles open) is simply exit without showing a crash box. Not really much nicer than just flat out crashing, and crashing at least gives the ability for the *system* to send a crash report to the developer.

re: The Case of the Frozen Clock Gadget

Norman Diamond — Wed, 07 Nov 2007 03:09:18 GMT

'"Report an error" is going to require heap memory to display a dialog box.'

Yup. Well let's take one of my examples from last month. Depending on what order some DLL's had been loaded in, I couldn't malloc 1.2GB for a particular structure. Oops, I could get enough heap memory to display a dialog box, so the user could see the error message instead of being left hanging. Sorry for not meeting your quality standards.

'"Let the user save their work" is definitely going to require heap memory to manage the save file'

True, which is why some textbooks recommend allocating that memory when your program starts, and then using it when you need it.

'If malloc() returns NULL then your program is dead in the water'

Go back to school. Do not pass go.

re: The Case of the Frozen Clock Gadget

Frymaster — Mon, 12 Nov 2007 16:30:51 GMT

> I've submitted those automatic crash reports for years, without seeing any change after

> many updates. I've concluded that if I'm very lucky and the crash is resolved,

>it will be in the next version of Windows, not this one.

The only answer really is "it depends" - an annoying occasional bluescreen I got every now and then when using nmap (winpcap was the actual crashing program) was solved by an update to the windows kernel about 3 months after I started sending error reports with my computer model, network card, and mentioning the word "nmap". Maybe that was coincidence, maybe not.

re: The Case of the Frozen Clock Gadget

CJ — Thu, 22 Nov 2007 07:47:44 GMT

I think the reason for this memory leak is simply the recursive call setInterval. All you have to do is log the D/T stamp and you'll see that the # of leakTimer() call will grow exponenially. So each call to leakTimer it in itself will trigger more calls to itself after the set delay (in addition to the original "timed" called).

function leakTimer() {

window.setInterval("leakTimer();", 1000);

}

re: The Case of the Frozen Clock Gadget

ShawnNa — Sat, 24 Nov 2007 07:02:34 GMT

Mark, thanks for the reply. Yes, I know the problem is with a recursive call to SetInterval. The point I was trying to make is actually how badly the operating system behaves under a timer leak (even a non-exponential leak). Try writing a simple C++ application which simply creates lots of timers with a NULL handle. If this was done over a sufficiently long period, you will see that the only visble symptom is a general slowness of all UI threads with excessive kernel mode cpu usage. There will be very few clues that point to what's wrong or which process is causing the slowness. I was wondering what debugging techniques you would employ to track this down.

re: The Case of the Frozen Clock Gadget

Joseph — Thu, 06 Dec 2007 11:17:12 GMT

> If malloc() returns NULL then your program is dead in the water.

Let's try to use malloc(), LocalAlloc() or HeapAlloc() to allocate 2GB memory at a time, you will definitely receive a NULL pointer. But this does not mean that you program have crashed. The NULL value only means that the API could not complete your request and memory is not allocated in the user space. You could still do a complete cleanup and graceful exit. Even though your memory allocation will not be as large as 2GB, if the system do not have a large enough continuous memory space, you could still receive a NULL pointer.

What does 'dead in the water' mean? To me, it means a memory chip failed and flips one or two bits, or the heap table is corrupted. In these cases, your program probably hangs in the kernel or issue an exception, and the alloc() functions simply do not return, and you hardly need to think of any way to handle it.

I still remember that several years ago when I read some old programming books, there still had some footnotes saying like "for clarity, the example does not illustrate the handling code, and the readers are responsible for doing this." However, some days later, I saw that most of the people just knew how to copy code, using just the piece of example code without any error handling. What a bad habit......

re: The Case of the Frozen Clock Gadget

James — Thu, 13 Dec 2007 12:21:29 GMT

I think Joseph and Norman are right: unless it was a very, very small memory allocation which failed (i.e. you've actually used up 99.999% of your available RAM, usually through a huge number of tiny allocations, as in the memory leak case here) you *can* - and should - try to report the error. It's much more likely for a larger allocation to fail, which can easily be reported and recovered from sensibly. Often the failure isn't a true shortage of resources, but erroneous usage of them: in Norman's case, a DLL which interferes with very large allocations later, in other cases a corrupt or misinterpreted file. (Rename almost any data file to .com/.exe and try running it, you'll get a memory allocation error from Windows: it tries to interpret the data as a program header, resulting in a 'program' which has nonsensical memory requirements like requesting 800k of the 640k conventional memory area.)

re: The Case of the Frozen Clock Gadget

Philip from Australia — Sat, 22 Dec 2007 12:22:05 GMT

ok, it's been a while since I have programmed. But couldn't you pre-allocate some space on the heap in order to gracefully handle an error event like that? Maybe you'd need a GOTO to get to it, but isn't that ok if your code is in a 'oh heck, I have to quit NOW' situation?

Just a thought for graceful exiting in a memory allocation failure situation.

re: The Case of the Frozen Clock Gadget

Bio — Sun, 30 Dec 2007 20:15:37 GMT

2 CJ and ShawnNa

there was window.setTimeout and not window.setInterval

Bill Gates now an Honorary Antiaging Wunderkind

Peter J. Lupo Esq. — Thu, 28 Feb 2008 06:26:44 GMT

This seemed a fitting place to make this wonderful announcement.

See the list w/ babyfaced Mr. Gates on it here:

antiaging4geeks.com

Just scroll down the page a bit.

AND, just for the record, I noticed this exact problem with my sidebar gadgets: it is especially interesting when an afterimage remains and I have to left click and attempt to select the area in order to remove it.

Most interesting.

I had been the most rabid Microsoft/Windows/PC supporter in the multiverse until recently. I am still a fan, but endless glitches and the iPhone have led me somewhat astray.

Still love Vista, though, for all its issues.

Peter J. Lupo Esq.

antiaging4geeks.com

re: The Case of the Frozen Clock Gadget

gfig — Mon, 17 Mar 2008 04:07:18 GMT

Unbelievably, this is not fixed in SP1. I'm curious if your "report" to Microsoft was even looked at.

Something tells me it sits in the same status as it did when reported in October 07. A shame really, because though I can turn the sidebar off, or use an alternative, you'd think Microsoft would give attention to something most casual PC users would play with. For these are the people who will be affected most, and not have a clue about the cause.

Thanks.

re: The Case of the Frozen Clock Gadget

Richard — Wed, 13 Aug 2008 11:49:40 GMT

Hi Mark,

Another great and informative blog, just one questions/critisism.

Can you please include the commands you are using in Windbg, when you use them, there are of course resources out their that list all of them, but some of us are fairly new to this game and a step by step would save alot of time and error when trying to run similar bugchecks

Thanks

Richard

re: The Case of the Frozen Clock Gadget

gina — Tue, 19 Aug 2008 15:23:43 GMT

Hi,

My gadgets have completely frozen!!!! I downlaoded about 30 new ones today & they all kept going on the sidebar which is now frozen. I shouldn't have put them all there but the side bar just kept making new columns & I've only had my laptop for about a month. It's a great computer, i have not had one single issue. Now the whole side bar is black, even after trying to restart the computer about 10 times!!! Please help! What should I do? You can email me at blondebabyfashionista@yahoo.com.au

Thankyou so much

re: The Case of the Frozen Clock Gadget

Yuhong Bao — Thu, 30 Oct 2008 04:43:01 GMT

So if I am experiencing the problem right now, what should I do? Contact PSS and get a hotfix?

PS: I think you need to do a blog article on this.