Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
The most interesting cases I receive are those that demonstrate a unique troubleshooting technique or uncover an interesting root cause. I received this one recently that has both characteristics. The case opened when a systems administrator got a report from a user that they were unable to print from their computer. There was no visible reaction to clicking on a print dialog or menu item, where normally they saw a dialog stating that the document had been sent to the printer and a tray icon appear representing the active print queue.
The first thing the administrator did was to scan the event logs of the user’s machine, looking for any printing-related events. He quickly came upon two that correlated with the user’s most recent printing attempt:
It appeared that the Spooler Service started when the user tried to print, but terminated, apparently with an exception (unexpectedly), about a minute later. The question was why?
The administrator turned to Sysinternals Procdump. Procdump is a utility that generates crash dump files of a process when triggers you specify occur. Implemented triggers include CPU usage, virtual memory usage, unhandled exceptions, and process termination. You can use the CPU usage trigger, for example, to capture the state of a process when it hits a short-lived CPU burst, allowing you to look into the process to see the reason for the spike. The administrator guessed that the stack trace of the terminating thread might provide a clue.
He knew that he had some time to get Procdump running after the Spooler Service started, so he launched Notepad, tried to print, and then executed Procdump with the –e option and the name of the Spooler Service process (Spoolsv) to have Procdump wait until the service exited before writing the dump file. A few seconds later Procdump reported that it had completed the job and saved a dump file:
He opened the dump in Windbg and executed the ‘k’ command, which has Windbg dump the stack of the thread that caused the crash. The stack trace, which essentially lists a record of the function calls executed before the crash, showed that the process died in a sequence of calls that included several Ldr functions, including LdrpLoadDll:
A web search revealed that LdrpLoadDll is a function related to the system’s DLL loader. Suspecting that the process was dying either because it couldn’t find a DLL it was looking for or was loading an incorrect DLL, he turned to Process Monitor, which would enable him to see the process’s DLL-related file system activity. He started Process Monitor, attempted to print again, and then stopped the capture. Working his way from the end of the trace back to the beginning, he scanned for hints of the root cause. Shortly before Spoolsvc exited, he saw it searching unsuccessfully for Localspl.dll in various directories on the system:
He assumed that the DLL was supposed to be there. When he looked at an another identically configured Windows XP system on his network, he found Localspl.dll in the \Windows\System32 directory, but not on the system experiencing the problem:
The file’s description reported it to be the Local Spooler DLL, which explained the Spooler’s inability to support printing operations. After he copied the file from the working system, he was able to print successfully.
As far as the user was concerned, the case was closed and he was able to get back to work, but the administrator was left with the question of what had happened to the original DLL. Another web search turned up forum posts from others that had experienced the same problem. One post in particular described the exact symptoms he’d seen, including the event log entries, suggested the same fix of copying Localspl.dll from another system, and blamed uninstallers of third-party print and fax software for incorrectly deleting Localspl.dll:
He couldn’t say for certain that was the case for this particular system because the end user didn’t remember uninstalling printing or fax software, but the post had at least given him a plausible theory to replace the unease he would have been left with that files were mysteriously being deleted from his systems. He could now close the case thanks to Procdump and Process Monitor.
If you solve an interesting case with Sysinternals tools, please send me screenshots (.PNG preferred) and log files so that I can share them with others in this blog and my presentations.
Two words immediately pop into my head whenever anything strange, unreliable or just plain shoddy occurs with printing: "Hewlett" and "Packard".
I am very rarely wrong.
The missing file could have been "quarantined" by anti-virus software. I've found that some missing files are due to false-positive matches.
Impressive troubleshooting. That guy deserves a raise. Most people I talk to (and quite possibly myself) would have just rebuilt the system and called it a day. Low tech but efficient.
I'm curious about the near throwaway mention in the forum post screenshot: why wouldn't System File Protection restore the deleted system file; and why didn't SFC note its absence?
Thanks Mark, great article as always. Procdump ought my attention just for a few days a good tool when processes misbehaves.
SFC as far as i understand, is only checking existing files. It doesn't have a list of system files, that should be on system.
Nice job! There is no localspl.dll in the screenshot either, though.
Yeah, why System File Protection didn't helped? Then... How does it work?
yeah, that "System File Protection" thing also could not protect "svchost.exe" from being deleted erronously by McAffee! Mysterious!
Reminds me that I need to debug my wifes computer: on about every 3rd boot the touchpad does not work (reboot fixes), on around every 10th boot touchpad and keyboard don't work (suspend and wakeup fixes keyboard, but not touchpad).
No messages in the event log. Probably just a messy touchpad driver from Fujitsu-Siemens, but installing the most recent did not change much.
When I got time....
Nice find, I solve lots of interesting problems with Process Monitor and Process Explorer, but I never save pictures etc.
I shall do from now on!
impressive! who is this guy, didn't he want the name credit?
and a good word to Yuval Sinai, a well known MVP that prove his excellence time after time :)
Thanks Mark for this wonderful piece of information.
i think today both viri and antiviri and installers/deinstallers know how to modify SFC database, so that SFC would not interfere with them.
Say, some virus registers itself in SFC, so that antiviri cannot delete it.
Then antiviri learn how to deregister it from SFC to cure it.
Then any false positive makes antiviri check if the file is potected by SFC and deregister it, so that SFC would not mind removing FP file.
Armor and bullet race.
I agree that this is a worrying lacuna in SFC's remit - it should do more than check versions, or at least flag up expected files that are missing.