Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
Given that my novel, Zero Day, will be published in a few weeks and is based on malware’s use as a weapon by terrorists, I thought it appropriate to post a case that deals with malware cleanup with the Sysinternals tools. This one starts when Microsoft support got a call from a customer representing a large US hospital network reporting that they had been hit with an infestation of the Marioforever virus. They discovered the virus when their printers started getting barraged with giant print jobs of garbage text, causing their network to slow and the printers to run out of paper. Their antivirus software identified a file named Marioforever.exe in the %SystemRoot% folder of one of the machines spewing files to the printers as suspicious, but deleting the file just resulted in it reappearing at the subsequent reboot. Other antivirus programs failed to flag the file at all.
The Microsoft support engineer assigned the case, started looking for clues by seeing if there were additional suspicious files in the %SystemRoot% directory of one of the infected systems. One file, a DLL named Nvrsma.dll, had a recent timestamp and although it was named similarly to Nvidia display driver components, the computer in question didn’t have an Nvidia display adapter. When he tried to delete or rename the file, he got a sharing violation error, which meant that some process had the file open and was preventing others from opening it. There are several Sysinternals tools that will list the processes that have a file open or a DLL loaded, including Process Explorer and Handle. Because the file was a DLL, though, the engineer decided on the Sysinternals Listdlls utility, which showed that the DLL was loaded by one process, Winlogon:
Winlogon is the core system process responsible for managing interactive logon sessions, and in this case was also the host for a malicious DLL. The next step was to determine how the DLL was configured to load into Winlogon. That had to be via an autostart location, so he ran the Autoruns utility, but there was no sign of Nvrsma.dll and all the autostart entries were either Windows components or legitimate third-party components. That appeared to be a dead end.
If he could watch Winlogon’s startup with a file system and registry monitoring utility like Process Monitor, he might be able to determine the magic that got Winlogon to load Nvrsma.dll. Winlogon starts during the boot process, however, so he had to use Process Monitor’s boot logging feature. When you configure Process Monitor to log boot activity, it installs its driver so that the driver loads early in the boot process and begins monitoring, recording activity to a file named %SystemRoot%\Procmon.pmb. The driver stops logging data to the file either when someone launches the Process Monitor executable or until the system shuts down.
After configuring Process Monitor to capture boot activity and rebooting the system, the engineer ran Process Monitor and loaded the boot log. He searched for “nvrsma” and found this query by Winlogon of the registry value HKLM\Software\Microsoft\Windows NT\CurrentVersion\bwpInit_DLLs that returned the string “nvrsma”:
The engineer had never seen a value named bwpInit_DLLs, but the name was strikingly similar to an autostart entry point he did know of, AppInit_DLLs. The AppInit_DLLs value is one that User32.dll, the main window manager DLL, reads when it loads into a process. User32.dll loads any DLLs referenced in the value, so any Windows application that has a user-interface (as opposed to being command-line oriented) loads the DLLs listed in the value. Sure enough, a few operations later in the trace he saw Winlogon load Nvrsma.dll:
Its power to cause a DLL to get loaded into virtually every process has made AppInit_DLLs a favorite of malware authors. In fact, it’s become such a nuisance that in Windows 7 the default policy requires that DLLs listed in the value be code-signed to be loaded.
The boot trace had no reference to AppInit_DLLs, making it obvious that the malware had somehow coerced User32.dll into querying the alternate location. It also explained why the entry hadn’t shown up in the Autoruns scan. One question he had was why no other process had Nvrsma.dll loaded into it, but further into the trace he saw that an attempt to load the DLL by another process resulted in the same sharing violation error he’d encountered:
Simply loading a DLL won’t cause a handle to remain open and cause this kind of error, so he searched backward, looking for other CreateFile operations on the DLL that had no corresponding CloseFile operation. The last such operation before the sharing violation was performed by Winlogon:
The stack of the operation, which he viewed by double-clicking on the operation to open the properties dialog and then clicking on the Stack tab, showed that it was Nvrsma.dll itself that opened the file, presumably to protect itself from being deleted and to prevent itself from loading into other processes:
Now he had to determine how User32.dll was compromised. User32.dll is one of the system “Known DLLs”, which means that as a performance optimization Windows creates a file mapping at boot time that can then be used by any process that loads the DLL. These known DLLs are listed in a registry key that Autoruns lists in the KnownDLLs tab, so the engineer went back to the Autoruns scan to take a closer look. The most effective way to spot potential malware when using Autoruns is to run it with the Verify Code Signatures option set, which has Autoruns check the digital signature of the images it finds. Upon closer inspection, the engineer noticed that User32.dll, unlike the rest of the Known DLLs, did not have a valid digital signature:
The compromised User32.dll behaved almost identically to the actual User32.dll, otherwise applications with user-interfaces would fail, but it seemed to be different enough to cause it to query the alternate registry location. To verify this, he ran the Sysinternals Sigcheck utility on the tweaked copy and on one from a different, uninfected, system that was running the same release of Windows. A side-by-side comparison of the output, which includes MD5, SHA-1 and SHA-256 cryptographic hashes of the file, confirmed they were different:
As a final check to make sure that the difference was indeed responsible for the different behavior, the engineer decided to scan the strings in the DLL. Any registry keys and values, as well as file names, used by an executable will be stored in the executable’s image file and be visible to a string-scanning tool. He tried using the Sysinsternals Strings utility, but the sharing violation error prevented Strings from opening the compromised User32.dll, so he turned to Process Explorer. When you open the DLL view for a process and open the properties of a DLL, Process Explorer shows the printable strings on the Strings tab. The results, which revealed the modified APPInit_DLLs string, validated his theory:
With the knowledge of exactly how the malware’s primary DLL activated, the engineer set out to clean the malware off the system. Because User32.dll would be locked by the malware whenever Windows was online (otherwise you can rename the file and replace it, which is what the malware did), he booted the Windows Preinstallation Environment (WinPE) off a CD-ROM and from there copied a clean User32.dll over the malicious version. Then he deleted the associated malware files he’d discovered in his investigation. When through, he rebooted the system and verified that the system was clean. He closed the case by giving the hospital network administrators the cleaning steps he’d followed and submitted the malware to the Microsoft antimalware team so that they could incorporate automated cleaning into Forefront and the Malicious Software Removal Toolkit. He’d solved a seemingly impossible case by applying several Sysinternals utilities and helped the hospital get back to normal operation.
It's great reading stuff like this on your blog, although I can't help but to feel that the support engineer had his answer a few steps before he executed a solution. Was this a rational precaution as in better safe then sorry?
I have to say though the comments on this one leave me a bit surprised. It was pretty clear from the start to me at least that this hospitol was running a dated OS (XP I would guess), yet some of the posters don't seem to realise how far along MS Securtiy Center has come in its past few iterations. Also you would think that fellow Sys Admins would know some of the basics behind stuff like this: en.wikipedia.org/.../DLL_injection
@ComputerGeek01: Well, the way Mark laid it out to us, yes, you are right. But on scene, things aren't simple and straightforward as they look from here.
Still, if the things on scene were as they look here, ... well, you'd have been right...
What I get from reading this (hopefully right!): If we see an unverified user32.dll on the Known_DLLs tab in Autoruns, Appinit could have been tampered and we should not trust what is shown on the Appinit tab anymore. Instead, we should inspect the Appinit registry entry manually.
Probably a side note or note for a new topic, but pstools has been and now the suite is part of my everyday usage. Is it possible to make it a 1 time "I AGREE" for all the tools rather than have it for each tool, for every machine, for every profile they are run under? It is so ignored anyway and a few time I got stuck with a passive psexec hung on another machine waiting for the "I AGREE" user input.
For PsExec, there's the switch -accepteula.
To accept all EULAs for all tools at once, you may 'Bing' this script by a Sysinternal forum member: "syseula.cmd". It will set the AcceptEula key in the registry for all tools.
Alternatively, there's a third party automated tool to install all Sysinternals tools, accept the EULA, update system path, unattended, all at once. You may 'Bing' 'SSIBuild'. It is a tool made by DarkShadows of the MSFN forum.
Anti-virus doesn't flag an unverified dll, gotta love the usefullness of A/V.
I am very impressed with story, the tech and the the description of tools used. It was a very interesting and entertaining read as well as informative. It was loaded with information regarding the registry and what happens at Windows start-up. I really enjoyed it, thank you.
Good job done! What an awesome tech article! :)
I do about 10 or so virus removals per week and I refuse to reinstall unless there is corruption to the os. I can remove most virus in an hour, but occasionally I will run into a virus that keeps me up half the night. My question is this. Can a rootkit completely hide it's autostart entries from Autoruns. Does a rootkit hide it's malicious files from all software or just software it is programmed to be aware of? And my last question. Can a virus show as "verified" in Autoruns? In other words, can I trust all files that show as verified in Autoruns? Just things I have wondered and never really got a straight answer on.
Curious if at the time of infection (if it was windows 7) the user would have got a UAC dialog. Moral of the story is don't run under and administrative account. I wish i could heed this device but i can't. I'm a developer and i just run into to many problems IIS, ect....but running under a limit account. God knows i've tried. My solution is just to run utilities like sandboxie & appguard though i know these things are just to dang complicated for the general user.
@ advcom: A rootkit can hide a virus completely (if it is weel written). And yes a virus can be verified - it just take $70 to buy a signing cert. The majority of virus' are not signed though.
Great story, every one I read I end up using one more of the tools on a regular basis. I hadn't even thought of using boot time logging with procmon until I saw it at Tech-Ed last year.
@wanderSick thanks for the tip on the EULA batch file
Great article! Reminds me about hex editing of "Regedit.exe" to circumvent the "DisableRegistryTools" Policy.