Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
Given that my novel, Zero Day, will be published in a few weeks and is based on malware’s use as a weapon by terrorists, I thought it appropriate to post a case that deals with malware cleanup with the Sysinternals tools. This one starts when Microsoft support got a call from a customer representing a large US hospital network reporting that they had been hit with an infestation of the Marioforever virus. They discovered the virus when their printers started getting barraged with giant print jobs of garbage text, causing their network to slow and the printers to run out of paper. Their antivirus software identified a file named Marioforever.exe in the %SystemRoot% folder of one of the machines spewing files to the printers as suspicious, but deleting the file just resulted in it reappearing at the subsequent reboot. Other antivirus programs failed to flag the file at all.
The Microsoft support engineer assigned the case, started looking for clues by seeing if there were additional suspicious files in the %SystemRoot% directory of one of the infected systems. One file, a DLL named Nvrsma.dll, had a recent timestamp and although it was named similarly to Nvidia display driver components, the computer in question didn’t have an Nvidia display adapter. When he tried to delete or rename the file, he got a sharing violation error, which meant that some process had the file open and was preventing others from opening it. There are several Sysinternals tools that will list the processes that have a file open or a DLL loaded, including Process Explorer and Handle. Because the file was a DLL, though, the engineer decided on the Sysinternals Listdlls utility, which showed that the DLL was loaded by one process, Winlogon:
Winlogon is the core system process responsible for managing interactive logon sessions, and in this case was also the host for a malicious DLL. The next step was to determine how the DLL was configured to load into Winlogon. That had to be via an autostart location, so he ran the Autoruns utility, but there was no sign of Nvrsma.dll and all the autostart entries were either Windows components or legitimate third-party components. That appeared to be a dead end.
If he could watch Winlogon’s startup with a file system and registry monitoring utility like Process Monitor, he might be able to determine the magic that got Winlogon to load Nvrsma.dll. Winlogon starts during the boot process, however, so he had to use Process Monitor’s boot logging feature. When you configure Process Monitor to log boot activity, it installs its driver so that the driver loads early in the boot process and begins monitoring, recording activity to a file named %SystemRoot%\Procmon.pmb. The driver stops logging data to the file either when someone launches the Process Monitor executable or until the system shuts down.
After configuring Process Monitor to capture boot activity and rebooting the system, the engineer ran Process Monitor and loaded the boot log. He searched for “nvrsma” and found this query by Winlogon of the registry value HKLM\Software\Microsoft\Windows NT\CurrentVersion\bwpInit_DLLs that returned the string “nvrsma”:
The engineer had never seen a value named bwpInit_DLLs, but the name was strikingly similar to an autostart entry point he did know of, AppInit_DLLs. The AppInit_DLLs value is one that User32.dll, the main window manager DLL, reads when it loads into a process. User32.dll loads any DLLs referenced in the value, so any Windows application that has a user-interface (as opposed to being command-line oriented) loads the DLLs listed in the value. Sure enough, a few operations later in the trace he saw Winlogon load Nvrsma.dll:
Its power to cause a DLL to get loaded into virtually every process has made AppInit_DLLs a favorite of malware authors. In fact, it’s become such a nuisance that in Windows 7 the default policy requires that DLLs listed in the value be code-signed to be loaded.
The boot trace had no reference to AppInit_DLLs, making it obvious that the malware had somehow coerced User32.dll into querying the alternate location. It also explained why the entry hadn’t shown up in the Autoruns scan. One question he had was why no other process had Nvrsma.dll loaded into it, but further into the trace he saw that an attempt to load the DLL by another process resulted in the same sharing violation error he’d encountered:
Simply loading a DLL won’t cause a handle to remain open and cause this kind of error, so he searched backward, looking for other CreateFile operations on the DLL that had no corresponding CloseFile operation. The last such operation before the sharing violation was performed by Winlogon:
The stack of the operation, which he viewed by double-clicking on the operation to open the properties dialog and then clicking on the Stack tab, showed that it was Nvrsma.dll itself that opened the file, presumably to protect itself from being deleted and to prevent itself from loading into other processes:
Now he had to determine how User32.dll was compromised. User32.dll is one of the system “Known DLLs”, which means that as a performance optimization Windows creates a file mapping at boot time that can then be used by any process that loads the DLL. These known DLLs are listed in a registry key that Autoruns lists in the KnownDLLs tab, so the engineer went back to the Autoruns scan to take a closer look. The most effective way to spot potential malware when using Autoruns is to run it with the Verify Code Signatures option set, which has Autoruns check the digital signature of the images it finds. Upon closer inspection, the engineer noticed that User32.dll, unlike the rest of the Known DLLs, did not have a valid digital signature:
The compromised User32.dll behaved almost identically to the actual User32.dll, otherwise applications with user-interfaces would fail, but it seemed to be different enough to cause it to query the alternate registry location. To verify this, he ran the Sysinternals Sigcheck utility on the tweaked copy and on one from a different, uninfected, system that was running the same release of Windows. A side-by-side comparison of the output, which includes MD5, SHA-1 and SHA-256 cryptographic hashes of the file, confirmed they were different:
As a final check to make sure that the difference was indeed responsible for the different behavior, the engineer decided to scan the strings in the DLL. Any registry keys and values, as well as file names, used by an executable will be stored in the executable’s image file and be visible to a string-scanning tool. He tried using the Sysinsternals Strings utility, but the sharing violation error prevented Strings from opening the compromised User32.dll, so he turned to Process Explorer. When you open the DLL view for a process and open the properties of a DLL, Process Explorer shows the printable strings on the Strings tab. The results, which revealed the modified APPInit_DLLs string, validated his theory:
With the knowledge of exactly how the malware’s primary DLL activated, the engineer set out to clean the malware off the system. Because User32.dll would be locked by the malware whenever Windows was online (otherwise you can rename the file and replace it, which is what the malware did), he booted the Windows Preinstallation Environment (WinPE) off a CD-ROM and from there copied a clean User32.dll over the malicious version. Then he deleted the associated malware files he’d discovered in his investigation. When through, he rebooted the system and verified that the system was clean. He closed the case by giving the hospital network administrators the cleaning steps he’d followed and submitted the malware to the Microsoft antimalware team so that they could incorporate automated cleaning into Forefront and the Malicious Software Removal Toolkit. He’d solved a seemingly impossible case by applying several Sysinternals utilities and helped the hospital get back to normal operation.
Hi Mark, great post! I was just wondering how to wondering how to fight malware which doesn't have an associated autorun entry! Recently I read somewhere where a rootkit modifies a windows kernel mode driver and every time, winodws loads with no autorun entry, the malware also loads with windows! Moreover the modified file and the original file also were of exact same length.
I guess only CRC/MD5 check will find the difference. In such a case, will the verify code signatures find the difference? I think it has started to become more and more difficult clean malware without reinstalling the OS!
Also could you please tell me, if such a rootkit loads with a kernel mode driver, is it possible for that rootkit fool all the tools and conceal it's presence? Any chance to find it and clean it in WinXP and Win7?
Point taken. Can the system be re-structured, so as to limit the admin rights with regards to core system files; something like a super-system account, for internal os use only (at least w/o an "pre"-boot intervention)? Can we expect future security development in the system files area?
Would a mainboard with TPM and white-listing all the signed apps in a PC protect against this attack?
..understand the malware problem is unsolved; I'm thinking more along the lines of survivability and scalable detection & response. Why not incorporate and automate some of the Sysinternals tools into Windows and make Windows more tamper evident? For example, detect and generate an alert when an unverifiable knownDLL shows up..or an unsigned program file in \system32\..run process monitor in the background with circular logging..and while we're at it, make it easier to natively export logs and events out of windows to a log aggregation and/or threat management tool
Mark, the question is not whether corrupt user32.dll could or could not get onto the system. The question is how exactly did it happen?
If you don't answer that, you can't reasonalby close the case. 20 minutes after you clean the system it might get corrupted again through the same way it was infected before.
You might still be riding the elevator approaching the ground floor to exit the hospital — and that very moment the user32.dll gets replaced, computers restarted, the winlogong pulls the traitor in and the printers wake up to produce more garbage.
@Mihailik Unfortunately, that's a question just about anyone fighting a new malware infection will have a near impossible time of determining. Unless you actually see the infection as it takes place, you can't know - it could have been someone executing a malicious email attachment, opening an infected document, or via a network-spreading worm. Just because you don't know for sure how the infection happened doesn't mean that you shouldn't attempt a cleanup. And in an ideal world, once you find an infection you would repave every system on your network. Unfortunately, that's just not practical in most cases and even then the infection could occur again, if for example it spread from outside via a network vulnerability or the user that caused it in the first place launched the attachment again.
@Mark "I'm not sure you read what I wrote: "if malware is executing with admin rights, it can circumvent any protection"."
HDD or motherboard firmware can calculate a hash of every byte (not file) loaded during the boot process and compare this on every boot. This could be achieved by putting all the critical and rarely changing boot & OS sectors in one continous area and when the firmware detects that the read is happening inside this area it will perform the hashing, so that often changing files are not included in the hash.
Going with the "KISS" principle, the BIOS should simply either stop the boot and ask the user if the user performed a update involving system files, if user did not then it was malware. For IT-scenarios there'd need to be bit more involved process, possibly with the BIOS talking to a service managing the computers in the network.
Also your statement suggests that anyone running with admin right can instantly circumvent any protection. Reality check: Only poor protections are cracked by quick by "career crackers". Read up on cracking the consoles and certain PC music software which took years - enough time to overhaul the protection for next major version if they wanted (but they didn't so it was cracked quicker).
So realistically if Windows had proper protections, just getting to Admin wouldn't mean a total compromise, since ANY GOOD protection isn't just ONE layer (being admin) but a matrix of several layers of checksum checks and boundaries.
This is just a case of MS putting hands up "if they get admin game over" well obviously that is false truth that's repeated a lot by tech press but I know there's few people beside me that know better. The issue with TPM & Intel is the way it was marketed and published by tech press. It didn't come out as a security solution to protect users from bad things, it came as a way to lock people out of their computers and to protect IP & rights - what better way to get tech press to shoot you down. And of course it wasn't really thought out well - having such thing baked into CPU without choice isn't good due to perception that causes. It needs to be optional and in the motherboard (firmware). Of course later it turned out the TPM had problems but it was doomed even before the implementation issues came to light.
@zzz Great points: it is possible to really lock things down with secure launch rooted in tamper-resistenc firmware. Even that's not a magic bullet against malware infection, though, and Windows is a general purpose open operating system with a tremendous amount of application compatibility, not a console.
Slight correction, I said "on every boot" but:
The firmware must also hash the boot & OS boot/security file sectors prior to starting the OS update process, not just on boot and then invoke the update to happen from the re-certified files.
There's no doubt other details to be aware of but the critical OS files can be secured, both on disk and in memory. This is prob why Intel bought McAfee, they want to incorporate hash checks during execution. McAfee prob. had patents related to something along these lines.
In order to replace user32.dll under the nose of all that Windows Protection one has to 'accept' quite a few nagging message box popups. That activity is usually reflected in Event Logs too.
Given the date/time of nv*.dll you could reasonably narrow down the opportunity window and look there in the logs for the traces.
Or maybe Windows Protection was completely switched off in that hospital? Shouldn't you recommend switching it back on then?
Hopefully you've done all that investigation and just skipped it here for brewity :-)
Great Detective work. Eager to read Zero Day (I had ambitions of writing a software-crime story/book for a long time, so maybe your book will be a good basis for what works. :))
Its seems obvious the infection, as nearly always, with promiscuous and/or unauthorized loading of downloaded software. This generation is too trusting. I don't even think its part of their mindset to have any kind of suspicion - they just click. But too much time is increasingly being spent on this sort of complex detective work and frankly I hope it doesn't become the new norm (as a means to solve the entry point problem).
BTW, great work with Microsoft!
Hector Santos, CTO
Santronics Software, Inc.
At the moment I only use 4 tools with sysinternals, but I guess its time to increase that number.
Mark, please blog more incident response that uses sysinternal tools.
We would love to read and test it.
Nice article containing so many good use of the SysInternals tools. One small typo: "making if obvious" => "making it obvious". :-)
Mark, great war story as always!!
As I was reading through the comments I saw you remark of how it is nearly impossible to identify the initial attack vector. I found that interesting as that is something we always struggle with in our engagements, and while sometimes there is enough evidence left behind to give you a reasonable guess as to the initial vector, that by no means always the case.
So to try to tackle that problem we developed a sensor for our customers that tracks execution on a system, and stores that information. I like to think of it as a Tivo, that allows us to rewind "the show" and see what what happened in slow motion. As you can imagine there have been some hurdles, such as making sense of the huge amount of data collected, but with some trial and error we have been able to tweak it so that the data collected is manageable, and it doesn't impact the user's experience.
While it is still in its beta stage, we have used it in a couple IR incidents and found it to be effective and efficient for identification and remediation and more importantly, has saved us a bunch of time… the one problem with it is…well it has to be installed first :-)
Again thanks for sharing the case, they are always interesting reads.
Edgar Sevilla, CIO
@Decheng Thanks for the feedback and correction!