Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
Given that my novel, Zero Day, will be published in a few weeks and is based on malware’s use as a weapon by terrorists, I thought it appropriate to post a case that deals with malware cleanup with the Sysinternals tools. This one starts when Microsoft support got a call from a customer representing a large US hospital network reporting that they had been hit with an infestation of the Marioforever virus. They discovered the virus when their printers started getting barraged with giant print jobs of garbage text, causing their network to slow and the printers to run out of paper. Their antivirus software identified a file named Marioforever.exe in the %SystemRoot% folder of one of the machines spewing files to the printers as suspicious, but deleting the file just resulted in it reappearing at the subsequent reboot. Other antivirus programs failed to flag the file at all.
The Microsoft support engineer assigned the case, started looking for clues by seeing if there were additional suspicious files in the %SystemRoot% directory of one of the infected systems. One file, a DLL named Nvrsma.dll, had a recent timestamp and although it was named similarly to Nvidia display driver components, the computer in question didn’t have an Nvidia display adapter. When he tried to delete or rename the file, he got a sharing violation error, which meant that some process had the file open and was preventing others from opening it. There are several Sysinternals tools that will list the processes that have a file open or a DLL loaded, including Process Explorer and Handle. Because the file was a DLL, though, the engineer decided on the Sysinternals Listdlls utility, which showed that the DLL was loaded by one process, Winlogon:
Winlogon is the core system process responsible for managing interactive logon sessions, and in this case was also the host for a malicious DLL. The next step was to determine how the DLL was configured to load into Winlogon. That had to be via an autostart location, so he ran the Autoruns utility, but there was no sign of Nvrsma.dll and all the autostart entries were either Windows components or legitimate third-party components. That appeared to be a dead end.
If he could watch Winlogon’s startup with a file system and registry monitoring utility like Process Monitor, he might be able to determine the magic that got Winlogon to load Nvrsma.dll. Winlogon starts during the boot process, however, so he had to use Process Monitor’s boot logging feature. When you configure Process Monitor to log boot activity, it installs its driver so that the driver loads early in the boot process and begins monitoring, recording activity to a file named %SystemRoot%\Procmon.pmb. The driver stops logging data to the file either when someone launches the Process Monitor executable or until the system shuts down.
After configuring Process Monitor to capture boot activity and rebooting the system, the engineer ran Process Monitor and loaded the boot log. He searched for “nvrsma” and found this query by Winlogon of the registry value HKLM\Software\Microsoft\Windows NT\CurrentVersion\bwpInit_DLLs that returned the string “nvrsma”:
The engineer had never seen a value named bwpInit_DLLs, but the name was strikingly similar to an autostart entry point he did know of, AppInit_DLLs. The AppInit_DLLs value is one that User32.dll, the main window manager DLL, reads when it loads into a process. User32.dll loads any DLLs referenced in the value, so any Windows application that has a user-interface (as opposed to being command-line oriented) loads the DLLs listed in the value. Sure enough, a few operations later in the trace he saw Winlogon load Nvrsma.dll:
Its power to cause a DLL to get loaded into virtually every process has made AppInit_DLLs a favorite of malware authors. In fact, it’s become such a nuisance that in Windows 7 the default policy requires that DLLs listed in the value be code-signed to be loaded.
The boot trace had no reference to AppInit_DLLs, making it obvious that the malware had somehow coerced User32.dll into querying the alternate location. It also explained why the entry hadn’t shown up in the Autoruns scan. One question he had was why no other process had Nvrsma.dll loaded into it, but further into the trace he saw that an attempt to load the DLL by another process resulted in the same sharing violation error he’d encountered:
Simply loading a DLL won’t cause a handle to remain open and cause this kind of error, so he searched backward, looking for other CreateFile operations on the DLL that had no corresponding CloseFile operation. The last such operation before the sharing violation was performed by Winlogon:
The stack of the operation, which he viewed by double-clicking on the operation to open the properties dialog and then clicking on the Stack tab, showed that it was Nvrsma.dll itself that opened the file, presumably to protect itself from being deleted and to prevent itself from loading into other processes:
Now he had to determine how User32.dll was compromised. User32.dll is one of the system “Known DLLs”, which means that as a performance optimization Windows creates a file mapping at boot time that can then be used by any process that loads the DLL. These known DLLs are listed in a registry key that Autoruns lists in the KnownDLLs tab, so the engineer went back to the Autoruns scan to take a closer look. The most effective way to spot potential malware when using Autoruns is to run it with the Verify Code Signatures option set, which has Autoruns check the digital signature of the images it finds. Upon closer inspection, the engineer noticed that User32.dll, unlike the rest of the Known DLLs, did not have a valid digital signature:
The compromised User32.dll behaved almost identically to the actual User32.dll, otherwise applications with user-interfaces would fail, but it seemed to be different enough to cause it to query the alternate registry location. To verify this, he ran the Sysinternals Sigcheck utility on the tweaked copy and on one from a different, uninfected, system that was running the same release of Windows. A side-by-side comparison of the output, which includes MD5, SHA-1 and SHA-256 cryptographic hashes of the file, confirmed they were different:
As a final check to make sure that the difference was indeed responsible for the different behavior, the engineer decided to scan the strings in the DLL. Any registry keys and values, as well as file names, used by an executable will be stored in the executable’s image file and be visible to a string-scanning tool. He tried using the Sysinsternals Strings utility, but the sharing violation error prevented Strings from opening the compromised User32.dll, so he turned to Process Explorer. When you open the DLL view for a process and open the properties of a DLL, Process Explorer shows the printable strings on the Strings tab. The results, which revealed the modified APPInit_DLLs string, validated his theory:
With the knowledge of exactly how the malware’s primary DLL activated, the engineer set out to clean the malware off the system. Because User32.dll would be locked by the malware whenever Windows was online (otherwise you can rename the file and replace it, which is what the malware did), he booted the Windows Preinstallation Environment (WinPE) off a CD-ROM and from there copied a clean User32.dll over the malicious version. Then he deleted the associated malware files he’d discovered in his investigation. When through, he rebooted the system and verified that the system was clean. He closed the case by giving the hospital network administrators the cleaning steps he’d followed and submitted the malware to the Microsoft antimalware team so that they could incorporate automated cleaning into Forefront and the Malicious Software Removal Toolkit. He’d solved a seemingly impossible case by applying several Sysinternals utilities and helped the hospital get back to normal operation.
Bravo! That was an awsome case of great detective work and demonstration the use of different tools. I am very impress. Kudos to the tech.
Good article, but how did the malware replace User32.dll while Windows
was running in the first place? As you stated above, Windows will
always be running this dll. If there is a way to do this can you
perform the same method instead of dropping into WinPE?
Awesome work indeed! Can you please elaborate how the malware infected the original User32.dll if it's locked whenever Windows is online (MoveFileEx?)?
Because bwpInit and appInit are the same length, I'm assuming they used an automated binary editor to edit the string inside the dll? Reminds me of a hack we used a few jobs back where every time we would exit the fax application, it would return some unexplainable error. Other than that, the app worked perfectly. So I edit the DLL with that error message and replaced the string with a "Thanks for using FaxWorks!" message that was exactly the same length string! :)
@Yaron, @Eric: I clarified the closing paragraph to make it clear that it's the malware's open handle that locks the file. Under normal circumstances you can rename system DLLs, since the file mappings created by the loader don't keep the file handles open.
A procmon pmb file will survive a bsod and a pml won't right? Is there a way to have procmon log to a pmb instead of a pml without going through boot logging?
Couldn't the malware also use the MoveFileEx-API and Mark's own MoveFile.exe to replace files? It could initiate a reboot or just wait for the user to reboot.
I thought Windows protected core files like User32.dll from being replaced. Was this disabled in some way? Is user32.dll not protected?
Wouldn't whitelisting solutions be a better alternative to this attack?
What is not explained is how User32.dll got patched in the first place.
Also I find is strange why the malware author bothered with creating a payload dll that he loaded with a patched User32.dll. Surely if he found a way to patch User32.dll he could drop his payload directly there, why bother with a different dll.
Why don't we give Windows stronger tamper resistance or detection - at least for objects like knownDLLs?
Where was WFP during all of this? Granted WFP can be disabled, but couldn't Windows could at least detect the appearance of an unsigned or unverifiable knownDLL and log/alert on this?
If the size of the original DLL changes, it will refuse to load or crash.
@Craig, @Aram: there's no magic bullet here. Windows does try and prevent tampering: in Windows XP Windows File Protection replaces modified versions with one from a backup location (\DllCache), but the malware modifies that version as well. On Windows Vista the files are locked down so that only the Trusted Installer account. But the fundamental problem remains: if malware is executing with admin rights, it can circumvent any protection.
How about digital signatures and hash checking (at least added to security essentials if not windows itself)?
@Sol I'm not sure you read what I wrote: "if malware is executing with admin rights, it can circumvent any protection".