Robert Hensing's Blog

Software Security . . . and stuff.

Blogs

Advanced hiding techniques: The mystery of the trojaned Winlogon.exe

  • Comments 25
  • Likes

So the war between the miscreants and the first responders / incident responders is just that - it's a war with casulaties (servers, workstations, work life / home life balance) and it is complete with an arms race in the form of stealthing (miscreants) and detection (incident responders) technologies.  They hack and hide - we try to find them and recover the servers.  My goal with this blog is to expose the techniques used by the miscreants to educate the incident responders of the world to keep things at parity.

Over the weekend I got a phone call from Kenny - an engineer on my team out west who had a hacking case he wanted me to review with him.  He shared out the output of our incident response toolkit (called WOLF - Windows Online Forensics).  He had already been reviewing the output and needed some advice.  Kenny had done a great job of identifying a series of intrusions but this intrusion was un-like the hundreds of others we examine each year - with this one we stumbled on to something new and interesting.

The call came in the same way they always do - the customer noticed some strange behavior on the box and his Symantec AV software had mysteriously started flagging some files on his machine as a backdoor.  Here are the relevant event log entries from Symantec (where XXX was the machine name which I have removed to protect the innocent):


1/15/2005 1:57:00 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 1:57:36 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 2:12:20 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 2:15:56 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied
                       

Norton Antivirus had pulled down updated signatures and as soon as they went into effect it identified a 'Backdoor.Trojan' (not very helpful) and moreover - it could not clearn or quarantine the file (even less helpful - but hey, at least you know something's up!).  So the customer called Microsoft and got routed to our team for some investigative work on a possible hacking case.

What our team does in cases like this where there is a suspected intrusion is we talk to customers about working with law enforcement and we encourage them to work with law enforcement on cases where they suspect customer data may have been stolen or tremendous financial loss can or will occur as the result of the intrusion etc.  We also tell them if they're going to work with law enforcement they don't want us touching the box with any of our tools . . . the vast majority of our customers choose not to work with law enforcement and instead would rather work with us.  Law enforcement agencies are notoriously picky about who they will work with on investigations and who they trust as it is all to easy for some hoo-ha (technical term) to come in with guns ablazin' and trample all over critical evidence that may be found on the server in places like the slack space or free space of the hard drive.  I remember vividly over 3 years ago I was en-route to the airport to work on-site with a customer (after having worked with him unsuccessfully for a few hous on the phone) on a very important security case when I got a call from the president of the company I was about to visit telling me to turn around and go home - I wasn't welcome anymore.  I was a bit upset since I had already purchased travel and invested time on the case.  The reason the customer gave?  They had involved the FBI and my response time was . . . ahem, a bit better than theirs.  When he informed the FBI that I was to arrive onsite within 4 hours - they said "it's either him, or us - you decide who you want to work this case" . . . and that was the end of my involvement (and it was fortunate too as I later learned the FBI managed to aprehend the fellow responsible).  I use this story all the time when educating customers on the reasons they may want to get law enforcement involved and why if they do so - I should NOT be involved.

If our customers decide to work with us we send them our IR toolkit known as WOLF which is more of a data collection agent.  This runs on a live system and collects volatile and non-volatile data from the box and it places all of the data in a .CAB file for analysis back here at Microsoft.  We then have a special viewer we've written in C# to make analysis and searching very fast and efficient.  What used to take us 3-4 hours per server can now be done in 15 minutes or less usually.

What we find in the vas majority of cases is that customers who have been hacked once, have usually been hacked more than once and they never knew about previous intrusions.  The latest guy to hack you is usually simply the one who got 'sloppy' and got caught and ruined it for everyone.  This was to be the case with this particular machine.

During the investigation Kenny found a copy of Hacker Defender that had been dropped on the machine back in May of last year (the customers antivirus softawre had still not detected it as of the time of writing and we manually removed it). 
It had been configured to run as the following service:


Dynamic Storage Allocation - [running]
Automatic
LocalSystem
C:\WINNT\system32\msdsa.exe
Provides multiple storage heaps in data transactions for dynamic allocation. If this service is disabled, simultaneous data transactions across the local system may fail.

registry.txt
====================
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\MSDSA]
"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\MSDSA]
"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSDSA]
"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"

The files had been dropped on the box:


Files containing instances of 'msdsa.exe'
Number of Files Searched: 6
Time to Search Files: 1 seconds
dir_creation_time_C_drive.txt

====================
Directory of c:\WINNT\system32
05/16/2004 03:38a 36,352 msdsa.exe

dir_last_access_time_C_drive.txt
====================
Directory of c:\WINNT\system32
01/15/2005 01:23p 36,352 msdsa.exe

dir_last_write_time_C_drive.txt
====================
Directory of c:\WINNT\system32
05/14/2004 02:23p 36,352 msdsa.exe

Looking at other activity around that time, the files appear as if from out of nowhere in the wee hours of the morning and a new user profile directory was created (TSInternaluser, which is also a member of the administrators group - wonder what the password is?) as if it had been used for a local logon shortly thereafter:


2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 183,296 msdfm.exe
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 38,912 admdll.dll
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe
2004-05-16 03:37:00 | dir_last_access_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 36,352 msdsa.exe
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 4,912 msdsadrv.sys
2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd
2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini
2004-05-16 03:40:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings - <DIR> tsinte~1 tsinternaluser

The properties of this account lead us to believe that it was there prior to 5/16/2004 - indicating perhaps it was the administrator account that was used to install the malware (which may indicate it had a weak or easily guessed password):

TsInternalUser
FullName TsInternalUser
AccountType User
Comment This user account is used by Terminal Services
HomeDrive
HomeDir
Profile
LogonScript
Workstations
PswdCanBeChanged Yes
PswdLastSetTime 4/28/2004 12:14 AM
PswdRequired Yes
PswdExpires Yes
PswdExpiresTime 4/28/2005 12:14 AM
AcctDisabled No
AcctLockedOut No
AcctExpiresTime Never
TrueLastLogonTime Never
LastLogonServer XXX
LogonHours All
RasDialin No
RasCallback None
RasCallbackNumber

Given the circumstances under which this malware was installed our best guess based on the evidence we have is that this malware was copied to the machine using the default admin shares (C$ / admin$ etc.) using a local administrator account (TSInternaluser) with a weak password that is only set to expire once a year.

Now that's not what I came here to talk about - this is merely an example of a prior intrusion that had gone un-noticed by this particular customer for over 7 months.  The latest intrusion was caught by Symantec and is the reason the customer called us up.  Let's have a look at that.  We have a starting point, which I like to call "a lead" - and that is the file named by Symantec - USERS.DLL in SYSTEM32.

Let's figure out the MAC times on this file:


Files containing instances of 'users.dll'
Number of Files Searched: 6
Time to Search Files: 0 seconds
dir_creation_time_C_drive.txt
====================
Directory of c:\WINNT\system32
12/20/2004 12:39p 54,272 users.dll

dir_last_access_time_C_drive.txt
====================
Directory of c:\WINNT\system32
01/15/2005 01:21p 54,272 users.dll

dir_last_write_time_C_drive.txt
====================
Directory of c:\WINNT\system32
12/20/2004 12:39p 54,272 users.dll

So it appears that this file was dropped on 12/20/2004 - what else happened on 12/20/2004 - let's have a look:


2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif

XXX was the username of an administrator on the box and has of course been changed to protect the innnocent.


Notice we have some ".GIF" files and a couple DLL's that were dropped around the same time . . . which brings me to my next miscreant hiding technique.  Some miscreants favor active stealthing using rootkits to hide from administrators.  Others prefer to hide in plain site using obfuscation techniques . . . this is an example of the latter.  Do you think those are .GIF's you're looking at?  Do you think that's air your breathing?

Our first clue that these are not really .GIF files comes from a tool written by Matt, an engineer on our team that is designed to find binaries (executable files) that have been renamed to non-executable extensions. (More on this in a minute).  Matt's tool - flags these as executables:


Copied from: findbin_C_drive.txt
==========================
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64\bits[1].gif
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\kill[1].gif
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\list[1].gif

There are various techniques one can use to spot an executable . . . you can use a hex editor . . . you can use something like dumpbin.exe . . . or you can use strings from Sysinternals to pull strings out of the file and have a look.  You can even use Notepad and look for 'MZ' as the first two characters you see when you open the file.

Here are some strings from 'kill.gif':


Strings v2.1
Copyright (C) 1999-2003 Mark Russinovich
Systems Internals -
www.sysinternals.com

jjh
jjj
(null)
         (((((                  H
5/?
   
;T^h<U_i=V`j>Wak?Xbl@YcmAZdnB[eoC\fpD]gq
BINRES
KILLSVC
VS_VERSION_INFO
StringFileInfo
040904b0
Comments
CompanyName
Systems Internals
FileDescription
pkill
FileVersion
1.03
InternalName
pkill
LegalCopyright
Copyright
 1999
LegalTrademarks
OriginalFilename
pkill.exe
PrivateBuild
ProductName
Systems Internals pkill
ProductVersion
1.03
SpecialBuild
VarFileInfo
Translation
jjj
jh1
!This program cannot be run in DOS mode.
r<[

This is obviously not a .GIF.  So why would a miscreant take 'pkill.exe' from Sysinternals and rename it to a .GIF?  The answer may surprise you - did you know that in Windows you can actually run a .GIF?  That's right - take any .EXE and rename it to .GIF . . . now open a command shell (cmd.exe) and navigate to the directory where you placed the renamed .EXE.  Now type <filename>.gif and watch the EXE run.  This trick won't work in Explorer for various reasons I won't get into here - but you can run any file, regardless of its extension from inside a command shell if its executable.  We tend to find that the more advanced miscreants favor 'stealthing' using this technique rather than using rootkits (which they know can be easily detected using a variety of anti-rootkit tools).

On an interesting side note:  I once was involved in a case where the miscreants had hacked a large number of servers . . . we had sent resources on-site and were able to develop a tool that could detect the rootkit which was a new, custom rootkit that I don't think anyone had previously seen or detected before.  We refined our tool to the point where it worked amazingly well and we started pushing it down to machines via logon scripts and startup scripts.  At some point the miscreant got wise to our activities and stopped using the rootkit to hide his backdoor - instead he renamed his backdoor DLL (being hidden by the rootkit previously) to a non-executable extension (like .GIF) and stopped copying the rootkit files to the box.  Yes, I realize that we wouldn't have had that problem had the boxes been disconnected from the Internet during this investigation - but that was advice that was ignored by this customer due to financial costs and SLA's - as is usually the case.

Okay - so we know what the .GIF's are - but what about the DLL's.  At first glance these seem to be legitimate Windows files based on the string version information compiled into the DLL's:


Strings v2.1
Copyright (C) 1999-2003 Mark Russinovich
Systems Internals -
www.sysinternals.com

G!@#$
FILE
VS_VERSION_INFO
StringFileInfo
040904b0
Comments
CompanyName
Microsoft Corporation
FileDescription
kernel
FileVersion
5.00.3700.6690
InternalName
kernel
LegalCopyright
Copyright (C) Microsoft Corp. 1981-1999
LegalTrademarks
OriginalFilename
kernel.dll
PrivateBuild
ProductName
Microsoft(R) Windows (R) 2000 Operating System
ProductVersion
1, 0, 0, 1
SpecialBuild
VarFileInfo
Translation
!This program cannot be run in DOS mode.
1j?
1~<
1j?
1Rich
OWrA
PrcWorks
PEC2
.rsrc
.reloc
.rsrc
8,
ntdll.
ERNEL32WTSGetActiv
eConsoleSessiId
NtQuerySystemInformatO:

But the "PEC2" string right above the .RSRC and .RELOC strings above indicate that this DLL has been compacted / compressed / packed using a utility called PECompact to make it smaller.  Miscreants like to 'pack' their malware for many reasons:

  1. If the packer is new enough - it may be enough to allow 'well known' malware to fly under the radar - avoiding detection by AV software that does not yet have support for this new packer / crypter.
  2. It usually makes the files smaller, much smaller - allowing for faster transfers on those slow links from foreign countries. :)
  3. It makes it harder for skilled investigators to investigate using tools like 'strings' from Sysinternals - they will have to either un-pack the file (if there is an unpacker available) or study it under a debugger which requires an entirely different set of skills.

There are some tools out there to assist incident responders in determining whether a file was packed and if so what it was packed with - one such tool is PeID that I have used with much success.  WARNING:  Studying malware with PeID is extremely dangerous - some of the plug-ins for PeID will actually load the malware to attempt to un-pack it - do not use this tool or others like it on a machine of any importance unless you like living dangerously.

At this point - there are not many interesting strings in the DLL since it has been compressed (it will decompress at runtime as it loads into memory) - so the DLL has been submitted to the antivirus vendors via our relationship with them (the Virus Information Alliance) and they will be pushing out signature updates as needed.  Even though the DLL was packed - there is enough information in the strings output to lead me to believe this is a new version of YYT_HAC's backdoor trojan (for more information on YYT_HAC, if you can read Chinese visit: www.yythac.com).  Here is the relevant string that tipped me off:
@HKDOOR>

I recognize that as the default 'prompt' you get when connecting up to YYT_HAC's backdoor.  This is not the first time I've come across malware from this author being used in the wild.

Now - up until this point we haven't really covered anything really 'new' . . . renaming executables to non-executable extensions, packing binaries, hacking boxes - it's all been done before (woohoohooo!) . . . what I'm about to show you is a development that is fairly new to Windows and signifies perhaps a new technique that is being used by the miscreants to hide from administrators and maintain control of their boxes. 

But first some more background on this investigation.  So we've identified some suspicious files on the disk - but we don't know how or where they are being used on the system and for what purpose (assume we haven't collected the files or run strings on them yet).  Let's find out how these DLL's are being used (i.e. how they get loaded).

checksym.csv
====================
,WINLOGON.EXE,228,C:\WINNT\SYSTEM32\USERS.DLL,SYMBOLS_NO,0,0,"December 31, 1969 18:00:00",0,,,,,,,,,,,,,

According to this output - this DLL is loaded inside of one process and only one process: WINLOGON.EXE. 
There are a few reasons a miscreant would want to load a DLL inside of WINLOGON.EXE.

  1. It runs as SYSTEM and is a required process - if you get your code in it (i.e. your backdoor) - you are running as SYSTEM too
  2. You can steal passwords from inside of WINLOGON
  3. If you try to kill WINLOGON you will bugcheck the box with a 0xC0000021a and the system will reboot.  For more information on the C21a stop code you can read the following KB article I wrote a while back: 156669 How to troubleshoot a "STOP 0xC000021A" error
    http://support.microsoft.com/?id=156669

The down-side to running your code inside of WINLOGON is that it has to be really clean code.  Any unhandled exceptions in WINLOGON.EXE will cause the process to crash which will cause the kernel to shutdown the machine with the STOP code mentioned above.

So there are numerous reasons why a miscreant would want to run code inside of WINLOGON (with some associated risk) and to date there are only a couple of techniques for getting that code there:

  1. You can use the GINADLL registry value to load a DLL inside of WINLOGON (easy)
  2. You can inject a thread into WINLOGON and forcibly load your DLL into WINLOGON post-boot. (advanced)

Option one listed above simply requires adding a registry entry for your DLL to a specific well known registry location discussed in my KB article and is easy to detect.
Option two listed above requires some OTHER malware to auto-start somehow and then inject code into WINLOGON and is harder to detect (you have to find the other malware responsible for injecting the DLL into winlogon and what if that other malware is hidden by a rootkit?)

But what if the criteria for option 1 and option 2 above were not met?  What if the GINADLL registry value was empty?  What if there was no other malware running on the machine that could inject your backdoor DLL into WINLOGON? 

This is the situation we were faced with this weekend.

I'm going to take a Quentin Tarantino like tangent here and we're going to jump to a completely different scene before we revisit this thought and conclude this blog post.

There were 2 DLL's we found that were dropped on this customers machine.  There was USERS.DLL and there was another called USERBIN.DLL.
Let's see if we can find them in the registry.

Here are the search results for USERS.DLL against this customer's registry:

Files containing instances of 'users.dll'
Number of Files Searched: 1
Time to Search Files: 0 seconds

Here are the search results for USERBIN.DLL against this customer's registry:

Files containing instances of 'userbin.dll'
Number of Files Searched: 1
Time to Search Files: 0 seconds
registry.txt
====================
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\BITS\Parameters]
"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\BITS\Parameters]
"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BITS\Parameters]
"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"

What's this?  This is a variation of a hiding technique I've discussed in a previous blog post but with a twist.  As it turns out - on Windows 2000 and later operating systems you can implement a service that runs as either an EXE or as a DLL.  If you choose to implement a service as a DLL it will run in what's called a 'service host' which are those 'svchost.exe' processes you see running on all Windows boxes.  This particular miscreant has written a service dll and has hijacked the DLL used by the BITS service and replaced it with his own entry.  If you were to look in the Services applet in control panel / administrative tools - the BITS service would appear normal (that is it's still pointing to: %SystemRoot%\system32\svchost.exe -k netsvcs).  However, below the BITS service registry entry is a 'Parameters' key and in there is a ServiceDLL value and this registry value is SUPPOSED to point to QMGR.DLL - not userbin.dll.

So while this was rather clever - it was easily detected by our analyst Kenny and he restored the registry value to its proper setting.  In addition to restoring this registry value to its proper setting, he also renamed USERS.DLL to a different file name since we had yet to determine how this DLL was getting loaded inside of WINLOGON.EXE.  He then told the customer to restart and when he did the customers box went into a 'reboot' loop at the place where you would normally see the CTRL-ALT-DELETE screen and the customer could only boot to the recovery console.  At the recovery console Kenny had the customer restore 'USERS.DLL' in the SYSTEM32 directory and reboot - and to his surprise the machine booted successfully.

And now we're back to the point where we previously diverged (I hope I haven't lost anyone).  There were no registry entries for USERS.DLL (assuming no rootkit was hiding their presence) - one might assume then that perhaps USERBIN.DLL, which was starting via the hijacked BITS service registry entry was injecting USERS.DLL into WINLOGON when it started . . . but this is not the case as simply restoring USERS.DLL (and leaving the BITS service fixed and pointing to the proper DLL) was enough to get the box to boot successfully.

At this point - Kenny and I theorized that somehow WINLOGON had become 'dependant' on this DLL to function and that if this DLL was removed, WINLOGON would crash with a C21a and the box would get stuck in an infinite reboot -> C21a cycle.

So how does one make WINLOGON dependant on some 3rd party backdoor DLL?  We were stumped . . . until Kenny noticed something intresting in the WOLF output from MBSA.

Warning MS04-011 835732
File version is greater than expected.
[C:\WINNT\system32\winlogon.exe, 5.0.2195.6970 > 5.0.2195.6898]
Note MS04-016 839643
Please refer to 306460 for a detailed explanation.
Patch NOT Found MS04-031 841533
File has an invalid checksum. [C:\WINNT\system32\winlogon.exe]

This MBSA output is confusing because it checks the version of WINLOGON.EXE against two different patches (normally MBSA would only complain about the most recent missing security update).  MBSA claims that the MS04-011 patch has a version is greater than what it was expecting.  This can be considered normal as sometimes there are some non-security related hotfixes that will redistribute files (like WINLOGON.EXE) that customers can apply to fix non-security related issues.  These 'hotfixes' can actually contain files that supersede, or are newer than, publicly released security updates (which are broadly distributed as opposed to hotfixes which are not).  In that sense - this MBSA output is 'normal' or can be explained in the presence of a non-security hotfix - but then we get to the MS04-031 check and MBSA complains of an invalid checksum?  On a completely differnet update?

First off - that IS the version of WINLOGON.EXE that shipped with MS04-031, but MBSA is saying the CRC32 based checksum is invalid.  There is a limitation in the current version of MBSA where it presently doesn't support multiple checksums for multiple versions of the same file contained in a security update.  This poses a problem on multi-proc machines and security updates where there are two versions of the same file in the security update (one for uniproc machines and one for multiproc machines).  The MSSECURE.XML for now always contains the checksum for the uniproc binaries and it will generate the invalid checksum message on multiproc machines that have the multiproc binaries installed.  This would be readily explainable IF this machine were multi-proc . . . but it is not.

I think you see where I'm going with this - it appears that WINLOGON.EXE itself was modified or replaced.  It is no longer the legitimate copy of WINLOGON.EXE that came with MS04-031 on this customers machine.  This would certainly explain the hard-coded dependancy on USERS.DLL and would explain why WINLOGON.EXE would crash when we deleted / renamed USERS.DLL.

As a final proof point we collected MD5/SHA-1 digests of the suspect WINLOGON.EXE and verified that it no longer matches up with any of the ones we ship!

For those who are curious - here are the MD5/SHA-1 digests of the suspect WINLOGON.EXE

MD5:    3A0F4A1C9FA13EE9D5BD88CA76A20758
SHA-1: 7B8F74B78959E266B91C70283292AB39367B2AC0

Now - some of the more clueful readers may wonder why SFP (System File Protection) did not replace WINLOGON.EXE with the original version or complain loudly about this imposter?  This is a reasonable question - for years I have watched in amusement as the miscreants try to overwrite system file only to have SFP put them back immediately.  I was saddened when @Stake released a tool that would temporarily disable SFP (until the next reboot) allowing miscreants to finally overwrite SFP protected files (the imposters would still be detected at the next SFC.EXE scan).  I was informed by our product group that SFP is a *reliability* feature and not a *security* feature and thus it was never intended to be 'hack proof' - the fact that it had been subverted was not a surprise to any of our security developers who said 'it was only a matter of time'.  I've always wondered why on Windows we don't see more trojaning of system binaries and I've always had two theories on why this almost never happens.

  1. We are a closed source company - it's very hard to take something like WINLOGON a core OS subsystem and write it from scratch without access to the source code and make it work in a way that would not break Windows.
  2. System File Protection, despite being only a reliability feature - was actually a formidable barrier for a while that protected the OS from arbitrary system file overwrites.

It is worth pointing out, however, that both of these barriers have been overcome.  As you may or may not know parts of the Windows 2000 operating system were stolen last year from a partner who had legal access to our source code.  In addition it has been known for over a year how to disable System File Protection during the current boot - allowing anyone with admin rights to replace SFP protected system binaries.

As a final thought - the process responsible for implementing System File Protection is none other than our very own WINLOGON.EXE a process on this customers machine which I no longer trust.

UPDATE:  The antivirus vendors have analyzed these two DLL's (USERS.DLL and USERBIN.DLL) and have come up with the following names for them:
USERS.DLL - Backdoor:Win32/Hackdoor.B
USERBIN.DLL - Backdoor:Win32/Bits.B

Win32/Hackdoor.B is an interesting backdoor.  When you run it you can specify a process you want it to "attach" to.  It will then modify that processes imports table by modifying the binary itself (thus confirming our theory that WINLOGON.EXE had been modified) - thus ensuring that when the process starts the DLL is loaded.  It does this rather well and manages to keep the file size the same making it necessary to generate message digests of the files to tell if they were modified.  When the DLL loads it also loads a driver which can intercept all incoming network traffic so that the backdoor can hijack an existing port (it doesn't need to bind to a new port - it can use any existing port).  Hacker Defender 1.0 has similar capabilities.

Comments
  • Thank you very much for these posts! Very educational and entertaining! It is information like this that lends credence to Microsoft's commitment to security. I look forward to further installments.

  • Sobering stuff. I work with home PCs that are hopefully not exposed to this detailed attack, and one thing I do is kill admin shares such as c$. I also disable "automatically restart on errors", so halting instead of rebooting on a STOP error gives a chance to see what's up.

    IMO the OS should never restart on errors if they occur during OS boot (or if you like, a Safe Mode OS boot). There's no point in endlessly rebooting an unattended PC that never reaches a point it can be remotely admin'd; you just shred the file system.

    In today's world of mainly stand-alone malware, it's easy to forget intrafile infectors exist <g>

    Finally, I notice some common drive-by malware drop a non-malware IDE driver, but what I've read doesn't explain why. It could be to uniquely id the HD (more useful than the session IP) but I'd worry about raw disk access, below NTFS's abstraction layer.

  • What I do not understand is if the system is compromised and so might likely provide false information to a security researcher, how come your wolf analysis program be able to gather all these info and be able to trust it.

  • I believe that Micosoft should provide security guide in the spirit of this weblog instead of their current hardening guides which are very beginner like.

  • In response to Nektars comment about being able to trust data captured from a possibly rootkit'd machine. This is a very real and very valid concern. We spend a lot of time researching rootkits and collecting them off of customers systems. We feel we have a very strong knowledge in this area and our IR toolkit has tools to be able to detect the presence of all known rootkits (and probably even ones we don't know about). In the event that our rootkit tools come back 'clean' and we still can't detect the presence of a rootkit but feel that one must be there - we request an image of the system and we restore that image in a lab and we do our analysis there.

  • And in response to Nektars later comment about security guidance being in blog form vs. guide form - actually we feel exactly the opposite. Our ONLY concern with blogs right now is that they may 'dilute' our security guidance. If it's worthy of being blogged about and its stuff that customers need to know - our thinking is that it should be in a security best practices document / guide vs. in a blog, this way all customers have equal access to it.

  • No needs to be able contact MS Product groups to find out that WFP is not a security measure.

    I've discussed this about 2 years ago with Secure@microsoft team.

    http://www.24.odessa.ua/neowin/secure.txt
    "This implies WFP can be circumvented or cancelled; it should not be presumed a replacement for having properly restricted user roles and a locked down system. So your assertion that WFP is sesigned to keep the system stable is correct - it is not designed as a security feature."

    Unfortunately - some of my security ideas I've discussed with Microsoft - still not implemented :-(

  • NOt sure how you fixed this for sure, but we had a similar problem today... Symantec found the dlls noted, but no fix. Tech had tried removing the dll and/or renaming and then it blue screened and couldn't get back. Ended up restoring (which wouldn't have had to do) and getting a fresh infected system. Files were dropped 12/19/2004. Booted in safe mode, deleted or renamed the dlls, deleted gifs noted, and copied winlogon.exe from the dllcache folder over. Found a user that had been created by someone else in admins group, removed this and we were in business. Thanks for the interesting reading.

  • Sorry I guess I should have called that out in the blog. Since winlogon.exe has been modified on disk to recover the system you need to boot to the recovery console and copy a good copy back to system32 (i.e. from the dllcache or from the latest hotfix uninstall directory). You may even be able to rename winlogon.exe while the system is booted to something like winlogon.bad and then copy the one from the latest security update back to system32

  • We just had a similar problem but we solved it in a different way. One of our W2K Servers was behaving "strange" but there were absolutely NO visible problems. All known virus scanners found NO problems. But there was one, we felt it.
    The solution was tricky :
    We just dig a defrag on the server. In the report the server said : ...was unable to defrag the following files... -> There was a listing of non visible files. Their location was in %SYSROOT%\SYSTEM32\drivers\etc\fonts{00 08...}\

    Impossible to browse, impossible to delete. There where also some services, not showing up in the service list, only visible via msinfo32.

    None of the services was visible with regedit.

    The only way to stop them was to rename them with another server (visible) and restart the infected one. Then it was also possible to delete the files via the command line (looked like a complete operating system with storage in RECYCLER.BIN).

    We deleted the invisible services with another server (network registry). The name of the services : Atarpisrv (calls ATA122.exe)
    PowersupplyManager
    r_server

    Thank you for your interesting article. Best regards from Martin