So the war between the miscreants and the first responders / incident responders is just that - it's a war with casulaties (servers, workstations, work life / home life balance) and it is complete with an arms race in the form of stealthing (miscreants) and detection (incident responders) technologies. They hack and hide - we try to find them and recover the servers. My goal with this blog is to expose the techniques used by the miscreants to educate the incident responders of the world to keep things at parity.
Over the weekend I got a phone call from Kenny - an engineer on my team out west who had a hacking case he wanted me to review with him. He shared out the output of our incident response toolkit (called WOLF - Windows Online Forensics). He had already been reviewing the output and needed some advice. Kenny had done a great job of identifying a series of intrusions but this intrusion was un-like the hundreds of others we examine each year - with this one we stumbled on to something new and interesting.
The call came in the same way they always do - the customer noticed some strange behavior on the box and his Symantec AV software had mysteriously started flagging some files on his machine as a backdoor. Here are the relevant event log entries from Symantec (where XXX was the machine name which I have removed to protect the innocent):
1/15/2005 1:57:00 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan. Action: Clean failed : Quarantine failed : Access denied 1/15/2005 1:57:36 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan. Action: Clean failed : Quarantine failed : Access denied 1/15/2005 2:12:20 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan. Action: Clean failed : Quarantine failed : Access denied 1/15/2005 2:15:56 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan. Action: Clean failed : Quarantine failed : Access denied
Norton Antivirus had pulled down updated signatures and as soon as they went into effect it identified a 'Backdoor.Trojan' (not very helpful) and moreover - it could not clearn or quarantine the file (even less helpful - but hey, at least you know something's up!). So the customer called Microsoft and got routed to our team for some investigative work on a possible hacking case.
What our team does in cases like this where there is a suspected intrusion is we talk to customers about working with law enforcement and we encourage them to work with law enforcement on cases where they suspect customer data may have been stolen or tremendous financial loss can or will occur as the result of the intrusion etc. We also tell them if they're going to work with law enforcement they don't want us touching the box with any of our tools . . . the vast majority of our customers choose not to work with law enforcement and instead would rather work with us. Law enforcement agencies are notoriously picky about who they will work with on investigations and who they trust as it is all to easy for some hoo-ha (technical term) to come in with guns ablazin' and trample all over critical evidence that may be found on the server in places like the slack space or free space of the hard drive. I remember vividly over 3 years ago I was en-route to the airport to work on-site with a customer (after having worked with him unsuccessfully for a few hous on the phone) on a very important security case when I got a call from the president of the company I was about to visit telling me to turn around and go home - I wasn't welcome anymore. I was a bit upset since I had already purchased travel and invested time on the case. The reason the customer gave? They had involved the FBI and my response time was . . . ahem, a bit better than theirs. When he informed the FBI that I was to arrive onsite within 4 hours - they said "it's either him, or us - you decide who you want to work this case" . . . and that was the end of my involvement (and it was fortunate too as I later learned the FBI managed to aprehend the fellow responsible). I use this story all the time when educating customers on the reasons they may want to get law enforcement involved and why if they do so - I should NOT be involved.
If our customers decide to work with us we send them our IR toolkit known as WOLF which is more of a data collection agent. This runs on a live system and collects volatile and non-volatile data from the box and it places all of the data in a .CAB file for analysis back here at Microsoft. We then have a special viewer we've written in C# to make analysis and searching very fast and efficient. What used to take us 3-4 hours per server can now be done in 15 minutes or less usually.
What we find in the vas majority of cases is that customers who have been hacked once, have usually been hacked more than once and they never knew about previous intrusions. The latest guy to hack you is usually simply the one who got 'sloppy' and got caught and ruined it for everyone. This was to be the case with this particular machine.
During the investigation Kenny found a copy of Hacker Defender that had been dropped on the machine back in May of last year (the customers antivirus softawre had still not detected it as of the time of writing and we manually removed it). It had been configured to run as the following service:
Dynamic Storage Allocation - [running]AutomaticLocalSystemC:\WINNT\system32\msdsa.exeProvides multiple storage heaps in data transactions for dynamic allocation. If this service is disabled, simultaneous data transactions across the local system may fail. registry.txt====================[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"
Dynamic Storage Allocation - [running]AutomaticLocalSystemC:\WINNT\system32\msdsa.exeProvides multiple storage heaps in data transactions for dynamic allocation. If this service is disabled, simultaneous data transactions across the local system may fail.
registry.txt====================[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSDSA]"ImagePath"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\msdsa.exe"
The files had been dropped on the box:
Files containing instances of 'msdsa.exe'Number of Files Searched: 6Time to Search Files: 1 secondsdir_creation_time_C_drive.txt ====================Directory of c:\WINNT\system3205/16/2004 03:38a 36,352 msdsa.exedir_last_access_time_C_drive.txt====================Directory of c:\WINNT\system3201/15/2005 01:23p 36,352 msdsa.exedir_last_write_time_C_drive.txt====================Directory of c:\WINNT\system3205/14/2004 02:23p 36,352 msdsa.exe
Files containing instances of 'msdsa.exe'Number of Files Searched: 6Time to Search Files: 1 secondsdir_creation_time_C_drive.txt
====================Directory of c:\WINNT\system3205/16/2004 03:38a 36,352 msdsa.exedir_last_access_time_C_drive.txt====================Directory of c:\WINNT\system3201/15/2005 01:23p 36,352 msdsa.exedir_last_write_time_C_drive.txt====================Directory of c:\WINNT\system3205/14/2004 02:23p 36,352 msdsa.exe
Looking at other activity around that time, the files appear as if from out of nowhere in the wee hours of the morning and a new user profile directory was created (TSInternaluser, which is also a member of the administrators group - wonder what the password is?) as if it had been used for a local logon shortly thereafter:
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 183,296 msdfm.exe2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 38,912 admdll.dll2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe2004-05-16 03:37:00 | dir_last_access_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 36,352 msdsa.exe2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 4,912 msdsadrv.sys2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini2004-05-16 03:40:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings - <DIR> tsinte~1 tsinternaluser
The properties of this account lead us to believe that it was there prior to 5/16/2004 - indicating perhaps it was the administrator account that was used to install the malware (which may indicate it had a weak or easily guessed password):
TsInternalUser FullName TsInternalUser AccountType User Comment This user account is used by Terminal Services HomeDrive HomeDir Profile LogonScript Workstations PswdCanBeChanged Yes PswdLastSetTime 4/28/2004 12:14 AM PswdRequired Yes PswdExpires Yes PswdExpiresTime 4/28/2005 12:14 AM AcctDisabled No AcctLockedOut No AcctExpiresTime Never TrueLastLogonTime Never LastLogonServer XXXLogonHours All RasDialin No RasCallback None RasCallbackNumber
Given the circumstances under which this malware was installed our best guess based on the evidence we have is that this malware was copied to the machine using the default admin shares (C$ / admin$ etc.) using a local administrator account (TSInternaluser) with a weak password that is only set to expire once a year.
Now that's not what I came here to talk about - this is merely an example of a prior intrusion that had gone un-noticed by this particular customer for over 7 months. The latest intrusion was caught by Symantec and is the reason the customer called us up. Let's have a look at that. We have a starting point, which I like to call "a lead" - and that is the file named by Symantec - USERS.DLL in SYSTEM32.
Let's figure out the MAC times on this file:
Files containing instances of 'users.dll'Number of Files Searched: 6Time to Search Files: 0 secondsdir_creation_time_C_drive.txt====================Directory of c:\WINNT\system3212/20/2004 12:39p 54,272 users.dlldir_last_access_time_C_drive.txt====================Directory of c:\WINNT\system3201/15/2005 01:21p 54,272 users.dlldir_last_write_time_C_drive.txt====================Directory of c:\WINNT\system3212/20/2004 12:39p 54,272 users.dll
So it appears that this file was dropped on 12/20/2004 - what else happened on 12/20/2004 - let's have a look:
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif
XXX was the username of an administrator on the box and has of course been changed to protect the innnocent.
Notice we have some ".GIF" files and a couple DLL's that were dropped around the same time . . . which brings me to my next miscreant hiding technique. Some miscreants favor active stealthing using rootkits to hide from administrators. Others prefer to hide in plain site using obfuscation techniques . . . this is an example of the latter. Do you think those are .GIF's you're looking at? Do you think that's air your breathing?
Our first clue that these are not really .GIF files comes from a tool written by Matt, an engineer on our team that is designed to find binaries (executable files) that have been renamed to non-executable extensions. (More on this in a minute). Matt's tool - flags these as executables:
Copied from: findbin_C_drive.txt==========================POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64\bits[1].gifPOSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\kill[1].gifPOSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\list[1].gif
There are various techniques one can use to spot an executable . . . you can use a hex editor . . . you can use something like dumpbin.exe . . . or you can use strings from Sysinternals to pull strings out of the file and have a look. You can even use Notepad and look for 'MZ' as the first two characters you see when you open the file.Here are some strings from 'kill.gif':
Strings v2.1Copyright (C) 1999-2003 Mark RussinovichSystems Internals - www.sysinternals.com jjhjjj(null) ((((( H5/? ;T^h<U_i=V`j>Wak?Xbl@YcmAZdnB[eoC\fpD]gqBINRESKILLSVCVS_VERSION_INFOStringFileInfo040904b0CommentsCompanyNameSystems InternalsFileDescriptionpkillFileVersion1.03InternalNamepkillLegalCopyrightCopyright 1999LegalTrademarksOriginalFilenamepkill.exePrivateBuildProductNameSystems Internals pkillProductVersion1.03SpecialBuildVarFileInfoTranslationjjjjh1!This program cannot be run in DOS mode.r<[
Strings v2.1Copyright (C) 1999-2003 Mark RussinovichSystems Internals - www.sysinternals.com
jjhjjj(null) ((((( H5/? ;T^h<U_i=V`j>Wak?Xbl@YcmAZdnB[eoC\fpD]gqBINRESKILLSVCVS_VERSION_INFOStringFileInfo040904b0CommentsCompanyNameSystems InternalsFileDescriptionpkillFileVersion1.03InternalNamepkillLegalCopyrightCopyright 1999LegalTrademarksOriginalFilenamepkill.exePrivateBuildProductNameSystems Internals pkillProductVersion1.03SpecialBuildVarFileInfoTranslationjjjjh1!This program cannot be run in DOS mode.r<[
This is obviously not a .GIF. So why would a miscreant take 'pkill.exe' from Sysinternals and rename it to a .GIF? The answer may surprise you - did you know that in Windows you can actually run a .GIF? That's right - take any .EXE and rename it to .GIF . . . now open a command shell (cmd.exe) and navigate to the directory where you placed the renamed .EXE. Now type <filename>.gif and watch the EXE run. This trick won't work in Explorer for various reasons I won't get into here - but you can run any file, regardless of its extension from inside a command shell if its executable. We tend to find that the more advanced miscreants favor 'stealthing' using this technique rather than using rootkits (which they know can be easily detected using a variety of anti-rootkit tools).
On an interesting side note: I once was involved in a case where the miscreants had hacked a large number of servers . . . we had sent resources on-site and were able to develop a tool that could detect the rootkit which was a new, custom rootkit that I don't think anyone had previously seen or detected before. We refined our tool to the point where it worked amazingly well and we started pushing it down to machines via logon scripts and startup scripts. At some point the miscreant got wise to our activities and stopped using the rootkit to hide his backdoor - instead he renamed his backdoor DLL (being hidden by the rootkit previously) to a non-executable extension (like .GIF) and stopped copying the rootkit files to the box. Yes, I realize that we wouldn't have had that problem had the boxes been disconnected from the Internet during this investigation - but that was advice that was ignored by this customer due to financial costs and SLA's - as is usually the case.
Okay - so we know what the .GIF's are - but what about the DLL's. At first glance these seem to be legitimate Windows files based on the string version information compiled into the DLL's:
Strings v2.1Copyright (C) 1999-2003 Mark RussinovichSystems Internals - www.sysinternals.com G!@#$FILEVS_VERSION_INFOStringFileInfo040904b0CommentsCompanyNameMicrosoft CorporationFileDescriptionkernelFileVersion5.00.3700.6690InternalNamekernelLegalCopyrightCopyright (C) Microsoft Corp. 1981-1999LegalTrademarksOriginalFilenamekernel.dllPrivateBuildProductNameMicrosoft(R) Windows (R) 2000 Operating SystemProductVersion1, 0, 0, 1SpecialBuildVarFileInfoTranslation!This program cannot be run in DOS mode.1j?1~<1j?1RichOWrAPrcWorksPEC2.rsrc.reloc.rsrc8, ntdll.ERNEL32WTSGetActiveConsoleSessiIdNtQuerySystemInformatO:
G!@#$FILEVS_VERSION_INFOStringFileInfo040904b0CommentsCompanyNameMicrosoft CorporationFileDescriptionkernelFileVersion5.00.3700.6690InternalNamekernelLegalCopyrightCopyright (C) Microsoft Corp. 1981-1999LegalTrademarksOriginalFilenamekernel.dllPrivateBuildProductNameMicrosoft(R) Windows (R) 2000 Operating SystemProductVersion1, 0, 0, 1SpecialBuildVarFileInfoTranslation!This program cannot be run in DOS mode.1j?1~<1j?1RichOWrAPrcWorksPEC2.rsrc.reloc.rsrc8, ntdll.ERNEL32WTSGetActiveConsoleSessiIdNtQuerySystemInformatO:
But the "PEC2" string right above the .RSRC and .RELOC strings above indicate that this DLL has been compacted / compressed / packed using a utility called PECompact to make it smaller. Miscreants like to 'pack' their malware for many reasons:
There are some tools out there to assist incident responders in determining whether a file was packed and if so what it was packed with - one such tool is PeID that I have used with much success. WARNING: Studying malware with PeID is extremely dangerous - some of the plug-ins for PeID will actually load the malware to attempt to un-pack it - do not use this tool or others like it on a machine of any importance unless you like living dangerously.
At this point - there are not many interesting strings in the DLL since it has been compressed (it will decompress at runtime as it loads into memory) - so the DLL has been submitted to the antivirus vendors via our relationship with them (the Virus Information Alliance) and they will be pushing out signature updates as needed. Even though the DLL was packed - there is enough information in the strings output to lead me to believe this is a new version of YYT_HAC's backdoor trojan (for more information on YYT_HAC, if you can read Chinese visit: www.yythac.com). Here is the relevant string that tipped me off:@HKDOOR>
I recognize that as the default 'prompt' you get when connecting up to YYT_HAC's backdoor. This is not the first time I've come across malware from this author being used in the wild.
Now - up until this point we haven't really covered anything really 'new' . . . renaming executables to non-executable extensions, packing binaries, hacking boxes - it's all been done before (woohoohooo!) . . . what I'm about to show you is a development that is fairly new to Windows and signifies perhaps a new technique that is being used by the miscreants to hide from administrators and maintain control of their boxes.
But first some more background on this investigation. So we've identified some suspicious files on the disk - but we don't know how or where they are being used on the system and for what purpose (assume we haven't collected the files or run strings on them yet). Let's find out how these DLL's are being used (i.e. how they get loaded).
checksym.csv====================,WINLOGON.EXE,228,C:\WINNT\SYSTEM32\USERS.DLL,SYMBOLS_NO,0,0,"December 31, 1969 18:00:00",0,,,,,,,,,,,,,
According to this output - this DLL is loaded inside of one process and only one process: WINLOGON.EXE. There are a few reasons a miscreant would want to load a DLL inside of WINLOGON.EXE.
The down-side to running your code inside of WINLOGON is that it has to be really clean code. Any unhandled exceptions in WINLOGON.EXE will cause the process to crash which will cause the kernel to shutdown the machine with the STOP code mentioned above.
So there are numerous reasons why a miscreant would want to run code inside of WINLOGON (with some associated risk) and to date there are only a couple of techniques for getting that code there:
Option one listed above simply requires adding a registry entry for your DLL to a specific well known registry location discussed in my KB article and is easy to detect.Option two listed above requires some OTHER malware to auto-start somehow and then inject code into WINLOGON and is harder to detect (you have to find the other malware responsible for injecting the DLL into winlogon and what if that other malware is hidden by a rootkit?)
But what if the criteria for option 1 and option 2 above were not met? What if the GINADLL registry value was empty? What if there was no other malware running on the machine that could inject your backdoor DLL into WINLOGON?
This is the situation we were faced with this weekend.
I'm going to take a Quentin Tarantino like tangent here and we're going to jump to a completely different scene before we revisit this thought and conclude this blog post.
There were 2 DLL's we found that were dropped on this customers machine. There was USERS.DLL and there was another called USERBIN.DLL.Let's see if we can find them in the registry.
Here are the search results for USERS.DLL against this customer's registry:
Files containing instances of 'users.dll'Number of Files Searched: 1Time to Search Files: 0 seconds
Here are the search results for USERBIN.DLL against this customer's registry:
Files containing instances of 'userbin.dll'Number of Files Searched: 1Time to Search Files: 0 secondsregistry.txt====================[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\BITS\Parameters]"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\BITS\Parameters]"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BITS\Parameters]"ServiceDll"=(REG_EXPAND_SZ)"C:\\WINNT\\system32\\userbin.dll"
What's this? This is a variation of a hiding technique I've discussed in a previous blog post but with a twist. As it turns out - on Windows 2000 and later operating systems you can implement a service that runs as either an EXE or as a DLL. If you choose to implement a service as a DLL it will run in what's called a 'service host' which are those 'svchost.exe' processes you see running on all Windows boxes. This particular miscreant has written a service dll and has hijacked the DLL used by the BITS service and replaced it with his own entry. If you were to look in the Services applet in control panel / administrative tools - the BITS service would appear normal (that is it's still pointing to: %SystemRoot%\system32\svchost.exe -k netsvcs). However, below the BITS service registry entry is a 'Parameters' key and in there is a ServiceDLL value and this registry value is SUPPOSED to point to QMGR.DLL - not userbin.dll.
So while this was rather clever - it was easily detected by our analyst Kenny and he restored the registry value to its proper setting. In addition to restoring this registry value to its proper setting, he also renamed USERS.DLL to a different file name since we had yet to determine how this DLL was getting loaded inside of WINLOGON.EXE. He then told the customer to restart and when he did the customers box went into a 'reboot' loop at the place where you would normally see the CTRL-ALT-DELETE screen and the customer could only boot to the recovery console. At the recovery console Kenny had the customer restore 'USERS.DLL' in the SYSTEM32 directory and reboot - and to his surprise the machine booted successfully.
And now we're back to the point where we previously diverged (I hope I haven't lost anyone). There were no registry entries for USERS.DLL (assuming no rootkit was hiding their presence) - one might assume then that perhaps USERBIN.DLL, which was starting via the hijacked BITS service registry entry was injecting USERS.DLL into WINLOGON when it started . . . but this is not the case as simply restoring USERS.DLL (and leaving the BITS service fixed and pointing to the proper DLL) was enough to get the box to boot successfully.
At this point - Kenny and I theorized that somehow WINLOGON had become 'dependant' on this DLL to function and that if this DLL was removed, WINLOGON would crash with a C21a and the box would get stuck in an infinite reboot -> C21a cycle.
So how does one make WINLOGON dependant on some 3rd party backdoor DLL? We were stumped . . . until Kenny noticed something intresting in the WOLF output from MBSA.
Warning MS04-011 835732File version is greater than expected.[C:\WINNT\system32\winlogon.exe, 5.0.2195.6970 > 5.0.2195.6898]Note MS04-016 839643Please refer to 306460 for a detailed explanation.Patch NOT Found MS04-031 841533File has an invalid checksum. [C:\WINNT\system32\winlogon.exe]
This MBSA output is confusing because it checks the version of WINLOGON.EXE against two different patches (normally MBSA would only complain about the most recent missing security update). MBSA claims that the MS04-011 patch has a version is greater than what it was expecting. This can be considered normal as sometimes there are some non-security related hotfixes that will redistribute files (like WINLOGON.EXE) that customers can apply to fix non-security related issues. These 'hotfixes' can actually contain files that supersede, or are newer than, publicly released security updates (which are broadly distributed as opposed to hotfixes which are not). In that sense - this MBSA output is 'normal' or can be explained in the presence of a non-security hotfix - but then we get to the MS04-031 check and MBSA complains of an invalid checksum? On a completely differnet update?
First off - that IS the version of WINLOGON.EXE that shipped with MS04-031, but MBSA is saying the CRC32 based checksum is invalid. There is a limitation in the current version of MBSA where it presently doesn't support multiple checksums for multiple versions of the same file contained in a security update. This poses a problem on multi-proc machines and security updates where there are two versions of the same file in the security update (one for uniproc machines and one for multiproc machines). The MSSECURE.XML for now always contains the checksum for the uniproc binaries and it will generate the invalid checksum message on multiproc machines that have the multiproc binaries installed. This would be readily explainable IF this machine were multi-proc . . . but it is not.
I think you see where I'm going with this - it appears that WINLOGON.EXE itself was modified or replaced. It is no longer the legitimate copy of WINLOGON.EXE that came with MS04-031 on this customers machine. This would certainly explain the hard-coded dependancy on USERS.DLL and would explain why WINLOGON.EXE would crash when we deleted / renamed USERS.DLL.
As a final proof point we collected MD5/SHA-1 digests of the suspect WINLOGON.EXE and verified that it no longer matches up with any of the ones we ship!
For those who are curious - here are the MD5/SHA-1 digests of the suspect WINLOGON.EXE
MD5: 3A0F4A1C9FA13EE9D5BD88CA76A20758SHA-1: 7B8F74B78959E266B91C70283292AB39367B2AC0
Now - some of the more clueful readers may wonder why SFP (System File Protection) did not replace WINLOGON.EXE with the original version or complain loudly about this imposter? This is a reasonable question - for years I have watched in amusement as the miscreants try to overwrite system file only to have SFP put them back immediately. I was saddened when @Stake released a tool that would temporarily disable SFP (until the next reboot) allowing miscreants to finally overwrite SFP protected files (the imposters would still be detected at the next SFC.EXE scan). I was informed by our product group that SFP is a *reliability* feature and not a *security* feature and thus it was never intended to be 'hack proof' - the fact that it had been subverted was not a surprise to any of our security developers who said 'it was only a matter of time'. I've always wondered why on Windows we don't see more trojaning of system binaries and I've always had two theories on why this almost never happens.
It is worth pointing out, however, that both of these barriers have been overcome. As you may or may not know parts of the Windows 2000 operating system were stolen last year from a partner who had legal access to our source code. In addition it has been known for over a year how to disable System File Protection during the current boot - allowing anyone with admin rights to replace SFP protected system binaries.
As a final thought - the process responsible for implementing System File Protection is none other than our very own WINLOGON.EXE a process on this customers machine which I no longer trust.
UPDATE: The antivirus vendors have analyzed these two DLL's (USERS.DLL and USERBIN.DLL) and have come up with the following names for them:USERS.DLL - Backdoor:Win32/Hackdoor.BUSERBIN.DLL - Backdoor:Win32/Bits.B
Win32/Hackdoor.B is an interesting backdoor. When you run it you can specify a process you want it to "attach" to. It will then modify that processes imports table by modifying the binary itself (thus confirming our theory that WINLOGON.EXE had been modified) - thus ensuring that when the process starts the DLL is loaded. It does this rather well and manages to keep the file size the same making it necessary to generate message digests of the files to tell if they were modified. When the DLL loads it also loads a driver which can intercept all incoming network traffic so that the backdoor can hijack an existing port (it doesn't need to bind to a new port - it can use any existing port). Hacker Defender 1.0 has similar capabilities.