Stuff n Things : Incident Response

Cheap real time monitoring for Conficker clients

kfalde — Mon, 09 Mar 2009 18:21:27 GMT

I already did one post about using eventcomb/logparser to look for clients but found a better way to do it on a case last night which I wanted to share. The first thing you need is to enable netlogon debug logging on all of your DC’s save the following as a .reg file and import it on all your DC’s that you want to monitor:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters]
"DBFlag"=dword:2080ffff

This will cause netlogon to start logging extended entries to %windir%\debug\netlogon.log. You may take a slight perf hit on your DC’s for this just as an FYI. You will not fill up your drives as it logs to 2 20Mb files.. netlogon.log and netlogon.bak and uses circular logging where it starts overwriting once it fills up.

What we are looking for is the value “0xC000006A” on any line which indicates “The value provided as the current password is not correct”. So we already have articles on using NLParse to look for these as well as you could use logparser etc but these are all non-realtime right, you copy all the files somewhere you parse them you look at the output etc.. wouldn’t it be nicer if you could just have a console where you just glance and see when you have a malware client acting up :).. The key to this is using a TAIL utility. In my case I used Tail for Win32 which you can find here http://tailforwin32.sourceforge.net/.

On the system you plan on monitoring from install this piece of software. Strangely you have to open a log file before you can start modifying the settings for the client so open your first netlogon.log file. The client supports network paths so you will be opening \\DCNAME\C$\Windows\Debug\Netlogon.log for each of your DC’s (yes this would stink for a large environment but hopefully if you’re that large you spent the money on a product to do this for you anyway… right??).

Once you have your netlogon files open go to Settings>Keywords and add 0xC000006A as a keyword. Also you will want to set “Show only Hot Lines” to enabled (this drops all the other netlogon junk which we don’t want to see) and set “Wrap Lines” to enabled. Basically what you end up with in the end is multiple windows open that are only looking for entries of bad password attempts and pulling those in close to real time from the netlogon logs. This allows you to easily see when a client is hammering away on user accounts and allows you to go shutdown and clean that system. Another tip is you can use the Window>Cascade or Tile option to automatically arrange the netlogon file windows to easily see when a DC is seeing a problem client.

Updated had the wrong value in the for the reg key :)

Update 2 3/11/09: This has a slight catch here apparently in that this is only for NTLM bad password attempts and you still need to use eventcomb or something else to see Kerberos bad password tried :( http://technet.microsoft.com/en-us/library/cc776964.aspx

“If the Netlogon logs from all domain controllers from the time of lockout but do not display data that pertains to any of the locked-out user accounts that you are analyzing, then NTLM authentication is not involved in the lockouts. This normally indicates that the authentication issues are between computers running Windows 2000 or later, because earlier versions of Windows used NTLM authentication exclusively. You should focus on Kerberos authentication troubleshooting by using Kerberos logging and examining the Security event logs.”

Update 3 4/9/09 http://baremetalsoft.com/baretail/ is also good actually seems better then tailforwin32, free windows gui tail utility to use with this solution.

Blocking and finding Conficker and Downadup systems

kfalde — Tue, 10 Feb 2009 03:47:00 GMT

EDIT 4/27/09: THIS NO LONGER WORKS WITH NEW VARIANTS OF CONFICKER HOWEVER THE CONCEPT IS STILL SOUND IF YOU ARE LOOKING FOR SYSTEMS THAT ARE QUERYING FOR SPECIFIC DNS NAMES.

I’ve already created one post on finding malware systems using eventcomb however when it comes to Conficker or Downadup and realistically other malware too all that method can really show you are clients that have decided to finally put a Domain Controller on their hitlist. One of the things we know about Conficker is that it utilizes a method to based on date to generate a url that it will connect to in order to report in to its evil overlords as well as to possibly see if there is some new update/commands available to it. The guys over at F-secure have put up a list of domains that the malware would possibly use for the month of February at http://www.f-secure.com/weblog/archives/00001593.html . Now if you have a proxy server or some type of firewall server that can do blocking or redirecting based on host names this is great. Not all customers have this though so we figured we would try to put something together that would work for most of our normal Windows customers :).

Basically what I did was utilize the dnscmd command to work with MS DNS to create new zones as well as wildcard records for each domain that may be possibly used. The download for the .cmd files you can get here:

If you know anything about MS DNS you realize this is going to look pretty darn messy :) we are talking about 7k new zones in your DNS server so I would either recommend putting this on a new box and forwarding all your DNS to this before going to the internet or not opening your DNS gui to look at zones that often after you add these :). The first script blocklistfebzoneadd.cmd will create all the zones. If you just want to block connections for these zones for February you can stop there… But no that’s wussing out we want to ERADICATE this thing!!! The next script is blocklistfebrecordadd.cmd you will want to edit this doing a find/replace and change 192.168.1.100 for a new IP address for a new IIS server in your environment. The beauty of this is that for this month everytime you have a Conficker / Downadup client try to connect to their control server they will instead connect to your new IIS setup. You just need to keep checking your IIS logs for that website you setup and cleaning those client IP addresses up.

Once the month of February is over you can use the 3rd script blocklistfebzonedelete.cmd which will go through and reverse the effects of this by deleting all of the zones we created. Hopefully this should be pretty simple but if you have any questions just let me know.

Happy malware hunting.

Using Logparser + Eventcomb to find malware

kfalde — Thu, 29 Jan 2009 01:04:00 GMT

During the course of these Conficker / Downadup issues we typically see cases that started because accounts are getting locked out. I pause briefly here to point out that account lockouts are the work of the devil and are a sorry excuse for most people to not use a complex password policy. So it seems that these cases for the most part are slowing down however customers are still looking for those few machines that fall between the cracks that are still causing account lockouts when they turn back on their account lockout policy (once again because they don’t want to use complex passwords).

So the tool to turn towards is the account lockout tools. Part of this toolset is eventcombmt (mt stands for multithreaded).

Eventcombmt is cool for all sorts of things but the only thing we are interested in is the built in Account Lockout Search. Select Searches>Built In Searches>Account Lockouts. You could also change your output directory if you want, the default is C:\temp.

Once you selected this it should put in the right event id’s and locate and select all your DC’s. Click Search and it’s off searching through all your DC’s Security Event Logs and dumping all the pertinent info to DomainControllername.txt files in the c:\temp directory.

While your waiting for this go download log parser. Install that you may want to make sure it’s in your path afterwards for this to work try typing logparser from a cmd prompt. Once you have logparser in place and Eventcomb has finished and output all of your DC’s .txt files then you will want to run the following logparser query in the directory with the .txt’s

Once that’s complete run the following in the directory with all your textfile outputs from the DC’s

logparser -i:textline "SELECT SUBSTR(SUBSTR(Text, ADD(INDEX_OF(Text, 'Address: '), 8)), 1, INDEX_OF(SUBSTR(Text, ADD(INDEX_OF(Text, 'Address: '), 9)),' ')) AS IPAddr INTO addrs.csv FROM *.txt"

Revision 5/8/09 You may need to use this instead of the Addresses are all at the end of the line:

logparser -i:textline "SELECT SUBSTR(Text, LAST_INDEX_OF(Text, 'Address: ')) AS IPAddr INTO addrs.csv FROM *.txt"

This should create a file called addrs.csv which has all of the IP addresses that has caused by password attempts. There will be lines for each attempt so we need to parse this down a little more to give use a column with IP address and a column with the number of bad password attempts so we will run the following logparser query next:

logparser -i:csv -o:csv "select IPaddr, count (*) as hits into final.csv from addrs.csv group by IPaddr order by hits desc"

This should leave us with a final.csv file which has 2 columns one for the IP address and another for the amount of times we have seen that IP address causing bad password attempts in our security event logs across our DC’s. In the case below that first IP had caused 85k bad password attempts guessing that machine has a problem :) …

Seeing we’ve primarily used this for Conficker it seems the following line works well to output just a list of IP addresses with no header on the file and no hits.

logparser -i:csv -o:csv -headers:OFF "select distinct IPaddr into ips.txt from addrs.csv"

You can then take the ips.txt file and use the import function on Mcafee’s Conficker Detection Tool to scan all of these IP’s quickly to see if they are infected with Conficker or not.

Update 1: 4/9/2009 Totally reworked this thanks to Neil Carpenter and some better logparser logic to filter this better and have a much cleaner final output. Also dropped need for .csv files as we are filtering using textline input instead.

Update 2: 7/10/2009 Added new logpaser query to output just IP’s to file with no hits etc which you can then easily import to Mcafee’s Conficker Detection Tool

Malware Win32/Conficker.B W32.Downadup.B

kfalde — Thu, 08 Jan 2009 08:37:00 GMT

So for the past 2 weeks now we are absolutely getting hammered with calls in CSS Security here at MS with organizations contracting this piece of malware.

You can find write-ups from various AV companies at the following URL’s

http://www.ca.com/us/securityadvisor/virusinfo/virus.aspx?id=76852

http://www.microsoft.com/security/portal/Entry.aspx?Name=Worm:Win32/Conficker.B

http://www.symantec.com/norton/security_response/writeup.jsp?docid=2008-123015-3826-99

http://www.symantec.com/norton/security_response/writeup.jsp?docid=2009-010717-4209-99

So the write-up’s are all pretty good some have details that the others don’t etc. We see a range of cases it seems from customers missing the patch to those who are completely patched but are still seeing this piece of malware in their environment causing issues. The interesting thing about this piece of malware is that it is really singling out organizations that have not done a good job with their security policies/procedures. The MMPC group made a post about this piece of malware http://blogs.technet.com/mmpc/archive/2008/12/31/just-in-time-for-new-years.aspx where they linked http://technet.microsoft.com/en-us/library/cc512606.aspx Jesper’s password paper from 2005!!! This guidance and similar guidance has been out for 3+ years now and we still have customers that aren’t following it. I sometimes wonder if we should just push out a “critical update” that applies to all DC’s and updates their Default Domain and Default Domain Controller Policies for them to something more acceptable :) of course we would probable pull a lot of flack for that :) . /Rant Off

So things you should look at doing if you are hit with this:

Disable Account Lockouts: You are already jacked why are you making it worse by leaving the account lockout policy in place?
If you are not patched (especially with MS08-067) do so immediately.
Find a machine that you know is infected and see if your AV will clean it up with the latest definitions/client. If it is not cleaning it then open a case with your AV vendor as well they are going to be the ones to update definitions to properly detect/remove the malware in the environment (believe me you want this instead of manually running around cleaning off systems)
Enable Password Complexity: Like the Account Lockouts this is in your Default Domain Policy if you don’t have it enabled odds are you have 10+% of your population using one of those weak passwords on the list from those write-ups on the malware, and if you have users with those passwords you are still going to have issues with malware spreading. Oh maybe you should get someone working on that org-wide email explaining to your users the new password policy like X characters and how they need 3/4 special characters/Upper/lower/numbers. You probably also want to look into a script/tools to expire accounts (selectively so you don’t whack things like service accounts you aren’t ready to change) Check out Joeware’s oldcmp and expire utilities at http://www.joeware.net/freetools/ you can dump selectively based on OU targeting to get lists of users’ password age and then pass the lists to the expire utility to force password changes across groups of users. Or if you’re a masochist you can just expire them all and deal with the consequences.
Password Complexity on local accounts: Is the password on your local Administrator accounts something on that list from the writeups? If so you better get it changed.
Share Permissions: This one is more complex to explain. Basically for any network shares that you know multiple users map drives to you need to have the permissions locked down in this fashion.. root of the share Remove Write/Modify access to Everyone.. Allow them full control to the contents of subfolders in the shares. The way the malware works is if you have say a N: drive mapped to \\FILE01\Data it will basically drop malware.exe in N: and an autorun.inf in the same N: pointing to malware.exe. The next user that is mapped to the same N: drive double clicks on drive icon and runs malware.exe (ok yes this can be mitigated by autorun settings but do you know those are set on your clients maybe a good idea for a GPO setting those as well :) )
Stop logging into infected machines with Domain Admins: One characteristic of the malware is that it can use impersonation and can be in the Run key so that it runs under the logged on user’s context. So when you log in on that infected system with your DA account guess what.. you just helped spread without it needing to force passwords use a vulnerability etc because hey its all allowed under your privileges.

I may add more to this possibly as it’s getting late my time and I’m sure it’s going to be another long day tomorrow. Hopefully this helps someone.

Changes to Microsoft Anti-Malware

kfalde — Wed, 19 Nov 2008 20:14:00 GMT

This doesn’t really affect the FCS world but it is an interesting development. https://www.microsoft.com/presspass/features/2008/Nov08/11-18AmyBarzdukasQandA.mspx apparently we are going to begin to offer a no-cost anti-malware solution in 2nd half of 2008. This is going to be more targeted at the end-user market it appears from the look of it. Check out the article for an interesting read.