AD Troubleshooting

AD and Domain-related issues and troubleshooting methods for Active Directory.

Troubleshooting account lockout the PSS way

Troubleshooting account lockout the PSS way

  • Comments 14
  • Likes

I‘ve been thinking for some time about pulling together the typical approaches we use when troubleshooting account lockout issues.

So... here is the CSS/PSS approach to troubleshooting Account Lockouts.


#1 - Look at the Account Lockout Threshold policy that is defined for the Domain.

Applications commonly do several retries of logons if the first logon fails (in case the first or second failures were because of transient network conditions - which is not unreasonable behavior).  This means that a user that enters an incorrect password ONCE can be locked out if the lockout threshold is low – depending on how the application behaves (how many retries it attempts). 
Anything below 10 is just asking for helpdesk calls.

I.e. is this simply a result of a too narrow lockout threshold policy or is this a real issue?


#2 Determine how many users are being affected

The easiest way to determine the extent of the problem is to
enable Netlogon debug logging on the PDC emulator DC (All other DC‘s will contact it for confirmation in case of an incorrect password).
Once it is enabled, allow it enough time (30-60 minutes or longer) to gather information from users attempting to log on and then copy the netlogon.log and netlogon.bak files to a location where you can examine them closer.

I.e. is this an outbreak of account lockouts that affects all users or is it just affecting a handful of specific users or a single user? What is the common factor?

I usually run the following against the Netlogon file to filter out the lockouts:
type netlogon.log |find /i  "0xC000006A“ >badpassword.txt
 type netlogon.log |find /i  "0xC0000234“ >lockedout.txt

This should give you a list of users which are supplying bad passwords and users who have been locked out.

Note: you can't do this against a live Netlogon.log file because of the running Netlogon service - simplest solution is to copy it elsewhere and run the commands against the copy.


#3 Look at which machines the bad password attempts are coming from

If you see logon attempts for different users coming from the same machine it can mean that it‘s a multi-user machine (like a Terminal Server).  It can however also mean that it has some suspect application running on it – in either case you should investigate closer.


#4 Drill down to which applications are sending the bad passwords

A network trace from the client or just examining which applications and service are running on it and stopping each in turn to isolate the issue will usually be enough.

TCPView from Sysinternals or Netstat are also good for this kind of investigation, matching the process ID of a service or application that creates a socket connection with a bad password attempt in the Netlogon log of a DC.

[Footnote: Windows 8 and Windows Server 2012 log the process ID of the application logging the lockout in the Netlogon log - which makes it much easier to troubleshoot this type of problme on W8/W2k12]

Common contributors can be OS components like Credman with stale passwords, services running under a specific domain account, dumb applications with insufficient retry logic, etc. 

The Conficker virus was also notorious for attempting brute force password attacks against members of the built-in Administrators group in the Domain.

 

Note also that if you have a mixed environment you may get Account Lockout issues when you change passwords on one OS (client-side or DC-side) and then move to another legacy client that doesn't understand the protocol or algorithm used.

  

Summary:
- If you enable a lockout policy on the domain – you will get accounts locked out at some point. 

- The lower the lockout attempt threshold – the more frequent the lockouts obviously.

-  Using an account lockout policy is exposing yourself to a potential Denial of Service attack – there‘s nothing that prevents one spiteful user from attempting to log on as another user and entering an incorrect password until they‘ve locked out the account.

The potential value of using a lockout policy in your domain is to prevent brute-force password attacks.  At the same time it exposes you to Denial-of-Service attacks if you enable it.

 When I started at Microsoft some years ago I came from a mixed support environment, during that time I had learned to love two extremly powerful string parsing utilities;  GAWK and SED.  

I wrote a simple batch file that uses those two commands to parse the Netlogon file into a more visually pleasing format, here is a sample output from it:

 

 Summary of Netlogon log file Netlogon.bak
Netlogon log file starts at 07/14 17:07:40
....................ends at 07/15 09:09:19
-----------------------------------------------
Total successful logons of users: 30271
Total locked out logons of users: 475
Total logons of users with bad password: 16596
Total logons of users with unknown username: 2
Total logons from undefined subnets: 2
Total Access Denied errors: 0
Total trust password errors: 0
Total KDC PAC verifications : 1137

 ......

 #of attempted logons with bad passwords from each machine
----------------------------------------------------------------------------------
  11430                 DOMAIN\USER9424 from WORKSTATION-43928 (via EXCHANGE09)
   2647                  DOMAIN\USER9323 from WORKSTATION-44367 (via EXCHANGE14)
    756                   DOMAIN\USER5254 from WORKSTATION-44791 (via EXCHANGE09)
    380                   DOMAIN\SVCACCT01 from EXCHANGE09 (via EXCHANGE09)
    380                   DOMAIN\SVCACCT02 from EXCHANGE01 (via EXCHANGE01)
 
 
 

In this log:

 -the top 3 users had a bad application running on their workstation which attempted to log on to their Exchange mailbox every 5 seconds if an incorrect password had been entered.  
-  the passwords had been changed twice for service accounts running for an antivirus agent on EXCHANGE01 and EXCHANGE09, but the password hadn‘t been updated to reflect that on the servers.

 

Account Lockout and Management Tools
http://www.microsoft.com/downloads/details.aspx?familyid=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en
  

Account Lockout Best Practices White Paper
http://www.microsoft.com/downloads/details.aspx?FamilyID=8C8E0D90-A13B-4977-A4FC-3E2B67E3748E&displaylang=en&displaylang=en

 

Troubleshooting Account Lockout
http://technet.microsoft.com/en-us/library/cc773155(WS.10).aspx

Account Lockout Tools
http://technet.microsoft.com/en-us/library/cc738772(WS.10).aspx

 

Comments
  • Thanks for the wonderful explanation and the supporting batch file. I'm trying it out here. I got "gawk" and "sed" from the link in script. I couln't find "head". I've "tail" from Windows Resource Kit. Could you please provide the link to download head. Thanks Again.

  • There are several projects available which have win32 ports of head - for example http://sourceforge.net/projects/unxutils/

  • Hi! Thanks for your blog.

    Do i need to modify Process.cmd? I have installed everything as mentioned including Head.exe

    But when i run the command i get following error, Please suggest

    C:\Documents and Settings\960138\Desktop>process.cmd "C:\WINDOWS\Debug\netlogon.

    log"

    This batch file requires the unix tools GAWK and SED to be either in the path

    or in the directory where it is run.

    These tools can be downloaded from f.x. http://gnuwin32.sourceforge.net/packages

    .html

    To use it, specify the Netlogon file that you want to process as the first argum

    ent

    -----------------------------------------------------------------------------

    Press any key to continue . . .

    The filename, directory name, or volume label syntax is incorrect.

    The filename, directory name, or volume label syntax is incorrect.

  • You need to specify the name of the netlogon.log or netlogon.bak file that you're trying to analyze.

    From the error you're getting it looks like you're running the batch file while specifying the running netlogon.log file which is going to be locked.

    The batch file cannot be run on an active netlogon.log file, you'll need to copy the netlogon.log file and run the batch file against the copy.

    Note that the format of the Netlogon logs tends to change over time as the developers add or remove entries to the log....this means that some slight tweaking of the batch file may be necessary to match this to the machine you're analyzing netlogon logs from.

  • 2008 Domain Controller (64-bit) SP2:

    1) Enabled netlogon debugging with "nltest /dbflag:0x2080ffff"

    2) Manually search the netlogon.log for "0xC0000234" and I get hits, but when I try to run "netlogon.log |find /i  "0xC0000234“ >lockedout.txt" I get the following error:

    "the process cannot access the file because it is in use by another process"

    3) Tried disabling symantec 11.5 AV but no change.

    Ideas to get past the error?

    thanks,  jw

  • Hi, Excellent BLOG!

    Can you explain what you mean with marginal and why in:

    "The difference in the protection against brute force attacks that a lockout policy of 50 offers when compared with a lockout policy of 5 is marginal. "?

  • A brute password attempt involves millions of attempted logons - the difference between a lockout threshold of 5 and 50 is marginal in that context.

    I.e. you're not going to brute force a password with 45 extra attempts unless it's an unusually bad password in the first place.

  • I also get the following error when running this command:

    type netlogon.log |find /i  "0xC000006A“ > badpassword.txt

    type netlogon.log |find /i  "0xC0000234“ > lockedout.txt

    How can I fix that:

    FIND: Parameter format not correct

    The process tried to write to a nonexistent pipe.

    Regards,

    Guido

  • Try typing the commands in manually, it looks like you have copy/pasted the command from jwheelock's comment into a command prompt, this sometimes plays havoc with the quotation mark (") at the receiving end.

    I.e. if you look at "0xC000006A“ in the syntax you're using you'll see that the first quotation mark is different from the second - the first one is the correct format and the second one should be identical.

    ...and also make sure you copy the netlogon.log file to a different folder, you cannot run this against a live Netlogon.log file.

  • Ingolfur, re "you're not going to brute force a password with 45 extra attempts". How about a large AD deployment with hundreds of domain controllers and a more sophisticated attack that tries to hit each domain controller (say with a targeted LDAP simple bind attempt) with 49 password attempts within the account lockout observation period. With 500 DCs could you approach 24,500 logon attempts then without triggering an account lockout condition on any one of those DCs? I've not found a documented provision in native AD (or ADDS as it's now called) to pool logon failures across DCs for calculating the threshold.

  • Each of the bad password attempts from multiple DC's will by default also hit your PDC that will increment the badpwdcount attribute in it's own NTDS database locally and lock out the user after 50 failed attempts (even if only 1 comes from each DC.

    Granted, with a sophisticated attack as you describe there are additional factors, however if you look at the recommendations on technet.microsoft.com/.../cc875814.aspx and assume the default password combination of a minimum of 8 characters, capitals+uppercase+numbers+symbols=72 different combinations for each then you have 722204136308736 possible different password combinations.

    In that context - even 24500 wouldn't be very likely (1/29477719849 chance) to brute force a password unless it is unusually bad to begin with.

    Steps like increasing the minimum password length, custom password filters on DC's, moving to requiring smartcards and/or monitoring the PDC for lockouts would be less intrusive than using a low account lockout threshold setting.

  • Hi Ingolfur Arnar Stangeland

                                                 Excelent Script thanks a lot...

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment