I worked the following case recently:
We have a single user that keeps getting his account locked out every 60 seconds. We've managed to isolate this down to coming from the Exchange server but there isn't anything pointing in the right direction as to what is causing it.The really strange bit is that if we shut down the users’ workstation the lockouts continue regardless. And no, we’re not using terminal services where a disconnected session could be doing this.
The event logged in the Security log of the authenticating DC was as follows:
Event ID: 529Source: SecurityType: Failure AuditLogon Failure:Reason: Unknown user name or bad passwordUser Name: User1Domain: Logon Type: 3Logon Process: Advapi Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0Workstation Name: EXCHSRV01Caller User Name: EXCHSRV01$Caller Domain: CONTOSOCaller Logon ID: (0x0,0x3E7)Caller Process ID: 1728
The standard approach to Account Lockout troubleshooting of enabling Netlogon Debug Logging unfortunately didn’t help much in this case; the only thing we saw in the resulting Netlogon.log file was a confirmation that the account lockouts were in fact being initiated by the Exchange server.
Looking at the Caller Process ID from the event was however more interesting, process #1728 on the Exchange server was the Inetinfo process. Inetinfo runs the web services that Exchange uses, including Outlook Web Access, SMTP Virtual Server, NNTP, IMAP4 and POP3 services.
Fair enough… that means we’re most likely either looking at a failed Outlook Web Access attempt that’s been cached by a browser or a program that’s resending it every 60 seconds *or* maybe we’re looking at an attempt to download POP3 mail or send authenticated SMTP mail (or even NNTP access).
Looking at the configuration of the Exchange server, we could dismiss the POP3/IMAP4/NNTP bits straight away; they simply weren’t enabled on the server.
The OWA logs in IIS didn’t show the username of the affected user at all, so this wasn’t being caused by an attempt to access the mailbox using Outlook Web Access.
That left the SMTP bit; we looked at the IIS logs for the SMTP Virtual Server but we didn’t see any reference to the user there either. After taking a network trace and analyzing it however we discovered why; the SMTP authentication simply doesn’t log the username into the IIS logs but we did see the IP address of the incoming clients and were able to use that to match the account lockouts in the Netlogon.log to determine where the lockouts were coming from.
Here’s what was happening:
- The affected user was the administrator of a web application
- That web application was able to send SMTP notifications
- To send the notifications, the web app needed credentials
- The web app admin had entered his username/password and the web app was using it when attempting to send authenticated SMTP mail
- After 2 password change cycles had occurred, the lockouts started
To resolve this; the web app admin removed his username/password from the web app configuration and replaced it with the service account that he had been assigned to use for it.
The morale of the story is that the amount of log data available to troubleshoot Account Lockouts resulting from SMTP authentication is very limited. When only one user is affected it’s also always good to look at the affected user and investigate whether they are doing anything special that could be triggering the problem (as in this case).
Log Formats (IIS 6.0)http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/083c66a1-4860-4b53-9eca-9736cee15138.mspx?mfr=true.
NCSA Common Log File Format (IIS 6.0)http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a3ca6f3a-7fc3-4514-9b61-f586d41bd483.mspx?mfr=true.
Setting IP Access Restrictions to Servers (IIS 6.0)http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/db81ac6a-4014-4189-91a2-12632d21aa64.mspx?mfr=true).
"..After taking a network trace and analyzing it however we discovered why;.."
It would have been nice if you could write a little (or much :) how you setup the capture and filtering, and what tools you are using (netmon, wireshark etc.). And how you debug the capture since it capture some amount of data.
Sounds like a good idea, I'll try and put something together on that topic.
did you get the chance to put some data together on this ?
Time is a luxury commodity at Microsoft unfortunately - that and the fact that I completely forgot about this.
Three years later, IIRC it was a case of filtering out everything but port 25 (SMTP) in the trace, then filtering out the IP of the external SMTP gateway and from that I saw the IP of the internal connecting client which in turn led us to the root cause.