if (ms) blog++;

Random bits of (hopefully) useful technical information on Windows, with a focus on understanding and troubleshooting.

HTTP.SYS / Cryptographic Services / LSASS.EXE deadlock

HTTP.SYS / Cryptographic Services / LSASS.EXE deadlock

  • Comments 11
  • Likes

NOTE: This issue has been subsequently fixed in KB 2379016 A computer that is running Windows Vista or Windows Server 2008 stops responding at the "Applying User Settings" stage of the logon process.
(This blog entry is left for reference.)

A recent case I had brought this issue to my attention, so I thought it useful to share the knowledge…

The problem encountered was a Windows Server 2008 x64 SP2 server running several websites was failing to start several services during startup, and attempts to logon stuck at “Applying User Settings…” indefinitely.
Starting in Safe Mode (or Safe Mode with Networking) worked fine.

By the time the case had been opened the symptom had already been removed by taking away some certificates – the server then started correctly when booted normally (adding re-adding the certificates did not reintroduce the problem).

 

From a VM image of the server in the problem state I made a complete memory dump when the server had been stuck ~10 minutes during user logon.

From the hang dump I could see the logon was stalled because one of the threads in LSASS.EXE was waiting for the Cryptographic Services service to start – an ALPC was sent to to the Service Control Manager (SCM) to poke the thread when the service was up.

SCM was in the process of starting HTTP.SYS, which held the lock for service startup (preventing the Cryptographic Services service from starting) and HTTP.SYS’s thread requires the services of LSASS.EXE…deadlock.

http-lsass-cryptsvc_deadlock

This is a classic timing issue – HTTP.SYS is slow to load (for some reason relating to the SSL binding information) and is holding the SCM lock long enough to stumble into the “Mexican stand-off”.

We can iron out the kink by putting a dependency for HTTP.SYS on Cryptographic Services (CryptSvc), so it will always wait, so when it makes a call into LSASS.EXE it is not holding the lock preventing LSASS.EXE’s dependency from starting.

This workaround is described here: Computer hangs at Applying Computer Settings or All Automatic Services Will Not Start After Reboot on Windows Server 2008

 

Does this mean that the symptom of “Automatic services failing to start” are caused by this one issue?
NO

The symptom is unfortunately very generic – there is so much going on with a system startup that there are many ways to encounter a “hang” with totally different root causes.

It is very easy to say “I am experiencing the symptoms described in KB article XXX, but the solution/workaround did not work” – this invariably means that you did not have that problem described by the article.

But if you have a server with certificates installed and SSL is configured (most probably for use by IIS), and you’re running Windows Server 2008 then it’s worth knowing about this issue and trivial workaround.

(Windows Server 2008 R2 does not have the problem, by the way.)

 

Comments
  • Just wanted to say this has really helped me out, I have been working for hours trying to get our web server back on-line.

    Our server was stuck on configuring updates stage 3 of 3 - 0% after installing some automatic updates and rebooting.  After removing the updates in safe mode the server booted backup and was then stuck at "Applying User Settings".  

    I spent ages going into safe mode and disabling services and trying to reboot again, I was ready to give up and start to task of rebuilding the server when came across this article and I am now finally back online!

  • Just like Kiffy I was stuck with this issue for quite a long time. In my case I was running a Server 2008 environment with SharePoint 2010. My VM would make it to the desktop, but most services were not running. The Event Viewer held very little information as to what was causing this.

    The behavior of the Service Manager quickly pointed to the Service Controller itself as I was unable to stop, start or alter any service. Additionally, Process Monitor showed very little activity when attempting to start a service (the actual executable of the underlying service was never touched). At first I managed to work around the issue by disabling the Windows Management Instrumentation service, but this of course caused all kinds of other problems.

    Eventually, I ended up with the "sc querylock" command, which told me that the SCM database was locked, which lead me to this article and the actual solution.

    In my opinion, they could have at least made the Service Manager more robust and produce an actual error when the SCM database is locked. Right now the Service Manager either crashes or produces a general "service did not respond" error.

    Many thanks!

  • I' m in the dark with same problem. I see the light now .

    Thanks a lot !!!

  • Thank you so much for posting this information.  I recently moved from self signed certificates to GoDaddy issued certificates in my IIS bindings, and in no way did I suspect such a crippling error to occur as a result.  How is it resolved in R2?  Also, is there anything particular about the origin of the certificates that slows it down?  We had to specifically install intermediate certificates on the server so I wonder if that chain has a problem.  Maybe they weren't placed in the Computer's store?

  • "How is it resolved in R2?"

    This is a classic timing issue, and even though there would appear to be little difference between Windows Server 2008 and (NT 6.0) Windows Server 2008 R2 (NT 6.1), there is a lot of code churn and redesign which modifies the order in which common activities occur, sometimes resolving potential timing issues as a side effect.

    There is nothing that I know of that is common between the certificates that trigger the problem, and the issue has not been widely reported to investigate a causal link - if the certificates themselves were incorrectly stored then I would expect them to fail to work at all, not just have a timing issue.

  • Update - there is now a hotfix to address the issue:

    support.microsoft.com/.../2379016

  • I had this happen to two servers of mine, the first I rebuilt after being unable to troubleshoot it.  This fixed the 2nd issue.  In hopes that search engines will pick this up, another sign that I was having was being unable to open the Windows Update dialog, my network connections would not start and I could not open the network connection and sharing dialogs.

  • Is there a fix for Windows XP and Server 2003? I am having this problem and it is affecting many computers. It has just recently started and there really has not been any changes, except installing SP3 for WXP.

  • The problem has never been reported for XP or Server 2003 to my knowledge, so I would check whether it's definitely this issue you are seeing (or just the symptom).

    Does the problem occur on every restart?

    Does the problem go away if you use the workaround of setting the dependency manually?

    If the answer to either of these is "no" then it's not the same issue.

    Unfortunately both of those products are in the Extended Support phase, which means only security (GDR) updates will be created for the next 3 years for them.

  • Have you recently installed a new HTTPS SSL Certificate and since then, your Server does not boot any more? Only safe mode is working, but many services do not start?

    The Cause is a deadlock beween some services, see this HTTP-CryptSvc-LSASS-Deadlock

  • Thanks, it worked for me, too!

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment