Thoughts from the EPS Windows Server Performance Team
Useful Microsoft Blogs
Every couple of months or so, we'll get a call from an Administrator reporting that his system hung when he tried to reboot it after installing patches. More often than not, the patches were installed by an Administrator logging on to the server via Remote Desktop (without using the /console or /admin switch) and using either the Windows Update web site or the "Automatic Updates" tray icon. Both methods do the same thing and use the same processes. After the updates are installed, the Administrator clicks on the "Restart Now" button to complete the installation. The Remote Desktop Session goes away, and the Administrator thinks that the server is in the process of rebooting.
However, the problem is that the server may not really be rebooting. When the Administrator tries to connect back into the server via RDP after several minutes, he discovers that he cannot. When he logs on at the console of the machine to investigate, he discovers that the RDP Listener is listening on port 3389 but no-one can connect via RDP. To resolve the issue, he has to reboot the server from the console. So what happened?
The first place to start is with the installation log files for the updates. In the %SYSTEMROOT% folder, there are several .log files created when patches are installed:
When you open up one of the files you can walk through the installation of the patch, including the following information:
[KB885836.log]0.656: ================================================================================0.656: 2008/02/05 10:20:34.046 (local)0.656: c:\53f60e03a81769236c7d3218\update\update.exe (version 220.127.116.11)0.656: Service Pack started with following command line: /q /z0.859: ---- Old Information In The Registry ------
1.703: Failed To Enable SE_SHUTDOWN_PRIVILEGE
Getting back to our problem, the most likely problem is that there is a global variable for Terminal Services which has been set. This is set at the beginning of the shutdown process. Since we are not in the Console Session, we "ask" other processes if we can restart the system as opposed to forcing them to close and restarting the system. If one of those processes does not "allow" us to restart, then the system shutdown is aborted. Since the RDP session has already exited, the Administrator thinks that the system is restarting, but we are really in limbo. The Terminal Services process restarts the listener on port 3389, but since the global variable has been set, any attempt to connect to RDP is unsuccessful. Obviously the reboot clears this, and everything appears normal once more.
So what's the solution? There are a couple of different ways to address this (beyond installing the patches at the physical console):
- David John
we get this problem all the time and not just when installing patches... we'll just reboot from the start menu and the same behavior happens... I'm fairly certain the /console switch doesn't help either... we have to either reboot it from the physical console or run a shutdown command from a remote system. This has been plaguing us for years!!!
Every couple of months or so, we'll get a call from an Administrator reporting that his system hung
This partly explains why I've been seeing this problem occur since session 0 support got ripped from the RDP client. The other part of the problem is that SBS RWW doesn't know about the session 0 change, so selecting "Log on to the selected computer as administrator" no longer works as expected when the newer client is used on the PC accessing SBS RWW.
Sorry - cut'n'paste error - that should be the "Log on to or resume the console session of the remote computer" option.
We use Process Explorer and use the restart command on it's File\Shutdown menu. Never had any issues.
@Clayton: we get the same thing a fair bit. we have even tested this - Log in via RDP (with our without console), do a Start/Shutown/Restart, get kicked from the rdp session as you expect, but the server doesn't rebooot
A ping -t to the server never shows a drop packet, but any new rdp sessions will fail.
Connect to the physical console (all our physical boxes are HP so we just iLo, for virtual machines use vmwares admin tool), and when you log in you can reboot.
We have always assumed there is some wierd race condition going on.
Is it possible to change the location of the patch installation logs ?
Thanks in advance,
@Clayton: Have a look at KB930045 <http://support.microsoft.com/kb/930045> which fixes some but not all race conditions when shutting down a server through a RDP session.
We have seen this behavior for a couple of years now, and although the /console switch to RDP helps, it still occurs often.
What I don't understand, is that for the past year, we have gotten the same thing when we use WSUS server to patch our servers.
I had found some forum discussions that attributed it to the version of WSUS we were using, WSUS 2.0.
Several months ago, I approved patches, (no SP's) for 200 + servers over the weekend.
Of the 200+ patched, 78 did not complete the restart and were left in the state described here.
It was supposed to be fixed in WSUS version 3.0
I have just run a test update from the version 3.01 server, and my test server balked at restarting!
1 additional piece of weirdness: on some of the hanging servers, as soon as a console session is established, whether it's ilo or KVM/IP, the server shuts down and reboots immediately.
Some posters have suggested Anti-Virus is the cause, but this is the 1st thread I have found describing the symptom exactly.
In doing security patching for a large corporation, we have noticed clearing other ts session is a must. In addition the 'restart now' via wsus or 'start->shutdown' methods appear to cause trouble.
In rare cases we will find another ts session we are unable to kick. Although we have the necessary permissions. This will cause a hang virtually every time.
For best results we use either a remote restart using the shutdown -i command or we run "shutdown /r /t 10 /c "Rebooting for Updates" on the local machine.
I know this is a very old post but we have been getting hit up automatic updates on our servers not rebooting. Many of our servers are set to automatically update and reboot but have been hanging on the "Windows is shutting down" part. I am wondering if what you have stated above applies in that situation. In addition, would setting "No auto-restart for scheduled Automatic Updates installations" to disabled (thus requiring a reboot) in a GPO fix this?