UPDATE: 12/2/2013: This has been fixed with KB2878378
The Operations Manager support team has identified a regression in a recent Windows update. After installing the following update on Operations Manager Management Servers, agents or servers may be affected by a deadlock.
KB2775511 - An enterprise hotfix rollup is available for Windows 7 SP1 and Windows Server (http://support.microsoft.com/kb/2775511)
Once in deadlock, Management Servers will generate HeartBeat failures and will go into a “greyed out” state. grayed out
As a result, devices managed by these Management Servers will also go into a “greyed out” or “not monitored state.”
When in this state:
1. No new events will be generated in the Operations Manager Event Log
2. Errors in State Change Events will appear frequently
3. Alerts will be generated during state change
Removal of KB2775511 will correct the issues introduced. The OpsMgr team recommends that Operations Manager users refrain from installing KB2775511 until this deadlock issue is resolved. New information will be posted as it becomes available.
Laura Cruz | CDM Program Manager | Microsoft
Get the latest System Center news on Facebook and Twitter:
System Center All Up: http://blogs.technet.com/b/systemcenter/ System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/ System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/ System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/ System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/ System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm
Windows Intune: http://blogs.technet.com/b/windowsintune/ WSUS Support Team blog: http://blogs.technet.com/sus/ The AD RMS blog: http://blogs.technet.com/b/rmssupp/
The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/ The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/ The Forefront TMG blog: http://blogs.technet.com/b/isablog/ The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/
I saw this exact issue when I patched my Windows Server 2003 SCOM 2007 R2 RMS at the weekend. Fortunately, a reboot of the RMS resolved the issue! At least it has temporarily...
Are you suggesting we remove the hotfix from agents or just the management servers?
kb2775511 is really a bunch of other individual hotfixes. by my rough count at support.microsoft.com/.../2775511, the rollup contains close to 50 hotfixes. I assume the scom issue isn't caused by all 50 hotfixes in the rollup, so could we narrow the problem down to one or two hotfixes and uninstall those? on my machines, kb2775511 does not appear in Programs and Features; the individual hotfix kb numbers do.
"After installing the following update on Operations Manager Management Servers, agents or servers may be affected by a deadlock."
Could the scope of this be clarified? Microsoft recommended this rollup for all systems. We have found it very beneficial to a number of systems.
Is there any risk to an agent managed server which has this rollup installed?
It's good to know this. We have suffered by this 2115 warning for months... Finally...
Our recommendation is to not install the update on management servers but be prepared to uninstall from agents as well. The bug is a race condition that may happen under load. It’s more likely on a loaded management server but possible on agents as well, so if you're seeing agents going gray you may need to remove the update rollup from those systems as well.
As far as individual fixes go, KB277511 is definitely an update that can be installed standalone. The root bug was introduced with KB2734909. Since there is another advapi32 fix included with KB277511 it picked up both fixes.
Just wanted to clarify that the problematic KB itself is KB2734909 (support.microsoft.com/.../2734909).
Did you make sure the registry keys recommended in the article 2775511 were applied and this issue still reproed? ie, was the hotfix rollup installed properly?