Kevin Holman's OpsMgr Blog : Tools

Fixing troubled agents

kevinhol — Fri, 02 Oct 2009 00:23:46 GMT

Sometimes agents either will not “talk” to the management server upon initial installation, and sometimes an agent can get unhealthy long after working fine. Agent health is an ongoing task of any OpsMgr Admin’s life.

This post in NOT an “end to end” manual of all the factors that influence agent health…. but that is something I am working on for a later time. There are so many factors in an agent’s ability to communicate and work as expected. A few key areas that commonly affect this are:

DNS name resolution (Agent to MS, and MS to Agent)
DNS domain membership (disjointed)
DNS suffix search order
Kerberos connectivity
Kerberos SPN’s accessible
Firewalls blocking 5723
Firewalls blocking access to AD for authentication
Packet loss
Invalid or old registry entries
Missing registry entries
Corrupt registry
Default agent action accounts locked down/out (HSLockdown)
HealthService Certificate configuration issues.
Hotfixes required for OS Compatibility
Management Server rejecting the agent

How do you detect agent issues from the console? The problem might be that they are not showing up in the console at all! Perhaps they might be a manual install that never shows up in Pending Actions? Or a push deployment, that stays stuck in Pending actions and never shows up under “Agent Managed”. Or even one that does show up under “Agent Managed” but never shows as being monitored… returning agent version data, etc.

One of the BEST things you can do when faced with an agent health issue… if to look on the agent, in the OperationsManager event log. This is a fairly verbose log that will almost always give you a good hint as to the trouble with the agent. That is ALWAYS one of my first steps in troubleshooting.

Another way of examining Agent health – is by the built in views in OpsMgr. In the console – there is a view – Located at the following:

This view is important – because it gives us a perspective of the agent from two different points:

1. The perspective of the agent monitors running on the agent, measuring its own “health”.

2. The perspective of the “Health Service Watcher” which is the agent being monitored from a Management Server".

If any of these are red or yellow – that is an excellent place to start. This should be an area that your level 1 support for Operations manager checks DAILY. We should never have a high number of agents that are not green here. If they aren't – this is indicative of an unhealthy environment, or the admin team not adhering to best practices (such as keeping up with hotfixes, using maintenance mode correctly, etc…

Use Health Explorer on these views – to drill down into exactly what is causing the Agent, or Health Service Watcher state to be unhealthy.

Now…. the following are some general steps to take to “fix” broken agents. These are not in definitive order. The order of steps really comes down to what you find when looking at the logs after taking these steps.

Start the HealthService on the agent. You might find the HealthService is just not running. This should not be common or systemic. Consider enabling the recovery for this condition to restart the HealthService on Heartbeat failure. However – if this is systemic – it is indicative of something causing your HealthService to restart too frequently, or administrators stopping SCOM. Look in the OpsMgr event log for verification.

Bounce the HealthService on the agent. Sometimes this is all that is needed to resolve an agent issue. Look in the OpsMgr event log after a HealthService restart, to make sure it is clean with no errors.

Clear the HealthService queue and config (manually). This is done by stopping the HealthService. Then deleting the “\Program Files\System Center Operations Manager 2007\Health Service State” folder. Then start the HealthService. This removes the agent config file, and the agent queue files. The agent starts up with no configuration, so it will resort to the registry to determine what management server to talk to. From the registry – it will find out if it is AD integrated, or a fixed management server to talk to if not. This is located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\PROD1\Parent Health Services\ location, in the \<#>\NetworkName string value. The agent will contact the management server – request config, receive config, download the appropriate management packs, apply them, run the discoveries, send up discovery data, and repeat the cycle for a little while. This is very much what happens on a new agent during initial deployment.

Clear the HealthService queue and config (from the console). When looking at the above view (or any state view or discovered inventory view which targets the HealthService or Agent class) there is a task in the actions pane - “Flush Health Service State and Cache”. This will perform a very similar action to that above…. as a console task. This will only work on an agent that is somewhat responsive…. if it does not work you need to perform this manually as the agent is really broken from communication with the management server. This task will never complete, and will not return success – because the task breaks off from itself as the queue is flushed.

“Repair” the agent from the console. This is done from the Administration pane – Agent Managed. You should not run a repair on any AD-integrated agent – as this will break the AD integration and assign it to the management server that ran the repair action. A “repair” technically just reinstalls the agent in a push fashion, just like an initial agent deployment. It will also apply/reapply any agent related hotfixes in the management server’s \Program Files\System Center Operations Manager 2007\AgentManagement\ directories.

Reinstall the agent (manually). This would be for manual installs or when push/repair is not possible. This section is where the combination of options gets a little tricky. When you are at this point… where you have given up, I find just going all the way with a brute force reinstall is the best way. This means performing the following steps:

Uninstall the agent via add/remove programs.
Run the Operations Manager Cleanup Tool CleanMom.exe or CleanMOM64.exe. This is designed to make sure that the service, files, and all registry entires are removed.
Ensure that the agent’s folder is removed at: \Program Files\System Center Operations Manager 2007\
Ensure that the following registry keys are deleted:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService

Reboot the agent machine (if possible)
Delete the agent from Agent Managed in the OpsMgr console. This will allow a new HealthService ID to be detected and is sometimes a required step to get an agent to work properly, although not always required.
Now that the agent is gone cleanly from both OpsMgr console and the agent Operating System…. manually reinstall the agent. Keep it simple – install it using a named management server/management group, and use Local System for the agent action account (these will remove any common issues with a low priv domain account, and AD integration if used) If it works correctly – you can always reinstall again using low priv or AD integration.
Remember to import certificats at this point if you are using those on the individual agent.
As always – look in the OperationsManager event log…. this will tell you if it connected, and is working, or if there is a connectivity issue.

To summarize…. there are many things that can cause an agent issue, and many methods to troubleshoot. However – to summarize at a very general level, my typical steps are:

Review OpsMgr event log on agent
Bounce HealthService
Bounce HealthService clearing \Health Service State folder.
Complete brute force reinstall of the agent.

If it an external issue is causing the issue (DNS, Kerberos, Firewall) then these steps likely will not help you…. but those should be available from the OpsMgr event log.

Also – make sure you see my other posts on agent health and troubleshooting during deployment:

Console based Agent Deployment Troubleshooting table

Agent discovery and push troubleshooting in OpsMgr 2007

Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

Agent Pending Actions can get out of synch between the Console, and the database

Which hotfixes should I apply-

Alert Notification Subscription Variables, and linking that to the console, database, and SDK

kevinhol — Wed, 23 Sep 2009 20:54:00 GMT

Attached you will find a spreadsheet, with all the possible alert notification subscription variables that I am aware of. In this spreadsheet, I link these to the same values in the Alert table of the DB, the alert view of the DB, the Console alert view, the SDK (Get-Alert), and lastly the new R2 Connector Key pairs.

My thought was to better understand each data property type of an alert, and what you can managed, from each area. Hope this is beneficial.

Most of these are listed at my other blog post on alert description and notification variables: http://blogs.technet.com/kevinholman/archive/2007/12/12/adding-custom-information-to-alert-descriptions-and-notifications.aspx

Here is a sample shot of the spreadsheet tool:

See attached:

New web based forums for OpsMgr – no more NNTP newsgroups

kevinhol — Sat, 23 May 2009 01:37:46 GMT

Getting help from the community for OpsMgr just got easier.

We released the new interface to OpsMgr forums – on the web. No more NNTP newsgroups to parse, with a hard to use web interface.

http://social.technet.microsoft.com/Forums/en-US/category/systemcenteroperationsmanager

Check it out!

What’s the diff? Or – how to see exactly what is new with an updated MP

kevinhol — Thu, 12 Mar 2009 23:02:19 GMT

Back in MOM 2005…. we had some really cool reskit tools to look at differences in Management Packs. In the MOM 2005 reskit, there was a tool called MP2XML.exe which would take an AKM MP, and convert it to XML. Then, you could run another reskit tool, MPDiff.exe, which could graphically show you all the rules, scripts, providers, groups, attributes, etc… that had changed between the two MP versions. This was great – because we could really get a solid idea of what has changed in a MP update, down to the nuts and bolts. This is necessary – as a first step in understanding what's new, and making sure it wont break anything that we depend on.

In OpsMgr 2007, we really don't have a good tool here. There are several reasons… but mostly the XML options in SCOM are WAY more in number than what was available in MOM 2005. Writing this new diff tool would be nice… but is also pretty complicated. I am going to walk through an example of how you can accomplish this today, using some free and 3rd part tools.

First – we need to get our sealed and encrypted MP files into raw XML. The best way to do this, without having to depend on importing and exporting from a live management group, is to use Boris’s script:

http://blogs.msdn.com/boris_yanushpolsky/archive/2007/08/16/unsealing-a-management-pack.aspx

For this example – I am going to use the recently updated SQL 2005 discovery MP. This was updated from 6.0.6441.0 to 6.0.6460.0, in order to fix how we discover large SQL databases.

First – we will convert BOTH MP files to XML, using Boris’s script. I copied the script to my C:\BIN\ directory, and copied each MP file to C:\BIN\OLD and C:\BIN\NEW.

powershell C:\bin\MpToXml.ps1 -mpFilePath:'c:\bin\old\Microsoft.SQLServer.2005.Discovery.mp' -outputDirectory:'c:\bin\old\'

powershell C:\bin\MpToXml.ps1 -mpFilePath:'c:\bin\new\Microsoft.SQLServer.2005.Discovery.mp' -outputDirectory:'c:\bin\new\'

Now – once I have the raw XML, I can diff the files. There are lots of XML diff tools out there. XML Notepad is free from Microsoft, and I will also give an example from Beyond Compare 2, which I happen to use.

Using XML Notepad – open the old XML file. Then – View – Compare XML files:

Choose the new file. This will launch XmlDiff – and will have hyperlinks to click to jump to each detected change in the MP:

Using this tool – we can quickly see that the DiscoverSQL2005DB.vbs was the primary change here:

Using a tool like “Beyond Compare” will give a few more powerful features, like the ability to show all the code, or JUST the differences. This shows the actual code changes in the script… instead of just marking the script as what was changed:

There are likely other good XML diff tools out there. Obviously it would be nice to have one that could provide a summary in the context of OpsMgr… like Attributes, Discoveries, Class definitions, Scripts, Groups, etc…. and put that in a UI in plain English…. but until then, this should give us the information we need to digest what is new.

If any of you have some better/cooler XML diff tools, or approaches to this challenge – love to hear it!

How many consoles are connected to my RMS?

kevinhol — Mon, 27 Oct 2008 19:59:52 GMT

This discussion comes up quite a bit.

There are a couple ways to track this data.... even to alert us if it breaches a threshold.

One way to look at the number of connected consoles, is to examine the number of users that are connected to the SDK service. We can do this via powershell, and perfmon:

In powershell, this will show each connection to the SDK – whether it be a console session, or powershell session, or some other SDK connection, like managed code running:

$mgc=get-managementgroupconnection
$mgc.ManagementGroup.GetConnectedUserNames()

Here is a one liner version:

Get-ManagementGroupConnection | foreach-object {$_.ManagementGroup.getConnectedUserNames()}

If you’re looking for the actual NUMBER of users connected

Get-ManagementGroupConnection | foreach-object {$_.ManagementGroup.getConnectedUserNames()} | Measure-Object

That is great.... however, as stated, this shows SDK connections.... not specific to the console. I dont know of a way offhand to demonstrate which are true CONSOLE connections... but unless you run a lot of SDK processes like timed scripts against the SDK... most will be consoles in a typical environment.

Another way to view connections is via Perfmon. The perf object "OpsMgr SDK Service\Client Connections" and "OpsMgr SDK Service\Client Connections using cache" From what I can tell offhand, the "using cache" counter is the same as connected clients, minus the 2 default SDK connections from the service account.

We don't have a rule collecting this by default. However, you can create a collection rule, or even a performance threshold rule to alert you when this count goes above what you feel is supportable in your environment. Here is an example: (Note - Root Management Server is a good target for these rules/monitors... since it will be the only role with the SDK service running)

And the output of a monitoring view you can create:

For reporting purposes – you can see “connections” to the SDK – by writing an event collections rule for the following events, which is logged on each console session or powershell session that connects to the SDK: (this is logged on the RMS only)

Event Type:      Information
Event Source:   OpsMgr SDK Service
Event Category:            None
Event ID:          26328
Date:                10/16/2008
Time:                9:23:01 AM
User:                N/A
Computer:         OMRMS
Description:
A new client has connected.
UserName: OPSMGR\adadmin
SessionId: uuid:12e8d840-9cca-45df-b889-db39203136e0;id=15

Event Type:      Information
Event Source:   OpsMgr SDK Service
Event Category:            None
Event ID:          26329
Date:                10/16/2008
Time:                9:24:04 AM
User:                N/A
Computer:         OMRMS
Description:
A client has disconnected.
UserName: OPSMGR\adadmin
SessionId: uuid:12e8d840-9cca-45df-b889-db39203136e0;id=15

However – this won’t show you good concurrence…. just “who” is using the console…. and how often connections are made and disconnected. Since the username is a parameter..... you could potentially write reports and examine your most common console users, etc.

So - we have the ability to:

1. See "who is connected right now" - Powershell

2. Count the client connections right now - Powershell and Perfmon.

3. Collect the client count from the RMS - Perfmon rule

4. Alert when it reaches a threshold we set - Perfmon Monitor

5. Collect and report on actual user connections - Event rule and report.

MPViewer 1.7 released

kevinhol — Wed, 02 Jul 2008 23:52:05 GMT

MPViewer 1.7 - Now works with latest E12 MP

Get it.

A simple command to backup all Management Packs

kevinhol — Mon, 16 Jun 2008 12:42:23 GMT

Launch the System Center Operations Manager \ Command Shell:

▪get-managementpack | export-managementpack -path c:\backup

Also, for some advanced automation.... check out:

http://www.systemcenterforum.org/automate-mp-export-for-disaster-recovery/

http://derekhar.blogspot.com/2008/02/unsealed-mp-backup-with-retention.html

OpsMgr SP1 is HERE!

kevinhol — Sat, 23 Feb 2008 04:01:00 GMT

The final Service Pack 1 has been released!

Version 6.0.6278.0

Tired of your console throwing errors - or manually clearing your console cache?

kevinhol — Thu, 20 Dec 2007 00:43:00 GMT

You can tell the Ops console to clear the cache everytime you open it! Just change your shortcut - and add the /clearcache parameter:

"C:\Program Files\System Center Operations Manager 2007\Microsoft.MOM.UI.Console.exe" /clearcache

Keep in mind - this is no longer recommended post SP1 RTM..... just when you are having a problem. By using this setting on all your users, you will experience a bigger hit on the RMS because we arent using the cache effectively.