Kevin Holman's OpsMgr Blog : agents

29106 event on RMS – Index was out of range. Wait. What?

kevinhol — Tue, 10 Nov 2009 20:01:36 GMT

Was working with a customer on this one – figured it might help others.

Saw a lot of these VERY SPECIFIC 29106 events on the RMS, specifically with the text:

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

Here is the full event:

Event Type:      Warning
Event Source:    OpsMgr Config Service
Event Category: None
Event ID:        29106
Date:            11/10/2009
Time:            12:43:24 PM
User:            N/A
Computer:        AGENTNAME
Description:
The request to synchronize state for OpsMgr Health Service identified by "3688d65d-a16c-2be6-7e84-5faf8a9cffe0" failed due to the following exception "System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index

What we found was – that we could look up these health service ID’s – by pasting them in the following SQL query:

select * from MTV_HealthService
where BaseManagedEntityId = '3688d65d-a16c-2be6-7e84-5faf8a9cffe0'

This would give us the name of the agent.

In the console, under Agent Managed – we found all of these agents were in “Unmonitored” state – on the agents themselves, they were stuck. They looked like they got installed, but could not get config. We deleted them from agent managed, waited a few minutes, and let them show back up in Pending Management. Approved them – then they were able to come back in and work properly. These looked for the most part like orphaned machines, and several were computers that were renamed, or old DC’s that were demoted.

What is config churn?

kevinhol — Mon, 05 Oct 2009 08:29:49 GMT

There have been a couple good articles briefly covering this topic…. you might have read them. I will reference some below. Config churn is basically, when your RMS is in an almost never-ending loop of generating config. This can be caused by “less than optimized” management packs, pushing agents all the time, or injecting major changes into a management group, such as overrides or custom rules and monitors, or importing updated management packs. By examining this topic in depth – we will re-state some already known best practices with maintaining a healthy management group, and get some deeper knowledge as to why they are best practices in the first place.

Any time you push agents, or create rules and monitors, or overrides for widespread classes….. you can create a config update on the RMS that must be sent down to ALL agents in the management group. For small management groups (under 500 agents) this is generally not a big deal and processes rather quickly. For large management groups over 1000 agents, this can cause high resource utilization of the RMS and SQL Database, in terms of CPU, Memory, and Disk I/O. This can impact data insertion, and console performance during these times. For these reasons, we like to keep those activities down to a minimum during working hours, and schedule these major changes in an off-hours maintenance window.

What about “less than optimized” management packs? What does that mean? Well, this means management packs that you might be using, that have poorly written discoveries.

We have long known that a worst practice in Management Pack development, is to have a discovery that discovers instances of a class, that has properties for those instances that are likely to change frequently. Here is a write-up from OpsManJam on the topic: LINK

Ok… wait… Whaaaaat?

Let me put that in English:

Say we have a discovery for a Logical Disk. This will discover any logical disk, like C:, D:, E:, Q:, etc…. When we write the discovery for a logical disk, we can add properties to that discovery. These are attributes of the discovered instances. So – in this case – lets say we decided to add “Size” of the disk as a property, and “Free Space” as a property. And for the discovery frequency – we will run this discovery every hour, looking for new disks.

“Size” is an excellent property for the Logical Disk class. We like to know the size of the disks…. we can use this property group them if needed. “Size” of a logical disk is not something that we would expect to change very often.

“Free Space” is a horrible property for the Logical Disk class. Free space is something that will likely change, just a small amount even, between each run of the discovery. Free space is a property that is likely to change frequently, therefore – it should NOT be used in a discovery.

Make sense?

Ok – so… what's the big deal?

Well, the agent will run almost all discoveries that it knows about when the health service starts up (like when you bounce the service, or after a reboot). It will always send this discovery data to the management server. (this is another reason why agents restarting all the time is very bad) Then, it will run then based on the “Interval” frequency specified on the discovery. Sometimes this is as frequent as once per hour, sometimes as long as once per day. When the discovery runs, the agent will inspect the discovery data that it gets, and compare it to the last discovery data it sent to the management server. If nothing changed – the agent drops the discovery data and does nothing. IF anything changed in the values of the discovery data – it will re-submit the new data to the management server, which will submit this data to the database. The RMS will detect the change, and will have to recalculate (regenerate) configuration. You will see this on the RMS as a 21025 event:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          9/27/2009 11:51:49 PM
Event ID:      21025
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
OpsMgr has received new configuration for management group PROD1 from the Configuration Service. The new state cookie is "D7 9B A4 BE 00 90 CF 13 35 B5 9B 5F 3B 14 FF 78 D6 13 9A 2D "

The 21025 event isn’t really “bad”… it simply means the config service did its job. It re-generated its configuration file from the database data, and wrote it to: \Program Files\System Center Operations Manager 2007\Health Service State\Connector Configuration Cache\<MGNAME>\OpsMgrConnector.Config.xml The problem is – when this config file gets large (like in large agent count environments) and when the “Config Instance Space” is large (number of discovered objects in total). Recalculating this config can have a significant impact on the disk where the file exists on the RMS, use lots of memory and CPU on the RMS for the config service, and use significant disk I/O on the SQL database.

If the RMS is in a perpetual cycle of recalculating config, and sending these config updates to all agents…. the performance of the management group is impacted.

Daniele Grandini of Quaue Nocent Docent is pretty much the “godfather” of good information researching the 21025 event. Read his 3 part series on config churn here:

http://nocentdocent.wordpress.com/2009/07/09/troubleshooting-21025-events-wrap-up/

So – what can I do if I think I have too much config churn?

The biggest problem causing the most frequent config updates is management packs with noisy discoveries. However, lets wrap up all the issues that can cause it, and what you can do:

New agents. Discover/install/approve new agents in bulk and off-hours.
Overrides. Set overrides during off-hours, or create override MP’s in a lab, then synch to production management groups during schedule off-hours times.
Custom rules and monitors. Create these during off-hours, or create using the authoring console, test in a lab, then import to production during off-hours.
Newly discovered instances. For instance – someone adds a new disk, or SQL database, or DNS zone, to an existing agent. Not much we can do about this, except the expectation that this would be done during off hours.
Management packs with noisy discovery properties. See below.

Ok – the remainder of this article will touch on #5.

How can I tell which discoveries are noisy?

Danilele Grandini has put together a good query on this, from his link: http://nocentdocent.wordpress.com/2009/05/23/how-to-get-noisy-discovery-rules/

I will repost these (slightly modified) below:

/* Top Noisy Rules in the last 24 hours */

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes'
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

and

/* Modified properties in the last 24 hours */

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName

Wow – that returned a LOT of discoveries running all the time! What can I do?

Don't import too many MP’s! The FIRST line of defense – is NOT to import ANY management packs into a management group that you don't absolutely need RIGHT THEN. Management packs are constantly updated, and by the time you have an actual SLA in a technology area – there will likely be a newer, better MP available for it. The biggest mistake many customers make is to import any available MP for a technology that they have internally. They end up with a FLOOD of alerts, big fat databases, slow consoles, and lots of weird errors. MP’s should be transitioned slowly, one at a time – tuning and resolving as you go.

Disable the noisy discoveries. Probably not a great solution, unless they discover objects that you really don't care about – but there are other objects in the MP that you DO want to monitor.

Increase the interval of the discovery frequency. This means… essentially – change any “bad” discoveries to run only once per day.

Add a “synch time” override to the discovery – if possible. This option is not available unless the MP author of the discovery exposed it. What this will do – it cause all the agents to ONLY run the discovery at a distinct and specified time every day (say…. 1AM). This might cause too much discovery data to flood in at one time… but since it will all come in at the same time – it wont cause constant config churn all throughout the day.

Re-write the discovery. If this is a custom MP – rewrite the discovery/MP, and remove that property which changes too often.

Make sure your hardware and software is optimized for scalability. On your RMS – it is good to place your config file on fast disks, especially in large environments. I have worked with very large customers who were experiencing config churn, but had zero ill effects, because their RMS disk I/O was on a 4 spindle RAID10 with 15K spindles, CPU and memory were really good, and their SQL database disk I/O for the OpsDB was excellent with plenty of breathing room. I have also worked with smaller agent counts, where config churn has a serious impact…. mostly due to the RMS config file being places on the same RAID spindle set as that OS and pagefile, using only 2 older 10,000 RPM disks. The SQL disk I/O was also just borderline for their agent count.

Re-run the queries periodically – especially after importing/upgrading to a new management pack in your management group. This “instance space change” report should be part of your testing and evaluation of a new MP when brought into your lab…. if you have a large agent count environment.

Some very common discoveries I have seen – that have properties that change very frequently – are listed below. I often recommend these be overridden to run once per day (86,400 seconds)

Discovery Display Name	Discovery Target Class	Discovered Type	Default frequency	Modified frequency
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File	7260	86400
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File Group	7260	86400
Discover SQL 2000 Databases	SQL 2000 DB Engine	SQL 2000 DB	1800	86400
Discover Databases for a Database Engine	SQL 2005 DB Engine	SQL 2005 DB	7200	86400
DNS 2003 Component Discovery	DNS 2003 Server	DNS 2003 Zone	21600	86400
DNS 2008 Component Discovery	DNS 2008 Server	DNS 2008 Zone	21600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2003 Server Role	IIS FTP Site	3600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2000 Server Role	IIS NNTP Virtual Server	3600	86400

The above is just a sample – you should examine the query output of the query above and see what is impacting your management group the most.

Keep your management pack names SHORT in SP1!

kevinhol — Fri, 02 Oct 2009 23:31:08 GMT

I have seen this twice now… so I will blog about it. It seems to be rare in the wild, but it will completely cripple a management group when this occurs. So beware SP1 users!

This article does not apply to R2. This is only an issue in OpsMgr 2007 SP1.

When you create your custom management packs – and especially your override management packs – keep the names as simple and short as possible. There is an issue in OpsMgr SP1 – when an agent tries to download management packs – where it will fail if the MP ID (derived from the MP Name) is too long. The worst part about this problem is there there WONT be an error logged. What will happen – is that the agent will keep trying to re-download the MP in question, and it will block ALL other MP’s from being downloaded from that point forward.

There is no simple way to know this condition is impacting you. What will happen – is that an agent will continue to work just fine… but will not get any NEW management packs. So… you might think all is well, but not so. The symptoms thatmight lead you to notice that something is wrong:

Performance data not collected for newer MP’s.

Objects not being discovered for newer MP’s

Alerts not generating as expected from newer MP’s.

The root problem is an age old Windows issue…. file paths over 255 characters not supported well. This has been resolved in R2 in how the agent copies the files over.

In both cases I have seen – someone was creating an override MP for the “IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems” management pack. So – when they created their override MP – they named it something like: “Overrides - IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems”

This equated to a Management Pack ID of: Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems

That MP ID is 86 characters!

What happens…. is when this management pack is created, and an override is placed in it…. (or custom rule)… the agents that require it will:

Get contacted by the RMS to update their config, and then issue a config change request (21024 in the event log)

Receive new config from the RMS (event 21025)

Process new config – and realize they need a new MP, and request that MP (event 1200)

Where this process breaks…. is the next step in the chain should be that the agent RECEIVES the MP (event 1201) and then issues a statement that the new config has become active (event 1210). The never happen.

Behind the scenes…. from looking at a ETL tracelog, we can see this is failing, when we try to move the file from the “downloaded files” folder to the “management packs” folder:

Error CMPFileManager::MoveManagementPackFile(MPFileManager_cpp383)MoveFile from '\\?\C:\Program Files\System Center Operations Manager 2007\Health Service State\Downloaded Files\MGNAME\1\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' to 'C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' failed with code 3(ERROR_PATH_NOT_FOUND).

In the example above – the bad path is:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml

Which is 261 characters. The limit is 255.

Therefore – I recommend you keep your Management Pack *ID* to less than 60 characters. You can examine your long management packs by looking in the console – generally your longest display names will be the longest ID’s:

Even some Microsoft MP’s are dangerously close to the limit…. such as: Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMWare.HostPerformance with 76 characters. In most environments you can squeak by at 79 characters…. more or less depending on where you installed your agent path.

Here is a SQL query you can run against the OpsDB to also detect this condition…. and quick check all your potentially long MP’s:

select MPName from managementpack
WHERE len(MPName) > 60

Just change 60 to whatever character count you want.

DONT freak out if you have some more than 60. Just be aware.

DO freak out if you have some more than 80!

If someone had the time and was really handy – you could write a monitor – that runs against the RMS – that would in turn query the OpsDB, and run this query, and change the RMS to unhealthy when over a threshold. THAT would be cool…. and alert you when some author makes a really long MP that has the potential to break all your agents.

Another idea I had… was to create a correlated missing event monitor…. and when you get a 1200, but do NOT get a 1201 within, say, 15 minutes….. that might be a problem. Of course if you wrote this and were already impacted…. the bad agents would never get your new MP to tell you. :-)

Fixing troubled agents

kevinhol — Fri, 02 Oct 2009 00:23:46 GMT

Sometimes agents either will not “talk” to the management server upon initial installation, and sometimes an agent can get unhealthy long after working fine. Agent health is an ongoing task of any OpsMgr Admin’s life.

This post in NOT an “end to end” manual of all the factors that influence agent health…. but that is something I am working on for a later time. There are so many factors in an agent’s ability to communicate and work as expected. A few key areas that commonly affect this are:

DNS name resolution (Agent to MS, and MS to Agent)
DNS domain membership (disjointed)
DNS suffix search order
Kerberos connectivity
Kerberos SPN’s accessible
Firewalls blocking 5723
Firewalls blocking access to AD for authentication
Packet loss
Invalid or old registry entries
Missing registry entries
Corrupt registry
Default agent action accounts locked down/out (HSLockdown)
HealthService Certificate configuration issues.
Hotfixes required for OS Compatibility
Management Server rejecting the agent

How do you detect agent issues from the console? The problem might be that they are not showing up in the console at all! Perhaps they might be a manual install that never shows up in Pending Actions? Or a push deployment, that stays stuck in Pending actions and never shows up under “Agent Managed”. Or even one that does show up under “Agent Managed” but never shows as being monitored… returning agent version data, etc.

One of the BEST things you can do when faced with an agent health issue… if to look on the agent, in the OperationsManager event log. This is a fairly verbose log that will almost always give you a good hint as to the trouble with the agent. That is ALWAYS one of my first steps in troubleshooting.

Another way of examining Agent health – is by the built in views in OpsMgr. In the console – there is a view – Located at the following:

This view is important – because it gives us a perspective of the agent from two different points:

1. The perspective of the agent monitors running on the agent, measuring its own “health”.

2. The perspective of the “Health Service Watcher” which is the agent being monitored from a Management Server".

If any of these are red or yellow – that is an excellent place to start. This should be an area that your level 1 support for Operations manager checks DAILY. We should never have a high number of agents that are not green here. If they aren't – this is indicative of an unhealthy environment, or the admin team not adhering to best practices (such as keeping up with hotfixes, using maintenance mode correctly, etc…

Use Health Explorer on these views – to drill down into exactly what is causing the Agent, or Health Service Watcher state to be unhealthy.

Now…. the following are some general steps to take to “fix” broken agents. These are not in definitive order. The order of steps really comes down to what you find when looking at the logs after taking these steps.

Start the HealthService on the agent. You might find the HealthService is just not running. This should not be common or systemic. Consider enabling the recovery for this condition to restart the HealthService on Heartbeat failure. However – if this is systemic – it is indicative of something causing your HealthService to restart too frequently, or administrators stopping SCOM. Look in the OpsMgr event log for verification.

Bounce the HealthService on the agent. Sometimes this is all that is needed to resolve an agent issue. Look in the OpsMgr event log after a HealthService restart, to make sure it is clean with no errors.

Clear the HealthService queue and config (manually). This is done by stopping the HealthService. Then deleting the “\Program Files\System Center Operations Manager 2007\Health Service State” folder. Then start the HealthService. This removes the agent config file, and the agent queue files. The agent starts up with no configuration, so it will resort to the registry to determine what management server to talk to. From the registry – it will find out if it is AD integrated, or a fixed management server to talk to if not. This is located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\PROD1\Parent Health Services\ location, in the \<#>\NetworkName string value. The agent will contact the management server – request config, receive config, download the appropriate management packs, apply them, run the discoveries, send up discovery data, and repeat the cycle for a little while. This is very much what happens on a new agent during initial deployment.

Clear the HealthService queue and config (from the console). When looking at the above view (or any state view or discovered inventory view which targets the HealthService or Agent class) there is a task in the actions pane - “Flush Health Service State and Cache”. This will perform a very similar action to that above…. as a console task. This will only work on an agent that is somewhat responsive…. if it does not work you need to perform this manually as the agent is really broken from communication with the management server. This task will never complete, and will not return success – because the task breaks off from itself as the queue is flushed.

“Repair” the agent from the console. This is done from the Administration pane – Agent Managed. You should not run a repair on any AD-integrated agent – as this will break the AD integration and assign it to the management server that ran the repair action. A “repair” technically just reinstalls the agent in a push fashion, just like an initial agent deployment. It will also apply/reapply any agent related hotfixes in the management server’s \Program Files\System Center Operations Manager 2007\AgentManagement\ directories.

Reinstall the agent (manually). This would be for manual installs or when push/repair is not possible. This section is where the combination of options gets a little tricky. When you are at this point… where you have given up, I find just going all the way with a brute force reinstall is the best way. This means performing the following steps:

Uninstall the agent via add/remove programs.
Run the Operations Manager Cleanup Tool CleanMom.exe or CleanMOM64.exe. This is designed to make sure that the service, files, and all registry entires are removed.
Ensure that the agent’s folder is removed at: \Program Files\System Center Operations Manager 2007\
Ensure that the following registry keys are deleted:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService

Reboot the agent machine (if possible)
Delete the agent from Agent Managed in the OpsMgr console. This will allow a new HealthService ID to be detected and is sometimes a required step to get an agent to work properly, although not always required.
Now that the agent is gone cleanly from both OpsMgr console and the agent Operating System…. manually reinstall the agent. Keep it simple – install it using a named management server/management group, and use Local System for the agent action account (these will remove any common issues with a low priv domain account, and AD integration if used) If it works correctly – you can always reinstall again using low priv or AD integration.
Remember to import certificats at this point if you are using those on the individual agent.
As always – look in the OperationsManager event log…. this will tell you if it connected, and is working, or if there is a connectivity issue.

To summarize…. there are many things that can cause an agent issue, and many methods to troubleshoot. However – to summarize at a very general level, my typical steps are:

Review OpsMgr event log on agent
Bounce HealthService
Bounce HealthService clearing \Health Service State folder.
Complete brute force reinstall of the agent.

If it an external issue is causing the issue (DNS, Kerberos, Firewall) then these steps likely will not help you…. but those should be available from the OpsMgr event log.

Also – make sure you see my other posts on agent health and troubleshooting during deployment:

Console based Agent Deployment Troubleshooting table

Agent discovery and push troubleshooting in OpsMgr 2007

Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

Agent Pending Actions can get out of synch between the Console, and the database

Which hotfixes should I apply-

Getting lots of Script Failed To Run alerts? WMI Probe Failed Execution? Backward Compatibility Script Error?

kevinhol — Mon, 29 Jun 2009 17:24:31 GMT

In OpsMgr 2007, it is likely that your most common alert is not really a MP based alert from a technology management pack…. it could be a built-in alert that a script failed, or WMI could not be accessed. This is because when WMI is broken on a machine, almost EVERYTHING fails to execute properly on that agent.

At a recent health check at a customer site, we found the top 5 alerts in his environment (by cumulative repeat count) were:

WMI Probe Module Failed Execution
Service Check Data Source Module Failed Execution
Backward Compatibility Script Error
Script or Executable Failed to run
Service Check Probe Module Failed Execution

Sometimes – these alerts are normal…. the server is busy, or someone rebooted it without putting it into maintenance mode and allowing the workflows to unload gracefully.

However, if you have a high repeat count on these, it is typically indicative of something seriously broken on that agent(s). Most of the time – the failure is in WMI. Many customers get frustrated with these script errors, because they see them as “false alerts” because they don't know how to resolve the root cause, and we just tell you “this action broke”, we don't tell you why. It is critical that you examine these alerts, however, because these alerts will indicate something seriously wrong with an agent, such as broken WMI/cscript/OS issue. If you ignore them, or disable them – you will never know that monitoring is not functioning 100%.

Generally – here is how I attack script/WMI failures.

1. If the repeat count is 0 or 1, I ignore these as random failures, and close the alerts from time to time.

2. If the repeat count is very high, then something is wrong with the agent, and needs remediation on the agent OS. Investigate the OpsMgr event log on the agent for Warning/Critical events – to see if a lot of workflows are failing due to this issue.

The FIRST thing I do – is to see if WMI is responsive. I run WBEMTEST, and connect to “root\cimv2”. I then hit “query” and execute a “select * from win32_operatingsystem” to see if it returns results, or an error. Next – I look at the namespace from the alert in SCOM…. perhaps it is “root\MicrosoftDNS”, or “root\CCM”. Then – I try and run the query that is failing from the alert.

If EITHER of the above connections/queries fail…. then I know what's wrong. WMI has a core issue, and I punt this to my platform or application team to fix it. Sometimes it needs a MOF recompile, sometimes it needs WMI service bounced or the OS bounced.

If these all appear to work correctly, or, the problem is resolved after a WMI service bounce, then re-appears later – check out the following:

There are many things you can do to resolve/remediate these issues. Here is a list of the most common fixes:

1. Apply http://support.microsoft.com/kb/933061 This resolves a LOT of issues on the Windows 2003 OS with WMI. This should be one of your first steps. This applies to x86 or x64 Windows Server 2003 SP1 or SP2.

2. Registry modification for WMI buffer thresholds (see below)

“HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\Low Threshold On Events (B)" to 35000000 (default is 10000000)
”HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\High Threshold On Events (B)" to 70000000 (default is 20000000)

The registry modification to WMI buffers increases the amount of objects that WMI can hold before injecting sleep delays to the WMI service.

3. Apply http://support.microsoft.com/kb/955360 This updates the Windows Scripting Host (cscript) to version 5.7. This resolves script timeouts, and scripts consuming a LOT of CPU during execution, and problems with multiple scripts running at the same time. This applies to x86 or x64 Windows Server 2003 SP1 or SP2. This is a very good hotfix for DNS servers, DHCP servers, and Domain Controllers. This has been seen to lessen the impact of VBscripts consuming a large amount of CPU during runtime.

Making these three modifications should resolve the majority of systemic issues out there, unless WMI is completely corrupt/unresponsive and needs repair. Sometimes, rebooting a server, or bouncing WMI will temporarily resolve these issues as well, if you cannot apply the fixes immediately.

If you have applied all three of these above, and are still experiencing a systemic repeat of a WMI query/script failure…. the next step would be to try running the query directly, accessing the namespace in WBEMtest. I’d like to hear about any experiences here.

Health Service and MonitoringHost thresholds in R2 – how this has changed and what you should know

kevinhol — Tue, 23 Jun 2009 01:52:29 GMT

In THIS post – I described the way the agent HealthService will bounce on a regular basis, and how to alert on that, and change the thresholds. Please see that post for details on SP1.

Now – in R2 – much of this has changed in how it works. That said – the core challenge in SP1 still exists in R2:

1. The default agent threshold is 100MB for the HealthService and MonitoringHost process. That is too low for many of the typical agents in production environments.

2. When we bounce the agent for using more than 100MB, we do this silently, and do not alert. If your agents are constantly restarting in a loop, you will never know.

Lets take a look at R2:

First off – lets examine the HealthService class. There are two Monitors located at Health Service > Entity Health > Performance > Health Service Performance > Health Service State. They are for Handle Count threshold, and Private Bytes Threshold.

Health Service Handle Count Threshold: The default threshold is 2000 handles for an agent. There are built-in overrides for the Management servers – bumping this number up to 10,000 handles. Also, the new Native Exchange 2007 MP, bumps the threshold up to 5000, for Exchange 2007 computers. It is common that you *might* have to bump up this threshold for SOME agents. It is also common to bump the Management Server threshold up to 20,000 or even 50,000 if yours is constantly using more, but stable.

Health Service Private Bytes Threshold: The default is 100MB for all agents. There is a threshold override for Management servers to use up to 1.6GB. The new Native Exchange 2007 MP bumps the threshold to 600MB for Exchange 2007 servers. This is the monitor that needs the most attention! The default of 100MB is not enough for many server roles, especially if hosted on Server 2008 OS. You will likely need to override this monitor, for groups of Windows Computer objects, that are affected. Your agents will potentially be a in perpetual restart loop until this is done. Here is an example of mine – with a few overrides in place for SQL computers and DNS servers:

Now…. in R2 – the MonitoringHost workflows have changed, from rules, to Monitors. These are located under the “Agent” class. You will find them under Agent > Entity Health > Performance > Health Service Performance > Health Service State. They are named Monitoring Host Handle Count Threshold, and Monitoring Host Private Bytes Threshold.

***Note: In this view - you will also see the HealthService monitors, but note – these are inherited from the Health Service class. This is because there is a dependency rollup that rolls up the Health Service State of the HealthService, to the Agent. I will explain why in a moment.

Monitoring Host Handle Count Threshold: The default threshold is 2000 handles for an agent. There are built-in overrides for the Management servers – bumping this number up to 10,000 handles. Also, the new Native Exchange 2007 MP, bumps the threshold up to 5000, for Exchange 2007 computers. It is common that you *might* have to bump up this threshold for SOME agents. It is also common to bump the Management Server threshold up to 20,000 or even 50,000 if yours is constantly using more, but stable.

Monitoring Host Private Bytes Threshold: The default is 100MB for all agents. There is a threshold override for Management servers to use up to 1.6GB. The new Native Exchange 2007 MP bumps the threshold to 600MB for Exchange 2007 servers.

Now, in R2 – all four of these monitors roll their health state up to the Aggregate Roll-up monitor under the agent class, named “Health Service State”.

If we look at the properties of that aggregate monitor, we can see the recovery action to restart the HealthService is now on this monitor. Therefore – if ANY of the 4 monitors below it are in a critical state – they will roll up to this monitor, which will launch a script to bounce the HealthService on the agents.

This script, when it executes, will launch an event 6024 in the OpsMgr event log on the agent, that is is restarting the HealthService.

***NOTE – the text used in the event log is not technically accurate, in that it always states “Health Service exceeded Process\Handle Count or Private Bytes threshold.” It could be an issue with the Monitoring Host – NOT the HealthService, and this event might mislead you in troubleshooting. So just know that a 6024 event is a generic restart event – you need to look at the individual monitor state change history in Health Explorer to properly investigate.

So – to summarize the changes from SP1 to R2:

The MonitoringHost threshold rules are now standard monitors.
The Health Service monitors roll up to Agent - Health Service State rollup monitor.
The Health Service State rollup monitor has a recovery which runs a script to bounce the HealthService when it is in a critical state.
We still do not alert by default when the script bounces your agent, and you need to create a rule to look for this, or, alert on the state-change of the monitor.
You still will likely need to adjust the threshold of the Health Service Private Bytes monitor for many of your agents.

Lets talk about #4 above: You need to know when your agents are getting bounced, especially if they are caught in a loop of bouncing.

You have a few choices here…. but I like to either:

Create an alert rule, target “Agent”, Event ID 6024, in the OpsMgr event log.
Override the “Health Service State” rollup monitor, to “Generates Alert = True”.

Either one of those will give you a solution, that will detect the monitor state change, which results in bouncing the agent’s Health Service.

With regard to #5 above: You will likely need to adjust this default threshold for many agents. From my previous blog post on this topic – I have been seeing that mostly on the following types of servers:

Large SQL database servers
Server 2008 domain controllers
DHCP servers with large scope counts
DNS servers with large zone counts
Exchange 2007 servers
Large Exchange 2003 Mailbox servers
IIS7 (Server 2008) Servers
Proxy agents that perform special agent-less monitoring (Nworks/Vmware, etc…)

Create groups for these server types, and override the default threshold for this monitor for those groups. In general, I have found bumping to 250MB resolves most agent issues, but some special cases could need much more.

R2 – Improved Agent Proxy Alerts

kevinhol — Wed, 08 Apr 2009 00:46:17 GMT

Here is a nice add in R2: When we give you the old “agent proxy alert”, we now tell you the name of the Agent that needs agent proxy enabled, and resolve the name of the object type that it was bringing in:

Nice improvement. I enable agent proxy for SQL1CLN2 and get on with my day.

Are your agents restarting every 10 minutes? Are you sure?

kevinhol — Thu, 26 Mar 2009 21:50:00 GMT

**Updated 6-22-2009 – This post applies to SP1 ONLY!!! This architecture has changed for R2. The version of this article updated for R2 is located here: http://blogs.technet.com/kevinholman/archive/2009/06/22/health-service-and-monitoringhost-thresholds-in-r2-how-this-has-changed-and-what-you-should-know.aspx

Here is something I have been seeing with more and more customers…. and I think everyone should take a look and consider this.

They have a decent percentage of agents, that the HealthService is being programmatically restarted every 10-12 minutes (sometimes less often than 12 minutes, but still very frequent.

This is being caused by one of a few workflows. By default, there is a monitor that watches the HealthService resources, and runs a script to bounce the Health Service anytime that service is consuming too many resources. This is good – because we don't want an issue with a SCOM agent to ever impact the available resources on a monitored agent.

The bad? Well, we don't (by default) alert when the script is called that restarts the HealthService. This means – this can be affecting you – and you really have no way of knowing this out of the box. Also – sometimes the restart fails…. and this leaves you with a handful of agents that generate a Heartbeat failure. When you look at the agent – it is fine, just the HealthService isn't running…. you start it, and everything goes back to normal… or so you think. If every day you have to respond to a few Heartbeat failures…. and you find the OS up – just the agent stopped… this might be the cause.

Here are the two monitors which target “Health Service” – and can restart the service:

Here are the two rules, targeting “Agent”:

Most often – I see it is the monitor causing the restart… on the Health Service Private bytes, but it could be any of the above – and you should consider any of these if you are impacted by this.

Note the default overrides. We allow the management servers to use up to 1.6GB of HealthService privatebytes, and if you have the current Exchange 2007 MP – we have an override which allows Exchange 2007 agents to use up to 600MB, up from the default which is 100MB.

The Exchange 2007 MP was updated with this override, because this issue was already detected for large Exchange servers.

The problem is – that other large servers can potentially use more than 100MB.

I have been seeing this mostly on:

1. Large SQL database servers

2. Server 2008 domain controllers

3. DHCP servers with large scope counts

4. Exchange 2007 servers

5. Large Exchange 2003 Mailbox servers

6. IIS7 (Server 2008) Servers

Here is what ALL customers should implement… A rule that watches for the event, when the restart script is called.

Here is the event that gets created, in the OpsMgr event log on the agent, when this script tries to restart the HealthService:

Event Type:         Warning
Event Source:      Health Service Script
Event Category:   None
Event ID:             6024
Date:                    3/26/2009
Time:                   9:22:33 AM
User:                    N/A
Computer:            DC01
Description:
LaunchRestartHealthService.js : Launching Restart Health Service. Health Service exceeded Process\Handle Count or Private Bytes threshhold.

Here is the event created, when it is a problem with the MonitoringHost process:

Event Type:          Warning
Event Source:       Health Service Script
Event Category:    None
Event ID:              6025
Date:                     4/21/2009
Time:                     5:23:41 AM
User:                      N/A
Computer:             EX2CLN1
Description:
LaunchRestartHealthService.js : Launching Restart Health Service. Monitoring Host exceeded Process\Handle Count threshhold.

Event Type:          Warning
Event Source:       Health Service Script
Event Category:    None
Event ID:              6026
Date:                    3/26/2009
Time:                   10:14:30 AM
User:                    N/A
Computer:            DC01
Description:
LaunchRestartHealthService.js : Launching Restart Health Service. Monitoring Host exceeded Process\Private Bytes threshhold.

Here is the event after the restart is noted as a success:

Event Type:            Information
Event Source:         Health Service Script
Event Category:     None
Event ID:                6062
Date:                      3/26/2009
Time:                      10:35:30 AM
User:                       N/A
Computer:              DC01
Description:
RestartHealthService.js : Restarting Health Service. Service successfully restarted.

Here is the event after the restart failed for some reason:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          5/18/2009 11:58:28 AM
Event ID:      6061
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SQL3CLN1.opsmgr.net
Description:
RestartHealthService.js : Restarting Health Service. Failed to restart service.

So – in order to see if you are impacted by this – the simplest thing to do – is to create a custom rule – that alerts when these events happen.

Create a new rule – target “Windows Server Operating System”, (or whatever is your standard). Look in the OpsMgr event log, with an expression of “Event ID Equals 6024” OR “Event ID Equals 6025” OR “Event ID Equals 6026” OR “Event ID Equals 6062” OR “Event ID Equals 6061”. You could also just as easily write individual rules – once for each… and use a different name for the alert.

Once this is created – you will get an alert for any agent that is restarting, or failed to restart. This will tell you if you need to bump up the default values, for specific agents, or a group of agents, such as SQL servers, domain controllers, etc…. What I have found, is that bumping this number up to 250MB will generally address most agent’s issues…. but you need to monitor this in your environment and see how much memory your HealthService.exe and/or MonitoringHost.exe process needs.

Next – create a view in the Monitoring Console – just for these alerts. New view – Alert View – for all alerts with a given Name. Use something that matches your alert name. Here is mine:

Now – watch this view for any alerts that come in….

What we see is – that I have many agents restarting from time to time… due to the default threshold not being enough. The best way to monitor this… is to watch and see which agents are affected, and then bump up their threshold via overrides. I would recommend bumping the number up in small increments, like 100MB at a time, and see where your “happy place” is.

This article specifically addresses OpsMgr SP1. I am not sure yet if this is going to need to be done in R2, so I will update this article when R2 releases.

As a side note – one of the symptoms I see, is when the OpsDB StateChangeEvent table is one of the largest tables. This is caused – because the constant restart of the agent, causes state to be recalculated for every monitor on the agent. It sends this recalculated state on every restart of the agent, flooding the database with state data.

I’d recommend putting these rules in place to alert on this event, for ANY customer…. knowing is power.

As far as some recommended values…. the best thing is to find your own “happy place”… but as of this writing, I start with 250MB as an initial adjustment. You can create a group of Windows Computer objects that are affected, and simply add computers to this group that seems to need more privatebytes. Use this custom group for your override on the HealthService and/or Monitoring host workflows.

Recommendation examples:

I don't recommend overriding the Health Service Private Bytes Threshold Monitor “for all objects of type: Health Service”. I have seen this impact the Health Service on the management servers – even though there is an override for the management servers which should be more specific in a conflict case – but this doesn't always work in the field.

You can use groups of Windows Computer objects (Or groups of Health Service Instances if you so desire) for this override.

If we wanted to override this value – and increase it to 200MB for all agents, override for a group – and use the “Agent Managed Computer Group”:

You have to be VERY careful when doing the above – because this override will potentially conflict with other group overrides… if you want specific overrides for Exchange, DHCP, DNS, SQL, etc…

A better approach is like so:

Just make SURE that whatever groups you use – they don't have/share any of the same computers… or you can get some conflicting values here.

I am attaching a management pack below. This MP contains a simple rule as described about to alert on the HealthService restart, and contains a view which are scoped only to these alerts. You should keep an eye on these and take action on them. I added alert suppression on these rules – they will create a single alert for each identical computer and event ID, and increment repeat counts on the worst offenders. This is one warning alert you should NOT ignore.

The Cluster Service will automatically restart itself

kevinhol — Fri, 20 Mar 2009 17:02:47 GMT

Something I ran across with a customer.

There aren’t many situations where service recoveries run automatically in Microsoft MP’s, but this is one case where they do. The cluster service running is critical to a healthy cluster. In the current cluster MP, the service monitor for the cluster service will automatically start the cluster service on a node, if it detects it stops. There is a recovery action on the monitor to do just that.

As always – if you don't like this intended behavior – you can override just the recovery, and disable it.

Why do you need to know this?

Because – some service packs for clustered applications, require you to stop the cluster service, in order to apply. If you stop this service on a node while doing application maintenance, SCOM will restart it, almost immediately. The correct solution – is to use Maintenance Mode in SCOM, which will unload the monitors, and hence, any automatic recoveries will no long run. So…. make SURE you are effectively using maintenance mode if you ever need to stop your cluster service, or, disable this automatic recovery action.

Applying an OpsMgr hotfix to a RMS Cluster node? Some things to be aware of.

kevinhol — Wed, 25 Feb 2009 05:20:09 GMT

When you apply a SCOM hotfix to a RMS cluster, you need to be aware of some issues, and some workarounds. This is something I have seen several times in the field…

On any server/agent, the Hotfix installer will stop any discovered OpsMgr services, including the SDK, Config, and HealthService. This part is normal. It does this in order to update the files (DLL’s) that are part of the hotfix payload, and then it will start the services again when complete. This all works well, except for on RMS clusters.

The reason for this, is that the Hotfix installer is not 100% cluster aware.

In a RMS cluster… the passive node will have these three services stopped, and the services will be set to Manual Startup. On the active node – the OpsMgr services are also set to Manual Startup, but the services are running, because the Cluster service controls these services now. This is how a clustered service works, and we should not ever stop a clustered service in Service Control Manager, we really should take the resource offline, in Cluster Admin.

So I have two options… I can apply the hotfix to the Active Node… or the Passive node.

If I choose the active node – the hotfix installer will try and stop all the OpsMgr services, and this will cause the Cluster service to try and restart them, or eventually fail them over to the passive node – depending on your Cluster configuration settings. Therefore – it is probably best to patch the passive node first… ensure the hotfix applied correctly, and then move the cluster group and OpsMgr RMS group over to the freshly hotfixed node… and go patch the other one (now passive)

This works – but is not 100% smooth. When we apply the hotfix to the passive node, the hotfix installer will try and start the services at the end of the process, even though they were not running previously. We do NOT want these services trying to run on the passive node – since it does not own the cluster disk resources…. so the services will start, but cannot do anything but log errors.

You will also see an error from the HealthService – not being able to start. It is apparent that this service fails because it cannot access the disk resource, but the SDK and config services WILL start.

What is worse – is that the hotfix installer – changes the config of the service startup types to Automatic – which means these services will continue to try and run on the passive node across reboots.

So – the guidance I have, for RMS clusters – is:

Patch the passive node (we will call this Node 2)
Click ok on the HealthService start failure error.
Ensure the hotfix applied by inspecting the DLL(s) versions as documented in the KB.
Stop the running SDK and Config services on the passive node.
Set any OpsMgr services that were changed to Automatic – BACK to Manual.
Move the cluster resource groups over to the freshly patched Node 2.
On Node 1 (now passive) apply the hotfix, and repeat steps starting at Step 2 above.

NOTE: This is only applicable to OpsMgr specific hotfixes. For OS hotfixes – you would follow your standard clustered OS hotfix routine.

Getting and keeping the SCOM agent on a Domain Controller – how do YOU do it?

kevinhol — Fri, 20 Feb 2009 21:27:57 GMT

I’d like to hear some community feedback on this….

In OpsMgr – deploying a SCOM agent to a DC often presents companies with a bit of a challenge. The reason is – in order to install software to a DC and manage it – we need rights on the DC to accomplish this. These rights are needed, anytime we are going to deploy an agent, hotfix an agent, or run a repair on a broken agent to keep the agent healthy.

When we push agents from the console, the default account used to perform the push is the Management Server Action Account. If this account does not have Domain Admin rights – the push will fail to a DC, with an Access Denied. We do allow the option to type in temporary (encrypted) credentials, which are used to deploy the agent, one time, and then are discarded. See the image below:

Here is a list of the most common options I have observed, in place at customer sites… and potential custom options that can be developed. I’d be interested in any community feedback on any options you are using, that I dont cover or haven't seen before.

1. Grant the Management Server Action account Domain Admin or Builtin\Administrators.

a. Not recommended as a best practice, this gives rights to the MSAA that are not required for day to day activities.

b. Con - SCOM Admins now control a domain admin account.

2. Grant a SCOM Administrator a special domain account, for this purpose, that is a domain admin.

a. This allows us to track the actions of that SCOM admin, when he/she uses that special privileged account.

b. That SCOM admin will be able to do repairs, hotfixes, and deployments for DC’s.

c. Con – Domain Admin teams often wont delegate these rights as they are tightly controlled.

3. The SCOM admin team delegates console based agent management to a Domain Administrator for DC agent health.

a. The domain admin must become a SCOM Admin, and therefore could potentially hurt the SCOM environment.

b. Pro – the admins in charge of the DC’s now have full responsibility to keep the agents healthy.

c. Con – the Domain Admins might not understand components of SCOM, and create something that impacts the monitoring environment.

4. The SCOM admin team must partner with the Domain Admin team, and have the Domain Administrator type in his credentials any time the SCOM administrator needs to deploy/hotfix/repair an agent on a domain controller.

a. This is a bit more labor intensive… because the SCOM admin must wait for a domain admin to be available to work on DC agents, but tight security boundaries are maintained.

5. All DC based agents will be manually installed/updated/repaired.

a. This is very common, when the two teams do not trust each other. The Domain Admin team is now required to manually deploy agents to domain controllers, and keep them up to date, and healthy.

6. Use a software deployment tool already in place to deploy/update/repair agents.

a. If a software deployment tool is already in place on DC’s, like SMS/SCCM, you can create packages to deploy, hotfix, and repair agents, similar to your patching of the OS today.

7. Customized solution: Create a Run-As account that is a domain admin, one time, for use in agent deployment/repair.

a. This involves the domain admin typing in credentials ONCE, into a RUN-AS account, which is stored securely and encrypted in the SCOM database.

b. This run-as account can be associated with a run-as profile, which is used by a custom task, which will remotely deploy the agent to the domain controller. This task will execute under the security context of the privileged run-as account.

c. The benefit is that the domain admin gets to control the password for this account, the SCOM admin does not need to know the account credentials.

d. The downside, is that this run-as account could potentially be leveraged by some other workflow, if a SCOM admin intentionally misused it…. Similar to solution #2 above.

e. This is just an idea I had – curious if anyone has already developed a solution like this?

Console based Agent Deployment Troubleshooting table

kevinhol — Tue, 27 Jan 2009 20:35:15 GMT

This post is a list of common agent push deployment errors… and some possible remediation options.

Most common errors while pushing an agent:

Error	Error Code(s)	Remediation Steps
The MOM Server could not execute WMI Query "Select * from Win32_Environment where NAME='PROCESSOR_ARCHITECTURE'" on computer server.domain.com Operation: Agent Install Install account: domain\account Error Code: 80004005 Error Description: Unspecified error	80004005	1. Check the PATH environment variable. If the PATH statement is very long, due to lots of installed third party software - this can fail. Reduce the path by converting any long filename destinations to 8.3, and remove any path statements that are not necessary. Or apply hotfix: http://support.microsoft.com/?id=969572 2. The cause could be corrupted Performance Counters on the target Agent. To rebuild all Performance counters including extensible and third party counters in Windows Server 2003, type the following commands at a command prompt. Press ENTER after each command. cd \windows\system32 lodctr /R Note /R is uppercase. Windows Server 2003 rebuilds all the counters because it reads all the .ini files in the C:\Windows\inf\009 folder for the English operating system. How to manually rebuild Performance Counter Library values http://support.microsoft.com/kb/300956 3. Manual agent install.
The MOM Server could not execute WMI Query "Select * from Win32_OperatingSystem" on computer “servername.domain.com” Operation: Agent Install Install account: DOMAIN\account Error Code: 800706BA Error Description: The RPC server is unavailable. The MOM Server could not execute WMI Query "(null)” on computer “servername.domain.com” Operation: Agent Install Install account: DOMAIN\account Error Code: 800706BA Error Description: The RPC server is unavailable.	8004100A 800706BA	1. Ensure agent push account has local admin rights 2. Firewall is blocking NetBIOS access. If Windows 2008 firewall is enabled, ensure “Remote Administration (RPC)” rule is enabled/allowed. We need port 135 (RPC) and the DCOM port range opened for console push through a firewall. 3. Inspect WMI service, health, and rebuild repository if necessary 4. Firewall is blocking ICMP (Live OneCare) 5. DNS incorrect
The MOM Server failed to open service control manager on computer "servername.domain.com". Access is Denied Operation: Agent Install Install account: DomainName\User Account Error Code: 80070005 Error Description: Access is denied.	80070005 80041002	1. Verify SCOM agent push account is in Local Administrators group on target computer. 2. On Domain controllers will have to work with AD team to install agent manually if agent push account is not a domain admin. 3. Disable McAfee antivirus during push
The MOM Server failed to open service control manager on computer "servername.domain.com". Therefore, the MOM Server cannot complete configuration of agent on the computer. Operation: Agent Install Install account: DOMAIN\account Error Code: 800706BA Error Description: The RPC server is unavailable.	800706BA	1. Firewall blocking NetBIOS ports 2. DNS resolution issue. Make sure the agent can ping the MS by NetBIOS and FQDN. Make sure the MS can ping the agent by NetBIOS and FQDN 3. Firewall blocking ICMP 4. RPC services stopped.
The MOM Server failed to acquire lock to remote computer servername.domain.com. This means there is already an agent management operation proceeding on this computer, please retry the Push Agent operation after some time. Operation: Agent Install Install account: DOMAIN\account Error Code: 80072971 Error description: Unknown error 0x80072971	80072971	This problem occurs if the LockFileTime.txt file is located in the following folder on the remote computer: %windir%\422C3AB1-32E0-4411-BF66-A84FEEFCC8E2 When you install or remove a management agent, the Operations Manager 2007 management server copies temporary files to the remote computer. One of these files is named LockFileTime.txt. This lock file is intended to prevent another management server from performing a management agent installation at the same time as the current installation. If the management agent installation is unsuccessful and if the management server loses connectivity with the remote computer, the temporary files may not be removed. Therefore, the LockFileTime.txt may remain in the folder on the remote computer. When the management server next tries to perform an agent installation, the management server detects the lock file. Therefore, the management agent installation is unsuccessful. http://support.microsoft.com/kb/934760/en-us
The MOM Server detected that the following services on computer "(null);NetLogon" are not running. These services are required for push agent installation. To complete this operation, either start the required services on the computer or install the MOM agent manually by using MOMAgent.msi located on the product CD. Operation: Agent Install Remote Computer Name: servername.domain.com Install account: DOMAIN\account Error Code: C000296E Error Description: Unknown error 0xC000296E	C000296E	1. Netlogon service is not running. It must be set to auto/started
The MOM Server detected that the following services on computer "winmgmt;(null)" are not running	C000296E	1. WMI services not running or WMI corrupt
The MOM Server detected that the Windows Installer service (MSIServer) is disabled on computer "servername.domain.com". This service is required for push agent installation. To complete this operation on the computer, either set the MSIServer startup type to "Manual" or "Automatic", or install the MOM agent manually by using MOMAgent.msi located on the product CD. Operation: Agent Install Install account: DOMAIN\account Error Code: C0002976 Error Description: Unknown error 0xC0002976	C0002976	1. Windows Installer service is not running or set to disabled – set this to manual or auto and start it.
The Agent Management Operation Agent Install failed for remote computer servername.domain.com. Install account: DOMAIN\account Error Code: 80070643 Error Description: Fatal error during installation. Microsoft Installer Error Description: For more information, see Windows Installer log file "C:\Program Files\System Center Operations Manager 2007\AgentManagement\AgentLogs\servernameAgentInstall.LOG C:\Program Files\System Center Operations Manager 2007\AgentManagement\AgentLogs\servernameMOMAgentMgmt.log" on the Management Server.	80070643	1. Enable the automatic Updates service…. Install the agent – then disable the auto-updates service if desired.
Call was canceled by the message filter	80010002	Install latest SP and retry. One server that failed did not have Service pack installed
The MOM Server could not find directory \\I.P.\C$\WINDOWS\. Agent will not be installed on computer "name". Please verify the required share exists.	80070006	1. Manual agent install Possible locking on registry? http://www.sysadmintales.com/category/operations-manager/ Try manual install. Verified share does not exist.
The network path was not found.	80070035	1. Manual agent install
The Agent Management Operation Agent Install failed for remote computer "name". There is not enough space on the disk.	80070070	1. Free space on install disk
The MOM Server failed to perform specified operation on computer "name". The semaphore timeout period has expired.	80070079	NSlookup failed on server. Possible DNS resolution issue. Try adding dnsname to dnssuffix search list.
The MOM Server could not start the MOMAgentInstaller service on computer "name" in the time.	8007041D 80070102	NSlookup failed on server. Possible DNS resolution issue. Verify domain is in suffix search list on management servers.
The Agent Management Operation Agent Install failed for remote computer "name"	80070643	1. Ensure automatic updates service is started 2. Rebuild WMI repository 3. DNS resolution issue
The Agent Management Operation Agent Install failed for remote computer "name". Another installation is already in progress.	80070652	Verify not in pending management. If yes, remove and then attempt installation again.
The MOM Server detected that computer "name" has an unsupported operating system or service pack version	80072977	Install latest SP and verify you are installing to Windows system.
Not discovered		Agent machine is not a member of domain
Ping fails		1. Server is down 2. Server is blocked by firewall 3. DNS resolving to wrong IP.
Fail to resolve machine		1. DNS issue
The MOM Server failed to perform specified operation on computer "name". Not enough server storage…	8007046A	1. This is typically a memory error caused by the remote OS that the agent is being installed on.
There are currently no logon servers available to service the logon request.	8007051F	1. Possible DNS issue
This installation package cannot be installed by the Windows Installer service. You must install a Windows service pack that contains a newer version of the Windows Installer service.	8007064D	1. Install Windows Installer 3.1
The network address is invalid	800706AB	Possible DNS name resolution issue. Tried nslookup on server name and did not get response. Verify domain is in suffix search list on management servers.
The MOM Server failed to perform specified operation on computer servername.domain.com	80070040	1. Ensure agent push account has local admin rights
The MOM Server detected that the actual NetBIOS name SERVERNAME is not same as the given NetBIOS name provide for remote computer SERVERNAME.domain.com.	80072979	1. Correct DNS/WINS issue. 2. Try pushing to NetBIOS name

Agent Pending Actions can get out of synch between the Console, and the database

kevinhol — Mon, 29 Sep 2008 21:57:59 GMT

When you look at your agent pending actions in the Administration pane of the console.... you will see pending actions for things like approving a manual agent install, agent installation in progress, approving agent updates, like from a hotfix, etc.

This pending action information is also contained in the SQL table in the OpsDB - agentpendingaction

It is possible for the agentpendingaction table to get out of synch with the console, for instance, if the server was in the middle of updating/installing an agent - and the management server Healthservice process crashed or was killed.

In this case, you might have a lingering pending action, that blocks you from doing something in the future. For instance - if you had a pending action to install an agent, that did not show up in the pending actions view of the console. What might happen, is that when you attempt to discover and push the agent to this same server, you get an error message:

"One or more computers you are trying to manage are already in the process of being managed. Please resolve these issues via the Pending Management view in Administration, prior to attempting to manage them again"

The problem is - they don't show up in this view!

To view the database information on pending actions:

select * from agentpendingaction

You should be able to find your pending action there - that does not show up in the Pending Action view in the console, if you are affected by this.

To resolve - we should first try and reject these "ghost" pending actions via the SDK... using powershell. Open a command shell, and run the following:

get-agentpendingaction

To see a prettier view:

get-agentpendingaction | ft agentname,agentpendingactiontype

To see a specific pending action for a specific agent:

get-agentPendingAction | where {$_.AgentName -eq "servername.domain.com"}

To reject the specific pending action:

get-agentPendingAction | where {$_.AgentName -eq "servername.domain.com"}|Reject-agentPendingAction

We can use the last line - to reject the specific pending action we are interested in.

You might get an exception running this:

Reject-AgentPendingAction : Microsoft.EnterpriseManagement.Common.UnknownServiceE
xception: The service threw an unknown exception. See inner exception for details
. ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:
Exception of type 'Microsoft.EnterpriseManagement.Common.DataItemDoesNotExistExc
eption' was thrown.

If this fails, such as gives an exception, or if our problem pending action doesn't even show up in Powershell.... we have to drop down to the SQL database level. This is a LAST resort and NOT SUPPORTED.... run at your own risk.

There is a stored procedure to delete pending actions.... here is an example, to run in a SQL query window:

exec p_AgentPendingActionDeleteByAgentName 'agentname.domain.com'

Change 'agentname.domain.com' to the agent name that is showing up in the SQL table, but not in the console view.

Helper Objects are not copied to gateways

kevinhol — Fri, 11 Jul 2008 23:47:00 GMT

Something I noticed today..... when you deploy a gateway server - the helper object oomads.msi was not copied to the local \AgementManagement directory for agent push.

This means, that if you have a DC in the untrusted forest, managed by a gateway, that Oomads will not get copied or installed automatically. You will need to manually copy and install Oomands on any DC's you will monitor in the untrusted forest.

A report to show all agents missing a specific hotfix

kevinhol — Sat, 28 Jun 2008 01:11:00 GMT

This is a continuation of my previous post on determining which agents are missing a hot-fix:

How do I know which hotfixes have been applied to which agents-

I wrote up a report that allows you to paste in a KB article number into the report as a parameter, and then it will show all agents that are potentially missing that hotfix. This will help you easily find agent which need to be patched and got missed for some reason.

You can run this report if you create the SQL reporting data source as specified in my previous post:

Creating a new data source for reporting against the Operational Database

Once imported - it will show up in the console. Open the report, and paste in any KB article number for a OpsMgr hotfix you have applied. The number MUST begin and end with "%".... such as %951380% as shown:

The report is attached below: