Kevin Holman's OpsMgr Blog

29106 event on RMS – Index was out of range. Wait. What?

kevinhol — Tue, 10 Nov 2009 20:01:36 GMT

Was working with a customer on this one – figured it might help others.

Saw a lot of these VERY SPECIFIC 29106 events on the RMS, specifically with the text:

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

Here is the full event:

Event Type:      Warning
Event Source:    OpsMgr Config Service
Event Category: None
Event ID:        29106
Date:            11/10/2009
Time:            12:43:24 PM
User:            N/A
Computer:        AGENTNAME
Description:
The request to synchronize state for OpsMgr Health Service identified by "3688d65d-a16c-2be6-7e84-5faf8a9cffe0" failed due to the following exception "System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index

What we found was – that we could look up these health service ID’s – by pasting them in the following SQL query:

select * from MTV_HealthService
where BaseManagedEntityId = '3688d65d-a16c-2be6-7e84-5faf8a9cffe0'

This would give us the name of the agent.

In the console, under Agent Managed – we found all of these agents were in “Unmonitored” state – on the agents themselves, they were stuck. They looked like they got installed, but could not get config. We deleted them from agent managed, waited a few minutes, and let them show back up in Pending Management. Approved them – then they were able to come back in and work properly. These looked for the most part like orphaned machines, and several were computers that were renamed, or old DC’s that were demoted.

How to force the Web Console to open a specific view, instead of the default Monitoring Overview

kevinhol — Mon, 09 Nov 2009 23:34:11 GMT

When you open the Web Console – the default view will open to a “Monitoring Overview” pane. This view, in very large environments, can take considerable time to finish loading, before you can select any other views. Sometimes, loading this view may time out as well.

Here is a way to load a specific view by default. This helps when you have customers that only need to see a very specific pre-configured alert or state view.

The format is:

http://webconsoleserver:51908/default.aspx?ViewID=8DB1F5A7-F3F3-2646-6C6B-E34672F7ED98&ViewType=AlertView

Lets break that all down:

The first part of the URL is a constant – this should be self explanatory:

http://webconsoleserver:51908/default.aspx?ViewID=

The next part is an ID for the view. These will be constant for default built in views and views from Microsoft MP’s. Your custom MP’s will have their own unique view ID’s. I will talk more about how to find these ID’s below. My example ID is for the “Active Alerts” view at the top of the console view list.

8DB1F5A7-F3F3-2646-6C6B-E34672F7ED98

The last part is the ViewType. This describes to the console if we are dealing with an AlertView, StateView, or PerformanceView

&ViewType=AlertView

Here is a SQL query – to get all the view ID’s from the OperationsManager (OpsDB) for any view:

select vv.id as 'View Id',
vv.displayname as 'View DisplayName',
vv.name as 'View Name',
vtv.DisplayName as 'ViewType',
mpv.FriendlyName as 'MP Name'
from ViewsView vv
inner join managementpackview mpv on mpv.id = vv.managementpackid
inner join viewtypeview vtv on vtv.id = vv.monitoringviewtypeid
--where mpv.FriendlyName like '%default%'
--where vv.displayname like '%Service%'
order by mpv.FriendlyName, vv.displayname

Here are some examples of some built in views:

“Active Alerts” view at the top:

http://webconsoleserver:51908/default.aspx?ViewID=8DB1F5A7-F3F3-2646-6C6B-E34672F7ED98&ViewType=AlertView

“Windows Computers” state view:

http://webconsoleserver:51908/default.aspx?ViewID=E3D720DE-F6DD-185C-6FDC-0832377D910A&ViewType=StateView

“Operating System Performance” view from the BaseOS (Microsoft Windows Server) MP:

http://webconsoleserver:51908/default.aspx?ViewID=9B216021-6E88-EF6D-2A97-9E3EA1D6AD3B&ViewType=PerformanceView

Now – you can give a specific use this URL for their favorites – if they want to open the Web Console on this specific view.

OpsMgr 2007 SP1 cumulative rollup hotfix has shipped!

kevinhol — Mon, 09 Nov 2009 19:06:09 GMT

If you cannot or will not upgrade to OpsMgr 2007 R2 anytime soon – then this hotfix is for you!

Available at: http://support.microsoft.com/kb/971541

This updates OpsMgr 2007 SP1 to 6.0.6278.100. This is a rollup covering many new issues, plus most of the previously released critical hotfixes for OpsMgr. I recommend this rollup hotfix for anyone running OpsMgr 2007 SP1 that doesn't have very near term plans in place to upgrade to OpsMgr 2007 R2.

Overview

The Update Rollup for Operations Manager 2007 Service Pack 1 (SP1) combines previous hotfix releases for SP1 with additional fixes and support of SP1 roles on Windows 7 and Windows Server 2008 R2. This update also provides database role and SQL Server Reporting Services upgrade support from SQL Server 2005 to SQL Server 2008.
The Update Rollup includes updates for the following Operations Manager Roles:

Root Management Server, Management Server, Gateway Server
Operations Console
Operations Management Web Console Server
Agent
Audit Collection Server (ACS Server)
Reporting Server

The following tools and updates are provided within this update which may be specific to a scenario:

Support Tools folder – Contains SRSUpgradeTool.exe and SRSUpgradeHelper.msi (Enables upgrade of a SQL Server 2005 Reporting Server used by Operations Manager Reporting to SQL Server 2008 Reporting Server)
Gateway folder – Contains a MSI transform and script to update MOMGateway.MSI for successful installation on Windows Server 2008 R2
ManagementPacks folder – Contains an updated Microsoft.SystemCenter.DataWarehouse.mp which requires manual import

Feature Summary:

Providing a rollup that supersedes nearly all SP1 binary hotfixes in a single package (~50 fixes) . See KB971541 for exceptions.
Support for Windows 7 and Windows Server 2008 R2 - See KB974722 which will be updated to include data around the release of KB971541
Operational and DataWarehouse database support for upgrade to SQL 2008.
Additional stability hotfixes
SCCM Monitoring via our 64-bit agent in the latest SCCM MP. See the latest SCCM MP guide for details.
Exchange 2010 MP support
Fix for Ops console crashes seen on Vista and Windows 7

The Supported Configurations Guide and Upgrade Guide have also been updated.

My experience upgrading a lab management group to the SP1 rollup:

First – I’d recommend making a plan…. like reading the KB article, known issues, and plan out the order of update operations. The KB article dont specifically state a specific order, so I will probably do something like this:

Root Management Server (includes web console)
all secondary Management Servers (includes any that are audit collectors)
SCOM Reporting Server
any stand alone OpsMgr consoles
Agents (both from pending and manually installed)

Ok – getting started…..

COPY the SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-ENU.MSI file locally to the RMS.
Run the update from the MSI – install to default locations
If executing on Windows Server 2008 – run the MSI from an elevated command prompt.
READ the release notes (you can copy these out to word to make them more readable
Note the new splash screen for this hotfix:

We will “Run Server Update”
The hotfix installs with no further user interaction.
The installer will finish – you can click “Finish”. However – another installer will kick off immediately afterward. This is by design – documented in the release notes, and is for installing localization updates. Then click “Finish” on the second update screen.
At this point, you can click “Exit” on the Software Update splash screen.
Continue applying the updates to the different roles – as documented in the release notes.

Some interesting things you might notice:

We actually clean up all the old hotfixes from the agent files – and move them to the root of \AgentManagement folder:

This is good – we wont try and re-apply them to subsequent agent installs/updates. Now – there will be two new hotfix files in the \x86, \AMD64, and \ia64 folders:

These include the agent hotfix update, plus a localization update which will vary based on your localization settings.

As for double-checking the update applied successfully – you can add “File Version” column to windows Explorer:

You will notice several management packs got updated in the console:

***Note – per the release notes and KB article

Import the following management pack from the ManagementPacks folder:

Microsoft.SystemCenter.DataWarehouse.Reports.MP

The location is a bit confusion – the full path would be: \Program Files\System Center 2007 Hotfix Utility\KB971541\ManagementPacks\

You can import this at any time… I’d recommend importing this after you are done updating all the server roles, including reporting.

In the console – you will also note that any agents that were not manually installed – will require an agent update. I would hold off updating any agents until your management group server roles are fully updated, and then only update around 200 agents at a time. This process will cause significant database and management server activity – so I’d advise doing the agent updates during off-peak use hours for large management groups:

Next – I update my management servers, including any that run ACS (have a special ACS update for those too in the Hotfix installer splash screen)

Next up – reporting:

I kick this off on the SRS/SCOM Reporting server. Not prompted for anything… it completes in seconds.

Next on the list – update agents. They all updated just fine via pending actions except for one…. this happens to be the same server that is hosting the OpsDB. Interesting. I got the following error:

The MOM Server failed to perform specified operation on computer DB.opsmgr.net.

Operation: Agent Install

Install account: OPSMGR\localadmin

Error Code: 8007064A

Error Description: The configuration data for this product is corrupt. Contact your support personnel.

This turned out to be due to some bad data in my HKEY_CLASSES_ROOT\Installer\Products\C9A0067E2876122489E4BA987C08CDD2\Patches\Patches REG_MULTI_SZ value. I am not sure how this got messed up – probably due to me testing a bunch of SP1 hotfixes previously and fat fingering a registry edit – so I would not assume this will be a common error. Once I fixed this registry entry – this agent updated just fine as well.

How can I be sure all my agents got updated?

One of the simplest ways – is to look at the “Patchlist” column on a Health Service State view. Create a new State View in “My Workspace”. Target “Health Service”:

On the Display Tab – select only Name, and Patch List:

Voila!

Native Exchange 2007 MP 6.0.6741.0 ships for OpsMgr 2007 SP1 users

kevinhol — Mon, 09 Nov 2009 13:34:43 GMT

If for some reason you cannot or will not upgrade to OpsMgr 2007 R2, then this update is for you!

This Native MP replaces the old “converted” Exchange 2007 MP. If you are running the old conversion MP then I STRONGLY suggest making the effort to transition to this MP.

This MP version 6.0.6741.0 is for SP1 users ONLY! If you are running OpsMgr 2007 R2 – there is an R2-Only version of this MP for you on the catalog.

Some notes:

This MP is essentially designed after the R2-only version – with the R2-only enhancement features removed.

This MP is NOT UPGRADE COMPATIBLE with ANY previously public released Exchange 2007 MP. This means you need to remove your existing Exchange MP before you import this one…. and start a new overrides MP for this one, for tuning. Believe me – it will be worth it. Technically – you can run both MP’s side by side…. here is a blurb from the guide:

This new Exchange Server 2007 Management Pack for Operations Manager 2007 SP1 does not support an upgrade from the previously released, converted Exchange Server 2007 Management Pack for Operations Manager 2007. We recommend that you do not run both management packs in parallel. However, running both management packs in parallel is a supported scenario. If you decide to run both management packs in parallel, before importing this management pack, disable any synthetic transaction in the converted management pack. For more information about migrating custom settings from the previously released, converted management pack to the new, native management pack, see Appendix: Migrating from the Previously-Released Converted Exchange Server 2007 Management Pack.

Here is what's new from the guide:

Reports. This management pack provides a set of reports specific to Exchange 2007. For the list of reports and for more information about the reports, see Appendix: Reports.

· Improved disk monitoring. The management pack improves Exchange disk monitoring by providing support for mount points and by discovering three types of disks. This improvement enables you to establish a disk monitoring standard across all Exchange 2007 servers using fewer overrides. The types of disks discovered are Database (on the Mailbox server role), Log (on the Mailbox server role), and Queue (on the Hub and Edge server roles). For information about how to configure Exchange 2007 Disk Monitoring, see Configure Disk Monitoring. For information about classes in this management pack, see Classes.

· A significant number of rules and monitors that are not actionable or may be noisy are disabled. Note that many of these rules are still in the management pack so that you can enable them if necessary. For a list of rules and monitors that are disabled by default in this management pack, see Appendix: Monitors and Rules Disabled by Default.

· A number of performance collection rules are disabled. For a list of disabled performance collection rules, see Performance Collection Rules Disabled by Default.

· Support for monitoring any number of Exchange organizations using a single Operations Manager 2007 management group. There are no specific requirements imposed by this management pack for monitoring multiple Exchange organizations.

· Full support for clustered configurations using Windows Clustering technology. For more details, see Monitoring Clustered Mailbox Servers and Log Shipping.

· An extensive class model, showing the relevant Exchange 2007 server roles and components, as well as service-centric components, allowing you to measure availability or performance at a granular level. The class model fully supports the Distributed Application Designer, allowing you to create custom distributed applications using the appropriate Exchange 2007 components. The appropriate classes and monitors are declared public, enabling you to extend the monitoring if needed. For hierarchical diagrams of the Exchange 2007 management pack classes, see Classes.

· Improved low-privilege support. This management pack supports installing the agent with the minimum rights required by the Operations Manager agent.

· Improved tasks. The management pack includes a number of tasks that simplify troubleshooting and reduce the amount of time to resolve the alert.

· Improved topology discovery. This management pack introduces a number of improvements to topology discovery.

· There is no central discovery script. Each agent is responsible for discovering its piece of the topology. This ensures that potential permissions and trust issues are minimized. There are no requirements on trusts in order to generate the topology.

· The topology will show only agent-managed Exchange 2007 servers, ensuring that the topology consists only of servers that the Exchange administrator has chosen to monitor.

· Improved synthetic transactions. This management pack improves the support for the Exchange 2007 synthetic transactions in several ways:

· The management pack supports local mail flow transactions. For more information, see the Configuring Mail Flow Synthetic Transactions section.

· The management pack supports running the following Exchange 2007 synthetic transactions:

· Test-Mailflow (http://go.microsoft.com/fwlink/?LinkID=137740) (local mail flows only)

· Test-OwaConnectivity (http://go.microsoft.com/fwlink/?LinkID=137732) (internal only)

· Test-MapiConnectivity (http://go.microsoft.com/fwlink/?LinkID=137739)

· Test-ReplicationHealth (http://go.microsoft.com/fwlink/?LinkID=148797)

· The discovery of Exchange 2007 server roles is disabled by default and minimal Exchange 2007 monitoring is applied. This allows you to discover and monitor your servers gradually, as well as tune the management pack as you bring more agent-managed Exchange 2007 servers into the Operations Manager environment. For more information, see Enable Exchange 2007 Server Role Discovery.

Making groups of logical disks – an example from simple to advanced

kevinhol — Thu, 05 Nov 2009 06:00:37 GMT

I have been seeing this question come up a lot lately – as customers try and create groups of their disks – in order to create overrides for “certain” disks. So – I am creating this post to give some real world examples.

Well – I will start this simply. Say we want to create a group of all logical disks, with the drive letters of C: and D:?

I would start with creating a new group – and adding the “Windows Server 2003 Logical Disk” class. Now – I could just use the parent class of “Logical Disk” instead of the OS specific class if I wanted. The only issue with that is that most monitors targeting a disk – are OS specific – and duplicated three times. So it is best to create specific groups for these – but totally not required.

Ok – so in the Dynamic Members query builder – I click add, and pick a property. Since I know “Device Name” contains the drive letter – this will do nicely. I select device name “Equals” “C:”.

Now – I want to also include D:. There are many way to do – this – and I will go through them. First – I could simply Insert a new line for Windows Server 2003 Logical Disk – and replicate the line I have – adding one for D:

Only one problem – this is an “AND” grouping – I really need this to be an “OR” grouping to include both C: and D: drives. You can switch this grouping the in UI, just right click the word “AND” and change it to an OR grouping:

Voila!

This formula now looks like:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Equals C: ) OR ( Device Name Equals D: ) )

Save your group – then right click it – and choose “View Group Members”. This will ensure we are cooking with gas. It should contain all your Windows 2003 based C: and D: volumes.

So far – so good.

Now – what if I ONLY want C: and D: disks, that are HOSTED by specific Windows Computers? I can do that too! Lets say I want a group – of all the C: and D: logical disks, on servers that begin with the name “SR______”

If you look at the bottom of the list of properties for Logical Disks – you will see (Host=Windows Computer). From here – we can pick any attribute of the Windows Computer class as well to add to our expression – to limit our logical disks in our group to very specific Computers.

Go back to the properties of your group, edit the Dynamic Members, and you can construct something like this:

Which translates to the following formula:

( Object is Windows Server 2003 Logical Disk AND ( Windows Computer.NetBIOS Computer Name Matches wildcard sr* ) AND ( ( Device Name Equals C: ) OR ( Device Name Equals D: ) ) )

Now – I will be honest – getting all the “ands” and “ors” in the right place using the UI is a big pain. It is very easy to screw it up. I like to simplify this to the fewest lines possible – using Regex.

Using Regular Expressions – we can use modifiers to create very advanced expressions. my favorites are ^ which means the beginning of a new line or word, and | which is the “pipe” symbol – which means “or”.

So a simple way to accomplish the same example above – without all the complexity – is this:

WAY simpler!

However – you might notice – this doesn't work right. This is because Regex is case sensitive. If the Server NetBIOS name is detected in all CAPS, this expression wont match. I talk a little about this issue in this post: http://blogs.technet.com/kevinholman/archive/2009/04/21/quick-tip-using-regular-expressions-in-a-dynamic-group.aspx

So – based on that posts example – there is a simple way to make a RegEx case insensitive: (?i:blah)

Using that as an example – we can now make very advanced groupings, quite easily:

(?i: to make it case insensitive. ^ to signify the beginning of the word/line match. Here is the formula now:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Matches regular expression (?i:^C|^D) ) AND ( Windows Computer.NetBIOS Computer Name Matches regular expression (?i:^sr) ) )

Check it out:

Victory!

What if I wanted all logical disks that we NOT hosted by a Virtual Machine? Easy!

( Object is Logical Disk AND ( Windows Computer.Virtual Machine Equals False ) AND True )

This reveals a group of ALL logical disks hosted by a Windows Computer with the attribute of Virtual Machine = False:

As you can see – using the Hosting relationship of the disk – to the Windows Computer object, there is much more you can do with groups.

Updated Active Directory ADMP Management Pack released – Version 6.0.7065.0

kevinhol — Wed, 04 Nov 2009 19:53:34 GMT

This is now available on the catalog.

http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Changes in this update:

Support for monitoring Windows Server® 2008 R2 server operating systems as well as Windows® 7 client operating systems
Support for monitoring the Active Directory Web Service (ADWS) in Windows Server 2008 R2 as well as the Active Directory Management Gateway Service in Windows Server 2008 and Windows Server 2003

There were a few other minor fixes as well, such as the AD LSASS CPU overload monitor. This release also replaces a recently released ADMP version 6.0.7050.0 that was pulled to fix the aforementioned monitor.

Why do my group memberships for Windows Computers have machines that don't belong there?

kevinhol — Tue, 03 Nov 2009 05:35:50 GMT

Here is a little tip if you find that your Windows Computer Groups (and state views scoped by groups) contain computers that should not be there.

Have you noticed that you have state views or Windows Computer Groups that contain servers that you don't expect? Like Exchange Servers in your SQL Computers Group? Or SQL Servers in your Exchange 2007 computer group? Or maybe Hyper-V host servers in a LOT of your groups? If so – you are probably running Hyper-V, and using the Hyper-V MP version 6.0.6633.0.

Have a look at the below example: My SQL Management Pack “Computers” view – contains domain controllers, exchange servers… even XP clients… plus it also includes the Hyper-V host (VS3).

This can wreak havoc on your management group…. because we use groups of Windows Computers for Overrides, and for scoping console views.

The good news is – there is a very simple workaround: There is a relationship in this MP that we need to disable. This relationship attempts to associate the Windows Computer objects of a guest to its host – however it doesn't work properly, and isn't necessary.

Open Authoring in the console. Select “Object Discoveries”. Scope to “Hyper-V Virtual Machine”. Find the discovery named: “Hyper-V 2008 Guest Computer Relationship Discovery”

Create an override for this – “for all objects of class: Root Management Server”. Set this discovery to disabled.

Now that that is disabled – we need to run a little cleanup on aisle 7. We have a nifty little cmdlet in the OpsMgr command shell – named remove-disabledmonitoringobject. This cmdlet will basically remove any discovered objects – for any situation where they are explicitly disabled with an override on a discovery. Since that is what we just did (disabled a discovery) this will quickly delete any discovered relationships which previously existed.

Now – when I look at my state views scoped to SQL Computers group – I only see SQL servers, AND – the Hyper-V host. We don't want the Hyper-V host tho….. apparently the cmdlet cleanup doesn't take care of those. To resolve that membership – I generally bounce the HealthService on the Hyper-V hosts, and then the HealthService on the RMS, and in a few minutes they will be gone.

Voila!

Holy Management Packs Batman – we are surrounded!

kevinhol — Wed, 14 Oct 2009 00:00:19 GMT

Wow – what a plethora of new MP’s to look at.

Just in the past week – we have gotten a bunch of updated MP. I wont take any time to go into details on the changes in this post – but be aware! You probably need to get these in your evaluation and test cycles… looks like lots of good stuff in these updates.

From the MP catalog at: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Exchange Server 2003
10/12/2009
(6.0.6702.0)
Exchange Server 2003 Management Pack for System Center Operations Manager 2007
The Exchange Server 2003 Management Pack monitors the performance and availability of Exchange Server 2003 systems. It can also issue alerts for possible configuration problems Availability and performance monitoring is done using synthetic transactions. In addition, the Management Pack collects Event Log alerts and provides associated knowledge articles with additional user details, possible causes, and suggested resolutions. The Management Pack discovers and monitors the individual server components and functional areas on an Exchange Server 2003 server.

Exchange Server 2007
10/12/2009
(6.0.6702.0)
Microsoft Exchange Server 2007 Management Pack for System Center Operations Manager 2007 R2
The Exchange Server 2007 Management Pack is designed to monitor Exchange 2007 key health indicators, collect Exchange component-specific performance counters in one central location, and raise alerts for operator intervention as necessary. By detecting, sending alerts, and automatically responding to critical events, this Management Pack helps indicate, correct, and prevent possible service outages or configuration problems, allowing you to proactively manage Exchange servers and identify issues before they become critical. The Management Pack monitors and provides alerts for automatic notification of events indicating service outages, performance degradation, health monitoring, and centralized management.

Operations Manager 2007 R2 Management Pack
10/8/2009
(6.1.7533.0)
Operations Manager 2007 R2 Management Pack
The Operations Manager 2007 R2 Management Pack alerts you to problems with components such as agents, management servers, the Operations Manager database, agents, modules used by workflows and services so that you can continue to monitor the servers and clients that your business depends on. The management pack includes tasks that you can automate to get easy access to common diagnostic tools, such as restarting a health service or reloading an agent configuration.

Windows Server Operating System
9/18/2009
(6.0.6667.0)
Windows Server Operating System Management Pack for Operations Manager 2007
The Microsoft Windows Server management packs monitor the performance, health, and availability of Windows Server 2008 R2, 2008, 2003 R2, 2003, and Windows 2000 Server. By detecting, alerting on, and automatically responding to critical events and performance indicators, management packs reduce resolution times for issues and increase the overall availability and performance of your Windows Server operating systems, thereby helping to reduce the total cost of ownership.
(Chinese (Taiwan), German, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Chinese (PRC), Spanish)

Dude. Got R2? Then go get this update!

kevinhol — Fri, 09 Oct 2009 06:17:23 GMT

Seriously.

Operations Manager 2007 R2 Management Pack version 6.1.7533.0 from the MP CATALOG.

This is basically a set of the core MP’s in OpsMgr R2. You should expect to see these get updated on a fairly regular basis now. These are the MP’s that are built in to OpsMgr, that monitor the health of the Management group, from the server role, and agent role perspective. There should be a SP1 version of this update available soon to follow. These are a series of updates to OpsMgr, based on Community, Customer, and internal feedback…. and it’s good stuff.

Everyone running R2 should run these through your standard testing cycle and get these implemented. This is a simple MP import-update.

The MOMTEAM Blog has some more details here: http://blogs.technet.com/momteam/archive/2009/10/07/opsmgr-2007-r2-mp-version-6-1-7553-0-is-released.aspx

This includes a long list of updates (available in the guide in the download)… but I will hit a few high points.

You might have seen some of my blog posts on agents restarting all the time, in SP1 and R2, and the significant impact that can have:

http://blogs.technet.com/kevinholman/archive/2009/06/22/health-service-and-monitoringhost-thresholds-in-r2-how-this-has-changed-and-what-you-should-know.aspx

Well, this is mostly resolved in this update…. as the default threshold for HealthService and MonitoringHost has been changed from 100MB to 300MB. This will stop the majority of your agents from hitting this limit, and restarting. Once you import this MP update – you should review any overrids you made on these Monitors… and make sure you dont have any conflicts. The only other override I like to recommend is to generate an alert on these monitors – so that you will know when there is a problem that will cause the agent to get restarted – specifically on HealthService or MonitoringHost, and PrivateBytes or HandleCount:

You might find that there are still some agents that need a much higher threshold…. but those should be fairly rare, and limited to very large servers with a very high instance count of objects (large SQL clusters, large Exchange servers, etc) and also possibly for agents that are a watcher node or proxy for a large number of devices (network, VMWare, etc…)

In THIS blog post I discussed adding a custom perf counter collection rule and creating a view to see how many consoles or SDK connections you have. This is now built-in:

Lots of Knowledge and descriptions added to help troubleshoot alerts. For instance – on the common “Script or Executable Failed to run”

Summary

The Health Service was attempting to run an executable and was unable to create the process. This may affect some monitoring or discovery.

Causes

This can be caused by:

The executable could not be found.

The computer does not have enough resources (for example; memory) to run the executable.

The antivirus software on the computer blocking Visual Basic scripts or Java scripts. The following link is a link to the KB article regarding this issue.

Antivirus software blocking script execution

Resolutions

The alert description and context has information indicating which rule or monitor failed. The following link will display all events indicating a failure to run the executable:

View Batch Response Events

After reviewing the error in the context, check:

That the path to the executable exists on the computer.

The antivirus software is not blocking scripts from running.

That the computer is not over utilized.

Check Task Manager to see if there is enough free memory.

Check Task Manager to see if there are any processes consuming all the CPU.

Check the error information in the event or alert context for the script path and name. There could be a problem with the script not handling an error correctly and exiting. If the script exits without outputting the data that is expected (e.g. property bag data), this error is raised.

Another error can be caused by the misconfiguration of the workflow executing this script. The configuration (script params, policy, timeout) could be wrong causing no output or timeout.

There is a big list of other enhancements or fixes in the MP guide. Don't take this lightly – this is good stuff…. get it updated.

What is config churn?

kevinhol — Mon, 05 Oct 2009 08:29:49 GMT

There have been a couple good articles briefly covering this topic…. you might have read them. I will reference some below. Config churn is basically, when your RMS is in an almost never-ending loop of generating config. This can be caused by “less than optimized” management packs, pushing agents all the time, or injecting major changes into a management group, such as overrides or custom rules and monitors, or importing updated management packs. By examining this topic in depth – we will re-state some already known best practices with maintaining a healthy management group, and get some deeper knowledge as to why they are best practices in the first place.

Any time you push agents, or create rules and monitors, or overrides for widespread classes….. you can create a config update on the RMS that must be sent down to ALL agents in the management group. For small management groups (under 500 agents) this is generally not a big deal and processes rather quickly. For large management groups over 1000 agents, this can cause high resource utilization of the RMS and SQL Database, in terms of CPU, Memory, and Disk I/O. This can impact data insertion, and console performance during these times. For these reasons, we like to keep those activities down to a minimum during working hours, and schedule these major changes in an off-hours maintenance window.

What about “less than optimized” management packs? What does that mean? Well, this means management packs that you might be using, that have poorly written discoveries.

We have long known that a worst practice in Management Pack development, is to have a discovery that discovers instances of a class, that has properties for those instances that are likely to change frequently. Here is a write-up from OpsManJam on the topic: LINK

Ok… wait… Whaaaaat?

Let me put that in English:

Say we have a discovery for a Logical Disk. This will discover any logical disk, like C:, D:, E:, Q:, etc…. When we write the discovery for a logical disk, we can add properties to that discovery. These are attributes of the discovered instances. So – in this case – lets say we decided to add “Size” of the disk as a property, and “Free Space” as a property. And for the discovery frequency – we will run this discovery every hour, looking for new disks.

“Size” is an excellent property for the Logical Disk class. We like to know the size of the disks…. we can use this property group them if needed. “Size” of a logical disk is not something that we would expect to change very often.

“Free Space” is a horrible property for the Logical Disk class. Free space is something that will likely change, just a small amount even, between each run of the discovery. Free space is a property that is likely to change frequently, therefore – it should NOT be used in a discovery.

Make sense?

Ok – so… what's the big deal?

Well, the agent will run almost all discoveries that it knows about when the health service starts up (like when you bounce the service, or after a reboot). It will always send this discovery data to the management server. (this is another reason why agents restarting all the time is very bad) Then, it will run then based on the “Interval” frequency specified on the discovery. Sometimes this is as frequent as once per hour, sometimes as long as once per day. When the discovery runs, the agent will inspect the discovery data that it gets, and compare it to the last discovery data it sent to the management server. If nothing changed – the agent drops the discovery data and does nothing. IF anything changed in the values of the discovery data – it will re-submit the new data to the management server, which will submit this data to the database. The RMS will detect the change, and will have to recalculate (regenerate) configuration. You will see this on the RMS as a 21025 event:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          9/27/2009 11:51:49 PM
Event ID:      21025
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
OpsMgr has received new configuration for management group PROD1 from the Configuration Service. The new state cookie is "D7 9B A4 BE 00 90 CF 13 35 B5 9B 5F 3B 14 FF 78 D6 13 9A 2D "

The 21025 event isn’t really “bad”… it simply means the config service did its job. It re-generated its configuration file from the database data, and wrote it to: \Program Files\System Center Operations Manager 2007\Health Service State\Connector Configuration Cache\<MGNAME>\OpsMgrConnector.Config.xml The problem is – when this config file gets large (like in large agent count environments) and when the “Config Instance Space” is large (number of discovered objects in total). Recalculating this config can have a significant impact on the disk where the file exists on the RMS, use lots of memory and CPU on the RMS for the config service, and use significant disk I/O on the SQL database.

If the RMS is in a perpetual cycle of recalculating config, and sending these config updates to all agents…. the performance of the management group is impacted.

Daniele Grandini of Quaue Nocent Docent is pretty much the “godfather” of good information researching the 21025 event. Read his 3 part series on config churn here:

http://nocentdocent.wordpress.com/2009/07/09/troubleshooting-21025-events-wrap-up/

So – what can I do if I think I have too much config churn?

The biggest problem causing the most frequent config updates is management packs with noisy discoveries. However, lets wrap up all the issues that can cause it, and what you can do:

New agents. Discover/install/approve new agents in bulk and off-hours.
Overrides. Set overrides during off-hours, or create override MP’s in a lab, then synch to production management groups during schedule off-hours times.
Custom rules and monitors. Create these during off-hours, or create using the authoring console, test in a lab, then import to production during off-hours.
Newly discovered instances. For instance – someone adds a new disk, or SQL database, or DNS zone, to an existing agent. Not much we can do about this, except the expectation that this would be done during off hours.
Management packs with noisy discovery properties. See below.

Ok – the remainder of this article will touch on #5.

How can I tell which discoveries are noisy?

Danilele Grandini has put together a good query on this, from his link: http://nocentdocent.wordpress.com/2009/05/23/how-to-get-noisy-discovery-rules/

I will repost these (slightly modified) below:

/* Top Noisy Rules in the last 24 hours */

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes'
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

and

/* Modified properties in the last 24 hours */

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName

Wow – that returned a LOT of discoveries running all the time! What can I do?

Don't import too many MP’s! The FIRST line of defense – is NOT to import ANY management packs into a management group that you don't absolutely need RIGHT THEN. Management packs are constantly updated, and by the time you have an actual SLA in a technology area – there will likely be a newer, better MP available for it. The biggest mistake many customers make is to import any available MP for a technology that they have internally. They end up with a FLOOD of alerts, big fat databases, slow consoles, and lots of weird errors. MP’s should be transitioned slowly, one at a time – tuning and resolving as you go.

Disable the noisy discoveries. Probably not a great solution, unless they discover objects that you really don't care about – but there are other objects in the MP that you DO want to monitor.

Increase the interval of the discovery frequency. This means… essentially – change any “bad” discoveries to run only once per day.

Add a “synch time” override to the discovery – if possible. This option is not available unless the MP author of the discovery exposed it. What this will do – it cause all the agents to ONLY run the discovery at a distinct and specified time every day (say…. 1AM). This might cause too much discovery data to flood in at one time… but since it will all come in at the same time – it wont cause constant config churn all throughout the day.

Re-write the discovery. If this is a custom MP – rewrite the discovery/MP, and remove that property which changes too often.

Make sure your hardware and software is optimized for scalability. On your RMS – it is good to place your config file on fast disks, especially in large environments. I have worked with very large customers who were experiencing config churn, but had zero ill effects, because their RMS disk I/O was on a 4 spindle RAID10 with 15K spindles, CPU and memory were really good, and their SQL database disk I/O for the OpsDB was excellent with plenty of breathing room. I have also worked with smaller agent counts, where config churn has a serious impact…. mostly due to the RMS config file being places on the same RAID spindle set as that OS and pagefile, using only 2 older 10,000 RPM disks. The SQL disk I/O was also just borderline for their agent count.

Re-run the queries periodically – especially after importing/upgrading to a new management pack in your management group. This “instance space change” report should be part of your testing and evaluation of a new MP when brought into your lab…. if you have a large agent count environment.

Some very common discoveries I have seen – that have properties that change very frequently – are listed below. I often recommend these be overridden to run once per day (86,400 seconds)

Discovery Display Name	Discovery Target Class	Discovered Type	Default frequency	Modified frequency
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File	7260	86400
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File Group	7260	86400
Discover SQL 2000 Databases	SQL 2000 DB Engine	SQL 2000 DB	1800	86400
Discover Databases for a Database Engine	SQL 2005 DB Engine	SQL 2005 DB	7200	86400
DNS 2003 Component Discovery	DNS 2003 Server	DNS 2003 Zone	21600	86400
DNS 2008 Component Discovery	DNS 2008 Server	DNS 2008 Zone	21600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2003 Server Role	IIS FTP Site	3600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2000 Server Role	IIS NNTP Virtual Server	3600	86400

The above is just a sample – you should examine the query output of the query above and see what is impacting your management group the most.

Keep your management pack names SHORT in SP1!

kevinhol — Fri, 02 Oct 2009 23:31:08 GMT

I have seen this twice now… so I will blog about it. It seems to be rare in the wild, but it will completely cripple a management group when this occurs. So beware SP1 users!

This article does not apply to R2. This is only an issue in OpsMgr 2007 SP1.

When you create your custom management packs – and especially your override management packs – keep the names as simple and short as possible. There is an issue in OpsMgr SP1 – when an agent tries to download management packs – where it will fail if the MP ID (derived from the MP Name) is too long. The worst part about this problem is there there WONT be an error logged. What will happen – is that the agent will keep trying to re-download the MP in question, and it will block ALL other MP’s from being downloaded from that point forward.

There is no simple way to know this condition is impacting you. What will happen – is that an agent will continue to work just fine… but will not get any NEW management packs. So… you might think all is well, but not so. The symptoms thatmight lead you to notice that something is wrong:

Performance data not collected for newer MP’s.

Objects not being discovered for newer MP’s

Alerts not generating as expected from newer MP’s.

The root problem is an age old Windows issue…. file paths over 255 characters not supported well. This has been resolved in R2 in how the agent copies the files over.

In both cases I have seen – someone was creating an override MP for the “IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems” management pack. So – when they created their override MP – they named it something like: “Overrides - IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems”

This equated to a Management Pack ID of: Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems

That MP ID is 86 characters!

What happens…. is when this management pack is created, and an override is placed in it…. (or custom rule)… the agents that require it will:

Get contacted by the RMS to update their config, and then issue a config change request (21024 in the event log)

Receive new config from the RMS (event 21025)

Process new config – and realize they need a new MP, and request that MP (event 1200)

Where this process breaks…. is the next step in the chain should be that the agent RECEIVES the MP (event 1201) and then issues a statement that the new config has become active (event 1210). The never happen.

Behind the scenes…. from looking at a ETL tracelog, we can see this is failing, when we try to move the file from the “downloaded files” folder to the “management packs” folder:

Error CMPFileManager::MoveManagementPackFile(MPFileManager_cpp383)MoveFile from '\\?\C:\Program Files\System Center Operations Manager 2007\Health Service State\Downloaded Files\MGNAME\1\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' to 'C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' failed with code 3(ERROR_PATH_NOT_FOUND).

In the example above – the bad path is:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml

Which is 261 characters. The limit is 255.

Therefore – I recommend you keep your Management Pack *ID* to less than 60 characters. You can examine your long management packs by looking in the console – generally your longest display names will be the longest ID’s:

Even some Microsoft MP’s are dangerously close to the limit…. such as: Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMWare.HostPerformance with 76 characters. In most environments you can squeak by at 79 characters…. more or less depending on where you installed your agent path.

Here is a SQL query you can run against the OpsDB to also detect this condition…. and quick check all your potentially long MP’s:

select MPName from managementpack
WHERE len(MPName) > 60

Just change 60 to whatever character count you want.

DONT freak out if you have some more than 60. Just be aware.

DO freak out if you have some more than 80!

If someone had the time and was really handy – you could write a monitor – that runs against the RMS – that would in turn query the OpsDB, and run this query, and change the RMS to unhealthy when over a threshold. THAT would be cool…. and alert you when some author makes a really long MP that has the potential to break all your agents.

Another idea I had… was to create a correlated missing event monitor…. and when you get a 1200, but do NOT get a 1201 within, say, 15 minutes….. that might be a problem. Of course if you wrote this and were already impacted…. the bad agents would never get your new MP to tell you. :-)

Fixing troubled agents

kevinhol — Fri, 02 Oct 2009 00:23:46 GMT

Sometimes agents either will not “talk” to the management server upon initial installation, and sometimes an agent can get unhealthy long after working fine. Agent health is an ongoing task of any OpsMgr Admin’s life.

This post in NOT an “end to end” manual of all the factors that influence agent health…. but that is something I am working on for a later time. There are so many factors in an agent’s ability to communicate and work as expected. A few key areas that commonly affect this are:

DNS name resolution (Agent to MS, and MS to Agent)
DNS domain membership (disjointed)
DNS suffix search order
Kerberos connectivity
Kerberos SPN’s accessible
Firewalls blocking 5723
Firewalls blocking access to AD for authentication
Packet loss
Invalid or old registry entries
Missing registry entries
Corrupt registry
Default agent action accounts locked down/out (HSLockdown)
HealthService Certificate configuration issues.
Hotfixes required for OS Compatibility
Management Server rejecting the agent

How do you detect agent issues from the console? The problem might be that they are not showing up in the console at all! Perhaps they might be a manual install that never shows up in Pending Actions? Or a push deployment, that stays stuck in Pending actions and never shows up under “Agent Managed”. Or even one that does show up under “Agent Managed” but never shows as being monitored… returning agent version data, etc.

One of the BEST things you can do when faced with an agent health issue… if to look on the agent, in the OperationsManager event log. This is a fairly verbose log that will almost always give you a good hint as to the trouble with the agent. That is ALWAYS one of my first steps in troubleshooting.

Another way of examining Agent health – is by the built in views in OpsMgr. In the console – there is a view – Located at the following:

This view is important – because it gives us a perspective of the agent from two different points:

1. The perspective of the agent monitors running on the agent, measuring its own “health”.

2. The perspective of the “Health Service Watcher” which is the agent being monitored from a Management Server".

If any of these are red or yellow – that is an excellent place to start. This should be an area that your level 1 support for Operations manager checks DAILY. We should never have a high number of agents that are not green here. If they aren't – this is indicative of an unhealthy environment, or the admin team not adhering to best practices (such as keeping up with hotfixes, using maintenance mode correctly, etc…

Use Health Explorer on these views – to drill down into exactly what is causing the Agent, or Health Service Watcher state to be unhealthy.

Now…. the following are some general steps to take to “fix” broken agents. These are not in definitive order. The order of steps really comes down to what you find when looking at the logs after taking these steps.

Start the HealthService on the agent. You might find the HealthService is just not running. This should not be common or systemic. Consider enabling the recovery for this condition to restart the HealthService on Heartbeat failure. However – if this is systemic – it is indicative of something causing your HealthService to restart too frequently, or administrators stopping SCOM. Look in the OpsMgr event log for verification.

Bounce the HealthService on the agent. Sometimes this is all that is needed to resolve an agent issue. Look in the OpsMgr event log after a HealthService restart, to make sure it is clean with no errors.

Clear the HealthService queue and config (manually). This is done by stopping the HealthService. Then deleting the “\Program Files\System Center Operations Manager 2007\Health Service State” folder. Then start the HealthService. This removes the agent config file, and the agent queue files. The agent starts up with no configuration, so it will resort to the registry to determine what management server to talk to. From the registry – it will find out if it is AD integrated, or a fixed management server to talk to if not. This is located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\PROD1\Parent Health Services\ location, in the \<#>\NetworkName string value. The agent will contact the management server – request config, receive config, download the appropriate management packs, apply them, run the discoveries, send up discovery data, and repeat the cycle for a little while. This is very much what happens on a new agent during initial deployment.

Clear the HealthService queue and config (from the console). When looking at the above view (or any state view or discovered inventory view which targets the HealthService or Agent class) there is a task in the actions pane - “Flush Health Service State and Cache”. This will perform a very similar action to that above…. as a console task. This will only work on an agent that is somewhat responsive…. if it does not work you need to perform this manually as the agent is really broken from communication with the management server. This task will never complete, and will not return success – because the task breaks off from itself as the queue is flushed.

“Repair” the agent from the console. This is done from the Administration pane – Agent Managed. You should not run a repair on any AD-integrated agent – as this will break the AD integration and assign it to the management server that ran the repair action. A “repair” technically just reinstalls the agent in a push fashion, just like an initial agent deployment. It will also apply/reapply any agent related hotfixes in the management server’s \Program Files\System Center Operations Manager 2007\AgentManagement\ directories.

Reinstall the agent (manually). This would be for manual installs or when push/repair is not possible. This section is where the combination of options gets a little tricky. When you are at this point… where you have given up, I find just going all the way with a brute force reinstall is the best way. This means performing the following steps:

Uninstall the agent via add/remove programs.
Run the Operations Manager Cleanup Tool CleanMom.exe or CleanMOM64.exe. This is designed to make sure that the service, files, and all registry entires are removed.
Ensure that the agent’s folder is removed at: \Program Files\System Center Operations Manager 2007\
Ensure that the following registry keys are deleted:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService

Reboot the agent machine (if possible)
Delete the agent from Agent Managed in the OpsMgr console. This will allow a new HealthService ID to be detected and is sometimes a required step to get an agent to work properly, although not always required.
Now that the agent is gone cleanly from both OpsMgr console and the agent Operating System…. manually reinstall the agent. Keep it simple – install it using a named management server/management group, and use Local System for the agent action account (these will remove any common issues with a low priv domain account, and AD integration if used) If it works correctly – you can always reinstall again using low priv or AD integration.
Remember to import certificats at this point if you are using those on the individual agent.
As always – look in the OperationsManager event log…. this will tell you if it connected, and is working, or if there is a connectivity issue.

To summarize…. there are many things that can cause an agent issue, and many methods to troubleshoot. However – to summarize at a very general level, my typical steps are:

Review OpsMgr event log on agent
Bounce HealthService
Bounce HealthService clearing \Health Service State folder.
Complete brute force reinstall of the agent.

If it an external issue is causing the issue (DNS, Kerberos, Firewall) then these steps likely will not help you…. but those should be available from the OpsMgr event log.

Also – make sure you see my other posts on agent health and troubleshooting during deployment:

Console based Agent Deployment Troubleshooting table

Agent discovery and push troubleshooting in OpsMgr 2007

Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

Agent Pending Actions can get out of synch between the Console, and the database

Which hotfixes should I apply-

New Base OS MP 6.0.6667.0 adds file fragmentation monitor to all Logical Disks

kevinhol — Mon, 28 Sep 2009 19:09:03 GMT

I recently blogged about the new Base OS MP that was recently released: HERE

One of the things you will notice RIGHT off the bat… is that a huge percentage of your logical disks will go into a warning state, if you don't already have some sort of scheduled defragmentation set up. This will be true for virtual machines and physical machines…. anything over 10 percent file fragmentation (or the OS recommended setting) will get hit:

You will also get many warning alerts on this monitor…. the first time the condition is detected and the state changes for this monitor. This monitor checks status every Saturday, at 3:00AM by default, for all logical disks discovered.

If you don't care about this monitoring in SCOM – disable this monitor using overrides.

If you do care about seeing the state change – but don't want the alerts – turn the “Generates Alert” property to False, using overrides.

You can adjust the threshold from 10% to some other number…. but make sure you take note – this monitor will ignore the “File Percent Fragmentation” property by default, and always use the OS recommended setting. If you want to control this – you also need to set “Use OS Recommendation” to FALSE.

Here is an example of hard coding the frag percentage to 20% from the OS default:

“Use OS Recommendation” property description:

Lastly – one thing of interest…. If you want SCOM to “fix” the fragmentation issue…. it can. There is a recovery on this very monitor that can run a VBScript that will run a defrag job against your logical disks. It is disabled by default.

Keep in mind – if you turn on this defrag…. on your physical boxes – it wont be a big deal… it will simply fix the fragmentation issue. However – this will also run on ALL yours VM’s. If this was triggered all at the same time – Saturday at 3:00AM by default – this can kill the disk I/O on the disk subsystem hosting your VM/VHD files. Keep this in mind if you decide to enable this…. This recovery will only run when the state change is detected… as a recovery to the condition, so any disks that are already in a warning state will not run this recovery should you enable it. This defrag has a timeout of 1 hour…. so it should kill the defrag if it cannot complete within an hour.

Another cool thing to do – is to use the recovery action as a single run-time task. You can do this right from health explorer, to fix the disks on your own schedule:

Just click the link, and run the task:

Minimize this…. and just let it run – you can come back in 1 hour – and see if it completed, or timed out.

You can also monitor for task status in the Task Status list in the console:

On the agent – you will see the following events logged in the OpsMgr event log:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 10:50:04 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Perform Defragmentation (disk: C:; computer: OMDW.opsmgr.net).

And when completed:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 11:03:44 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Defragmentation completed (disk: C:; computer: OMDW.opsmgr.net): FilePercentFragmentation = 0.

New KMS MP released version 6.0.7234.0 and some things you should know

kevinhol — Thu, 24 Sep 2009 01:37:33 GMT

A new KMS (Key Management Services) MP is released. I don't expect this is a widely used MP out there…. but there are some things you should know if you ever imported the previous versions.

First – the new MP is available on the catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

DO NOT download this MP directly via the R2 direct download mechanism. You need to get this one from the web catalog. As in all MP’s – you need to read the guide FIRST… because the “Before you import” section is critical.

The new MP is not backward compatible with the previous MP. You MUST delete any previous version of the KMS MP you have installed.

If you used the previous version of the KMS MP – you might have noticed some issues. For instance, the old KMS MP added a unique dataset type to the warehouse, with a grooming procedure just for the KMS dataset. These didn't work. When you download the new MP, it comes with instructions and a cleanup script to fix this. If you ever imported the old MP – you are affected. You will see the following events on your RMS every 10-11 minutes:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          9/23/2009 4:20:53 PM
Event ID:      31552
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Failed to store data in the Data Warehouse.
Exception 'SqlException': Sql execution failed. Error 777971002, Level 16, State 1, Procedure StandardDatasetGroom, Line 303, Message: Sql execution failed. Error 2812, Level 16, State 62, Procedure StandardDatasetGroom, Line 145, Message: Could not find stored procedure 'KMS_EventGroom'.

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.StandardDataSetMaintenance
Instance name: KMS Activation Event Data Set
Instance ID: {35CE8740-3FCF-FA7F-D733-1ECD0FBE0D7F}
Management group: PROD1

So – here are the necessary steps to get this new MP working (from reading the MP guide):

Delete the old MP (typically going to be version 6.0.6278.9)
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.
Import the new MP.

If you had previously imported the old KMS MP – and are getting this event on your RMS, and you no longer wish to use the KMS MP anymore….. you can clean this up using the script that is included with the new MP. So for a cleanup ONLY – where you do not wish to use the KMS MP:

Make sure the old KMS MP is deleted.
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.

The script output simply displays this when complete:

A cleaned up environment will no longer have any KMS datasets, or stored procedures for grooming this dataset. You can inspect for these with the following queries against the warehouse:

select * from standarddataset

select * from sysobjects where name like ('%kms%')

Once the cleanup script has been run – these should return no results.

***Note – the new MP does not contain any reports. The old MP had reports the leveraged the KMS Dataset, but the new MP does not.

Alert Notification Subscription Variables, and linking that to the console, database, and SDK

kevinhol — Wed, 23 Sep 2009 20:54:00 GMT

Attached you will find a spreadsheet, with all the possible alert notification subscription variables that I am aware of. In this spreadsheet, I link these to the same values in the Alert table of the DB, the alert view of the DB, the Console alert view, the SDK (Get-Alert), and lastly the new R2 Connector Key pairs.

My thought was to better understand each data property type of an alert, and what you can managed, from each area. Hope this is beneficial.

Most of these are listed at my other blog post on alert description and notification variables: http://blogs.technet.com/kevinholman/archive/2007/12/12/adding-custom-information-to-alert-descriptions-and-notifications.aspx

Here is a sample shot of the spreadsheet tool:

See attached: