Kevin Holman's OpsMgr Blog : management pack

Tuning tip – turning off some over-collection of events

kevinhol — Wed, 25 Nov 2009 22:34:47 GMT

We often think of tuning OpsMgr by way of tuning “Alert Noise”…. by disabling rules that generate alerts that we don't care about, or modifying thresholds on monitors to make the alert more actionable for our specific environment.

However – one area of OpsMgr that often goes overlooked, is event overcollection. This has a cost… because these events are collected and create LAN/WAN traffic, agent overhead, OpsDB size bloat, and especially, DataWarehouse size bloat. I have worked with customers who had a data warehouse that was over one third event data….. and they had ZERO requirement for this nor did they want it. They were paying for disk storage, and backup expense, plus added time and resources on the framework, all for data they cared nothing about.

MOST of these events, are enabled out of the box, and are default OpsMgr collect rules from the “System Center Core Monitoring” MP. These events are items like "config requested”, “config delivered”, “new config active”. They might be interesting, but there is no advanced analysis included to use these to detect a problem. In small environments, they are not usually a big deal. But in large agent count environments, these events can account for a LOT of data, and provide little value unless you are doing something advanced in analyzing them. I have yet to see a customer who did that.

At a high level – here is how I like to review these events:

Review the Most Common Events query that your OpsDB has.
Create a “My Workspace” view for each event that has a HIGH event count.
Examine the event details for value to YOU.
View the rule that collected the event.
1. Does the rule also alert or do anything special, or does it simply collect the event?
2. Do you think the event is required for any special reporting you do?
Create an Override, in an Override MP for the rule source management pack, to disable the rule.
Continue to the next event in the query output, and evaluate it.

So, what I like to do – is to run the “Most Common Events” query against the OpsDB, and examine the top events, and consider disabling these event collection rules:

Most common events by event number and event publishername:

SELECT top 20 Number as EventID, COUNT(*) AS TotalEvents, Publishername as EventSource
FROM EventAllView eav with (nolock)
GROUP BY Number, Publishername
ORDER BY TotalEvents DESC

The trick is – to run this query periodically – and to examine the most common events for YOUR environment. The easiest way to view these events – to determine their value – is to create a new Events view in My Workspace, for each event – and then look at the event data, and the rule that collected it: (I will use a common event 21024 as an example:)

What we can see – is that this is a very typical event, and there is likely no real value for collecting and storing this event in the OpsDB or Warehouse.

Next – I will examine the rule. I will look at the Data Source section, and the Response section. The purpose here is to get a good idea of where this collection rule is looking, what events it is collecting, and if there is also an alert in the response section. If there is an alert in the response section – I assume this is important, and will generally leave these rules enabled.

If the rule simply collected the event (no alerting), is not used in any reports that I know about (rare condition) and I have determined the event provides little to no value to me, I disable it. You will find you can disable most of the top consumers in the database.

Here is why I consider it totally cool to disable these uninteresting event collection rules:

If they are really important – there will be different alert generating rule to fire an alert
They fill the databases, agent queues, agent load, and network traffic with unimportant information.
While troubleshooting a real issue – we would examine the agent event log – we wouldn’t search through the database for collected events.
Reporting on events is really slow – because we cannot aggregate them, so any views are reports dont work well with events.
If we find we do need one later – simply remove the override.

Here is an example of this one:

So – I create an override in my “Overrides – System Center Core” MP, and disable this rule “for all objects of class”.

Here are some very common event ID’s that I will generally end up disabling their corresponding event collection rules:

1206

1210

1215

1216

10102

10401

10403

10409

10457

10720

11771

21024

21025

21402

21403

21404

21405

29102

29103

I don't recommend everyone disable all of these rules… I recommend you periodically view your top 10 or 20 events… and then review them for value. Just knocking out the top 10 events will often free up 90% of the space they were consuming.

The above events are the ones I run into in most of my customers… and I generally turn these off, as we get no value from them. You might find you have some other events as your top consumers. I recommend you review them in the same manner as above – methodically. Then revisit this every month or two to see if anything changed.

I’d also love to hear if you have other events that you see as your top consumer that isn't my list above… SOME events are created from script (conversion MP’s) and unfortunately you cannot do much about those, because you would have to disable the script to fix them. I’d be happy to give feedback on those, or add any new ones to my list.

Native Exchange 2007 MP 6.0.6741.0 ships for OpsMgr 2007 SP1 users

kevinhol — Mon, 09 Nov 2009 13:34:43 GMT

If for some reason you cannot or will not upgrade to OpsMgr 2007 R2, then this update is for you!

This Native MP replaces the old “converted” Exchange 2007 MP. If you are running the old conversion MP then I STRONGLY suggest making the effort to transition to this MP.

This MP version 6.0.6741.0 is for SP1 users ONLY! If you are running OpsMgr 2007 R2 – there is an R2-Only version of this MP for you on the catalog.

Some notes:

This MP is essentially designed after the R2-only version – with the R2-only enhancement features removed.

This MP is NOT UPGRADE COMPATIBLE with ANY previously public released Exchange 2007 MP. This means you need to remove your existing Exchange MP before you import this one…. and start a new overrides MP for this one, for tuning. Believe me – it will be worth it. Technically – you can run both MP’s side by side…. here is a blurb from the guide:

This new Exchange Server 2007 Management Pack for Operations Manager 2007 SP1 does not support an upgrade from the previously released, converted Exchange Server 2007 Management Pack for Operations Manager 2007. We recommend that you do not run both management packs in parallel. However, running both management packs in parallel is a supported scenario. If you decide to run both management packs in parallel, before importing this management pack, disable any synthetic transaction in the converted management pack. For more information about migrating custom settings from the previously released, converted management pack to the new, native management pack, see Appendix: Migrating from the Previously-Released Converted Exchange Server 2007 Management Pack.

Here is what's new from the guide:

Reports. This management pack provides a set of reports specific to Exchange 2007. For the list of reports and for more information about the reports, see Appendix: Reports.

· Improved disk monitoring. The management pack improves Exchange disk monitoring by providing support for mount points and by discovering three types of disks. This improvement enables you to establish a disk monitoring standard across all Exchange 2007 servers using fewer overrides. The types of disks discovered are Database (on the Mailbox server role), Log (on the Mailbox server role), and Queue (on the Hub and Edge server roles). For information about how to configure Exchange 2007 Disk Monitoring, see Configure Disk Monitoring. For information about classes in this management pack, see Classes.

· A significant number of rules and monitors that are not actionable or may be noisy are disabled. Note that many of these rules are still in the management pack so that you can enable them if necessary. For a list of rules and monitors that are disabled by default in this management pack, see Appendix: Monitors and Rules Disabled by Default.

· A number of performance collection rules are disabled. For a list of disabled performance collection rules, see Performance Collection Rules Disabled by Default.

· Support for monitoring any number of Exchange organizations using a single Operations Manager 2007 management group. There are no specific requirements imposed by this management pack for monitoring multiple Exchange organizations.

· Full support for clustered configurations using Windows Clustering technology. For more details, see Monitoring Clustered Mailbox Servers and Log Shipping.

· An extensive class model, showing the relevant Exchange 2007 server roles and components, as well as service-centric components, allowing you to measure availability or performance at a granular level. The class model fully supports the Distributed Application Designer, allowing you to create custom distributed applications using the appropriate Exchange 2007 components. The appropriate classes and monitors are declared public, enabling you to extend the monitoring if needed. For hierarchical diagrams of the Exchange 2007 management pack classes, see Classes.

· Improved low-privilege support. This management pack supports installing the agent with the minimum rights required by the Operations Manager agent.

· Improved tasks. The management pack includes a number of tasks that simplify troubleshooting and reduce the amount of time to resolve the alert.

· Improved topology discovery. This management pack introduces a number of improvements to topology discovery.

· There is no central discovery script. Each agent is responsible for discovering its piece of the topology. This ensures that potential permissions and trust issues are minimized. There are no requirements on trusts in order to generate the topology.

· The topology will show only agent-managed Exchange 2007 servers, ensuring that the topology consists only of servers that the Exchange administrator has chosen to monitor.

· Improved synthetic transactions. This management pack improves the support for the Exchange 2007 synthetic transactions in several ways:

· The management pack supports local mail flow transactions. For more information, see the Configuring Mail Flow Synthetic Transactions section.

· The management pack supports running the following Exchange 2007 synthetic transactions:

· Test-Mailflow (http://go.microsoft.com/fwlink/?LinkID=137740) (local mail flows only)

· Test-OwaConnectivity (http://go.microsoft.com/fwlink/?LinkID=137732) (internal only)

· Test-MapiConnectivity (http://go.microsoft.com/fwlink/?LinkID=137739)

· Test-ReplicationHealth (http://go.microsoft.com/fwlink/?LinkID=148797)

· The discovery of Exchange 2007 server roles is disabled by default and minimal Exchange 2007 monitoring is applied. This allows you to discover and monitor your servers gradually, as well as tune the management pack as you bring more agent-managed Exchange 2007 servers into the Operations Manager environment. For more information, see Enable Exchange 2007 Server Role Discovery.

Making groups of logical disks – an example from simple to advanced

kevinhol — Thu, 05 Nov 2009 06:00:37 GMT

I have been seeing this question come up a lot lately – as customers try and create groups of their disks – in order to create overrides for “certain” disks. So – I am creating this post to give some real world examples.

Well – I will start this simply. Say we want to create a group of all logical disks, with the drive letters of C: and D:?

I would start with creating a new group – and adding the “Windows Server 2003 Logical Disk” class. Now – I could just use the parent class of “Logical Disk” instead of the OS specific class if I wanted. The only issue with that is that most monitors targeting a disk – are OS specific – and duplicated three times. So it is best to create specific groups for these – but totally not required.

Ok – so in the Dynamic Members query builder – I click add, and pick a property. Since I know “Device Name” contains the drive letter – this will do nicely. I select device name “Equals” “C:”.

Now – I want to also include D:. There are many way to do – this – and I will go through them. First – I could simply Insert a new line for Windows Server 2003 Logical Disk – and replicate the line I have – adding one for D:

Only one problem – this is an “AND” grouping – I really need this to be an “OR” grouping to include both C: and D: drives. You can switch this grouping the in UI, just right click the word “AND” and change it to an OR grouping:

Voila!

This formula now looks like:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Equals C: ) OR ( Device Name Equals D: ) )

Save your group – then right click it – and choose “View Group Members”. This will ensure we are cooking with gas. It should contain all your Windows 2003 based C: and D: volumes.

So far – so good.

Now – what if I ONLY want C: and D: disks, that are HOSTED by specific Windows Computers? I can do that too! Lets say I want a group – of all the C: and D: logical disks, on servers that begin with the name “SR______”

If you look at the bottom of the list of properties for Logical Disks – you will see (Host=Windows Computer). From here – we can pick any attribute of the Windows Computer class as well to add to our expression – to limit our logical disks in our group to very specific Computers.

Go back to the properties of your group, edit the Dynamic Members, and you can construct something like this:

Which translates to the following formula:

( Object is Windows Server 2003 Logical Disk AND ( Windows Computer.NetBIOS Computer Name Matches wildcard sr* ) AND ( ( Device Name Equals C: ) OR ( Device Name Equals D: ) ) )

Now – I will be honest – getting all the “ands” and “ors” in the right place using the UI is a big pain. It is very easy to screw it up. I like to simplify this to the fewest lines possible – using Regex.

Using Regular Expressions – we can use modifiers to create very advanced expressions. my favorites are ^ which means the beginning of a new line or word, and | which is the “pipe” symbol – which means “or”.

So a simple way to accomplish the same example above – without all the complexity – is this:

WAY simpler!

However – you might notice – this doesn't work right. This is because Regex is case sensitive. If the Server NetBIOS name is detected in all CAPS, this expression wont match. I talk a little about this issue in this post: http://blogs.technet.com/kevinholman/archive/2009/04/21/quick-tip-using-regular-expressions-in-a-dynamic-group.aspx

So – based on that posts example – there is a simple way to make a RegEx case insensitive: (?i:blah)

Using that as an example – we can now make very advanced groupings, quite easily:

(?i: to make it case insensitive. ^ to signify the beginning of the word/line match. Here is the formula now:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Matches regular expression (?i:^C|^D) ) AND ( Windows Computer.NetBIOS Computer Name Matches regular expression (?i:^sr) ) )

Check it out:

Victory!

What if I wanted all logical disks that we NOT hosted by a Virtual Machine? Easy!

( Object is Logical Disk AND ( Windows Computer.Virtual Machine Equals False ) AND True )

This reveals a group of ALL logical disks hosted by a Windows Computer with the attribute of Virtual Machine = False:

As you can see – using the Hosting relationship of the disk – to the Windows Computer object, there is much more you can do with groups.

Updated Active Directory ADMP Management Pack released – Version 6.0.7065.0

kevinhol — Wed, 04 Nov 2009 19:53:34 GMT

This is now available on the catalog.

http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Changes in this update:

Support for monitoring Windows Server® 2008 R2 server operating systems as well as Windows® 7 client operating systems
Support for monitoring the Active Directory Web Service (ADWS) in Windows Server 2008 R2 as well as the Active Directory Management Gateway Service in Windows Server 2008 and Windows Server 2003

There were a few other minor fixes as well, such as the AD LSASS CPU overload monitor. This release also replaces a recently released ADMP version 6.0.7050.0 that was pulled to fix the aforementioned monitor.

Why do my group memberships for Windows Computers have machines that don't belong there?

kevinhol — Tue, 03 Nov 2009 05:35:50 GMT

Here is a little tip if you find that your Windows Computer Groups (and state views scoped by groups) contain computers that should not be there.

Have you noticed that you have state views or Windows Computer Groups that contain servers that you don't expect? Like Exchange Servers in your SQL Computers Group? Or SQL Servers in your Exchange 2007 computer group? Or maybe Hyper-V host servers in a LOT of your groups? If so – you are probably running Hyper-V, and using the Hyper-V MP version 6.0.6633.0.

Have a look at the below example: My SQL Management Pack “Computers” view – contains domain controllers, exchange servers… even XP clients… plus it also includes the Hyper-V host (VS3).

This can wreak havoc on your management group…. because we use groups of Windows Computers for Overrides, and for scoping console views.

The good news is – there is a very simple workaround: There is a relationship in this MP that we need to disable. This relationship attempts to associate the Windows Computer objects of a guest to its host – however it doesn't work properly, and isn't necessary.

Open Authoring in the console. Select “Object Discoveries”. Scope to “Hyper-V Virtual Machine”. Find the discovery named: “Hyper-V 2008 Guest Computer Relationship Discovery”

Create an override for this – “for all objects of class: Root Management Server”. Set this discovery to disabled.

Now that that is disabled – we need to run a little cleanup on aisle 7. We have a nifty little cmdlet in the OpsMgr command shell – named remove-disabledmonitoringobject. This cmdlet will basically remove any discovered objects – for any situation where they are explicitly disabled with an override on a discovery. Since that is what we just did (disabled a discovery) this will quickly delete any discovered relationships which previously existed.

Now – when I look at my state views scoped to SQL Computers group – I only see SQL servers, AND – the Hyper-V host. We don't want the Hyper-V host tho….. apparently the cmdlet cleanup doesn't take care of those. To resolve that membership – I generally bounce the HealthService on the Hyper-V hosts, and then the HealthService on the RMS, and in a few minutes they will be gone.

Voila!

Holy Management Packs Batman – we are surrounded!

kevinhol — Wed, 14 Oct 2009 00:00:19 GMT

Wow – what a plethora of new MP’s to look at.

Just in the past week – we have gotten a bunch of updated MP. I wont take any time to go into details on the changes in this post – but be aware! You probably need to get these in your evaluation and test cycles… looks like lots of good stuff in these updates.

From the MP catalog at: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Exchange Server 2003
10/12/2009
(6.0.6702.0)
Exchange Server 2003 Management Pack for System Center Operations Manager 2007
The Exchange Server 2003 Management Pack monitors the performance and availability of Exchange Server 2003 systems. It can also issue alerts for possible configuration problems Availability and performance monitoring is done using synthetic transactions. In addition, the Management Pack collects Event Log alerts and provides associated knowledge articles with additional user details, possible causes, and suggested resolutions. The Management Pack discovers and monitors the individual server components and functional areas on an Exchange Server 2003 server.

Exchange Server 2007
10/12/2009
(6.0.6702.0)
Microsoft Exchange Server 2007 Management Pack for System Center Operations Manager 2007 R2
The Exchange Server 2007 Management Pack is designed to monitor Exchange 2007 key health indicators, collect Exchange component-specific performance counters in one central location, and raise alerts for operator intervention as necessary. By detecting, sending alerts, and automatically responding to critical events, this Management Pack helps indicate, correct, and prevent possible service outages or configuration problems, allowing you to proactively manage Exchange servers and identify issues before they become critical. The Management Pack monitors and provides alerts for automatic notification of events indicating service outages, performance degradation, health monitoring, and centralized management.

Operations Manager 2007 R2 Management Pack
10/8/2009
(6.1.7533.0)
Operations Manager 2007 R2 Management Pack
The Operations Manager 2007 R2 Management Pack alerts you to problems with components such as agents, management servers, the Operations Manager database, agents, modules used by workflows and services so that you can continue to monitor the servers and clients that your business depends on. The management pack includes tasks that you can automate to get easy access to common diagnostic tools, such as restarting a health service or reloading an agent configuration.

Windows Server Operating System
9/18/2009
(6.0.6667.0)
Windows Server Operating System Management Pack for Operations Manager 2007
The Microsoft Windows Server management packs monitor the performance, health, and availability of Windows Server 2008 R2, 2008, 2003 R2, 2003, and Windows 2000 Server. By detecting, alerting on, and automatically responding to critical events and performance indicators, management packs reduce resolution times for issues and increase the overall availability and performance of your Windows Server operating systems, thereby helping to reduce the total cost of ownership.
(Chinese (Taiwan), German, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Chinese (PRC), Spanish)

What is config churn?

kevinhol — Mon, 05 Oct 2009 08:29:49 GMT

There have been a couple good articles briefly covering this topic…. you might have read them. I will reference some below. Config churn is basically, when your RMS is in an almost never-ending loop of generating config. This can be caused by “less than optimized” management packs, pushing agents all the time, or injecting major changes into a management group, such as overrides or custom rules and monitors, or importing updated management packs. By examining this topic in depth – we will re-state some already known best practices with maintaining a healthy management group, and get some deeper knowledge as to why they are best practices in the first place.

Any time you push agents, or create rules and monitors, or overrides for widespread classes….. you can create a config update on the RMS that must be sent down to ALL agents in the management group. For small management groups (under 500 agents) this is generally not a big deal and processes rather quickly. For large management groups over 1000 agents, this can cause high resource utilization of the RMS and SQL Database, in terms of CPU, Memory, and Disk I/O. This can impact data insertion, and console performance during these times. For these reasons, we like to keep those activities down to a minimum during working hours, and schedule these major changes in an off-hours maintenance window.

What about “less than optimized” management packs? What does that mean? Well, this means management packs that you might be using, that have poorly written discoveries.

We have long known that a worst practice in Management Pack development, is to have a discovery that discovers instances of a class, that has properties for those instances that are likely to change frequently. Here is a write-up from OpsManJam on the topic: LINK

Ok… wait… Whaaaaat?

Let me put that in English:

Say we have a discovery for a Logical Disk. This will discover any logical disk, like C:, D:, E:, Q:, etc…. When we write the discovery for a logical disk, we can add properties to that discovery. These are attributes of the discovered instances. So – in this case – lets say we decided to add “Size” of the disk as a property, and “Free Space” as a property. And for the discovery frequency – we will run this discovery every hour, looking for new disks.

“Size” is an excellent property for the Logical Disk class. We like to know the size of the disks…. we can use this property group them if needed. “Size” of a logical disk is not something that we would expect to change very often.

“Free Space” is a horrible property for the Logical Disk class. Free space is something that will likely change, just a small amount even, between each run of the discovery. Free space is a property that is likely to change frequently, therefore – it should NOT be used in a discovery.

Make sense?

Ok – so… what's the big deal?

Well, the agent will run almost all discoveries that it knows about when the health service starts up (like when you bounce the service, or after a reboot). It will always send this discovery data to the management server. (this is another reason why agents restarting all the time is very bad) Then, it will run then based on the “Interval” frequency specified on the discovery. Sometimes this is as frequent as once per hour, sometimes as long as once per day. When the discovery runs, the agent will inspect the discovery data that it gets, and compare it to the last discovery data it sent to the management server. If nothing changed – the agent drops the discovery data and does nothing. IF anything changed in the values of the discovery data – it will re-submit the new data to the management server, which will submit this data to the database. The RMS will detect the change, and will have to recalculate (regenerate) configuration. You will see this on the RMS as a 21025 event:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          9/27/2009 11:51:49 PM
Event ID:      21025
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
OpsMgr has received new configuration for management group PROD1 from the Configuration Service. The new state cookie is "D7 9B A4 BE 00 90 CF 13 35 B5 9B 5F 3B 14 FF 78 D6 13 9A 2D "

The 21025 event isn’t really “bad”… it simply means the config service did its job. It re-generated its configuration file from the database data, and wrote it to: \Program Files\System Center Operations Manager 2007\Health Service State\Connector Configuration Cache\<MGNAME>\OpsMgrConnector.Config.xml The problem is – when this config file gets large (like in large agent count environments) and when the “Config Instance Space” is large (number of discovered objects in total). Recalculating this config can have a significant impact on the disk where the file exists on the RMS, use lots of memory and CPU on the RMS for the config service, and use significant disk I/O on the SQL database.

If the RMS is in a perpetual cycle of recalculating config, and sending these config updates to all agents…. the performance of the management group is impacted.

Daniele Grandini of Quaue Nocent Docent is pretty much the “godfather” of good information researching the 21025 event. Read his 3 part series on config churn here:

http://nocentdocent.wordpress.com/2009/07/09/troubleshooting-21025-events-wrap-up/

So – what can I do if I think I have too much config churn?

The biggest problem causing the most frequent config updates is management packs with noisy discoveries. However, lets wrap up all the issues that can cause it, and what you can do:

New agents. Discover/install/approve new agents in bulk and off-hours.
Overrides. Set overrides during off-hours, or create override MP’s in a lab, then synch to production management groups during schedule off-hours times.
Custom rules and monitors. Create these during off-hours, or create using the authoring console, test in a lab, then import to production during off-hours.
Newly discovered instances. For instance – someone adds a new disk, or SQL database, or DNS zone, to an existing agent. Not much we can do about this, except the expectation that this would be done during off hours.
Management packs with noisy discovery properties. See below.

Ok – the remainder of this article will touch on #5.

How can I tell which discoveries are noisy?

Danilele Grandini has put together a good query on this, from his link: http://nocentdocent.wordpress.com/2009/05/23/how-to-get-noisy-discovery-rules/

I will repost these (slightly modified) below:

/* Top Noisy Rules in the last 24 hours */

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes'
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

and

/* Modified properties in the last 24 hours */

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName

Wow – that returned a LOT of discoveries running all the time! What can I do?

Don't import too many MP’s! The FIRST line of defense – is NOT to import ANY management packs into a management group that you don't absolutely need RIGHT THEN. Management packs are constantly updated, and by the time you have an actual SLA in a technology area – there will likely be a newer, better MP available for it. The biggest mistake many customers make is to import any available MP for a technology that they have internally. They end up with a FLOOD of alerts, big fat databases, slow consoles, and lots of weird errors. MP’s should be transitioned slowly, one at a time – tuning and resolving as you go.

Disable the noisy discoveries. Probably not a great solution, unless they discover objects that you really don't care about – but there are other objects in the MP that you DO want to monitor.

Increase the interval of the discovery frequency. This means… essentially – change any “bad” discoveries to run only once per day.

Add a “synch time” override to the discovery – if possible. This option is not available unless the MP author of the discovery exposed it. What this will do – it cause all the agents to ONLY run the discovery at a distinct and specified time every day (say…. 1AM). This might cause too much discovery data to flood in at one time… but since it will all come in at the same time – it wont cause constant config churn all throughout the day.

Re-write the discovery. If this is a custom MP – rewrite the discovery/MP, and remove that property which changes too often.

Make sure your hardware and software is optimized for scalability. On your RMS – it is good to place your config file on fast disks, especially in large environments. I have worked with very large customers who were experiencing config churn, but had zero ill effects, because their RMS disk I/O was on a 4 spindle RAID10 with 15K spindles, CPU and memory were really good, and their SQL database disk I/O for the OpsDB was excellent with plenty of breathing room. I have also worked with smaller agent counts, where config churn has a serious impact…. mostly due to the RMS config file being places on the same RAID spindle set as that OS and pagefile, using only 2 older 10,000 RPM disks. The SQL disk I/O was also just borderline for their agent count.

Re-run the queries periodically – especially after importing/upgrading to a new management pack in your management group. This “instance space change” report should be part of your testing and evaluation of a new MP when brought into your lab…. if you have a large agent count environment.

Some very common discoveries I have seen – that have properties that change very frequently – are listed below. I often recommend these be overridden to run once per day (86,400 seconds)

Discovery Display Name	Discovery Target Class	Discovered Type	Default frequency	Modified frequency
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File	7260	86400
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File Group	7260	86400
Discover SQL 2000 Databases	SQL 2000 DB Engine	SQL 2000 DB	1800	86400
Discover Databases for a Database Engine	SQL 2005 DB Engine	SQL 2005 DB	7200	86400
DNS 2003 Component Discovery	DNS 2003 Server	DNS 2003 Zone	21600	86400
DNS 2008 Component Discovery	DNS 2008 Server	DNS 2008 Zone	21600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2003 Server Role	IIS FTP Site	3600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2000 Server Role	IIS NNTP Virtual Server	3600	86400

The above is just a sample – you should examine the query output of the query above and see what is impacting your management group the most.

Keep your management pack names SHORT in SP1!

kevinhol — Fri, 02 Oct 2009 23:31:08 GMT

I have seen this twice now… so I will blog about it. It seems to be rare in the wild, but it will completely cripple a management group when this occurs. So beware SP1 users!

This article does not apply to R2. This is only an issue in OpsMgr 2007 SP1.

When you create your custom management packs – and especially your override management packs – keep the names as simple and short as possible. There is an issue in OpsMgr SP1 – when an agent tries to download management packs – where it will fail if the MP ID (derived from the MP Name) is too long. The worst part about this problem is there there WONT be an error logged. What will happen – is that the agent will keep trying to re-download the MP in question, and it will block ALL other MP’s from being downloaded from that point forward.

There is no simple way to know this condition is impacting you. What will happen – is that an agent will continue to work just fine… but will not get any NEW management packs. So… you might think all is well, but not so. The symptoms thatmight lead you to notice that something is wrong:

Performance data not collected for newer MP’s.

Objects not being discovered for newer MP’s

Alerts not generating as expected from newer MP’s.

The root problem is an age old Windows issue…. file paths over 255 characters not supported well. This has been resolved in R2 in how the agent copies the files over.

In both cases I have seen – someone was creating an override MP for the “IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems” management pack. So – when they created their override MP – they named it something like: “Overrides - IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems”

This equated to a Management Pack ID of: Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems

That MP ID is 86 characters!

What happens…. is when this management pack is created, and an override is placed in it…. (or custom rule)… the agents that require it will:

Get contacted by the RMS to update their config, and then issue a config change request (21024 in the event log)

Receive new config from the RMS (event 21025)

Process new config – and realize they need a new MP, and request that MP (event 1200)

Where this process breaks…. is the next step in the chain should be that the agent RECEIVES the MP (event 1201) and then issues a statement that the new config has become active (event 1210). The never happen.

Behind the scenes…. from looking at a ETL tracelog, we can see this is failing, when we try to move the file from the “downloaded files” folder to the “management packs” folder:

Error CMPFileManager::MoveManagementPackFile(MPFileManager_cpp383)MoveFile from '\\?\C:\Program Files\System Center Operations Manager 2007\Health Service State\Downloaded Files\MGNAME\1\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' to 'C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' failed with code 3(ERROR_PATH_NOT_FOUND).

In the example above – the bad path is:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml

Which is 261 characters. The limit is 255.

Therefore – I recommend you keep your Management Pack *ID* to less than 60 characters. You can examine your long management packs by looking in the console – generally your longest display names will be the longest ID’s:

Even some Microsoft MP’s are dangerously close to the limit…. such as: Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMWare.HostPerformance with 76 characters. In most environments you can squeak by at 79 characters…. more or less depending on where you installed your agent path.

Here is a SQL query you can run against the OpsDB to also detect this condition…. and quick check all your potentially long MP’s:

select MPName from managementpack
WHERE len(MPName) > 60

Just change 60 to whatever character count you want.

DONT freak out if you have some more than 60. Just be aware.

DO freak out if you have some more than 80!

If someone had the time and was really handy – you could write a monitor – that runs against the RMS – that would in turn query the OpsDB, and run this query, and change the RMS to unhealthy when over a threshold. THAT would be cool…. and alert you when some author makes a really long MP that has the potential to break all your agents.

Another idea I had… was to create a correlated missing event monitor…. and when you get a 1200, but do NOT get a 1201 within, say, 15 minutes….. that might be a problem. Of course if you wrote this and were already impacted…. the bad agents would never get your new MP to tell you. :-)

New Base OS MP 6.0.6667.0 adds file fragmentation monitor to all Logical Disks

kevinhol — Mon, 28 Sep 2009 19:09:03 GMT

I recently blogged about the new Base OS MP that was recently released: HERE

One of the things you will notice RIGHT off the bat… is that a huge percentage of your logical disks will go into a warning state, if you don't already have some sort of scheduled defragmentation set up. This will be true for virtual machines and physical machines…. anything over 10 percent file fragmentation (or the OS recommended setting) will get hit:

You will also get many warning alerts on this monitor…. the first time the condition is detected and the state changes for this monitor. This monitor checks status every Saturday, at 3:00AM by default, for all logical disks discovered.

If you don't care about this monitoring in SCOM – disable this monitor using overrides.

If you do care about seeing the state change – but don't want the alerts – turn the “Generates Alert” property to False, using overrides.

You can adjust the threshold from 10% to some other number…. but make sure you take note – this monitor will ignore the “File Percent Fragmentation” property by default, and always use the OS recommended setting. If you want to control this – you also need to set “Use OS Recommendation” to FALSE.

Here is an example of hard coding the frag percentage to 20% from the OS default:

“Use OS Recommendation” property description:

Lastly – one thing of interest…. If you want SCOM to “fix” the fragmentation issue…. it can. There is a recovery on this very monitor that can run a VBScript that will run a defrag job against your logical disks. It is disabled by default.

Keep in mind – if you turn on this defrag…. on your physical boxes – it wont be a big deal… it will simply fix the fragmentation issue. However – this will also run on ALL yours VM’s. If this was triggered all at the same time – Saturday at 3:00AM by default – this can kill the disk I/O on the disk subsystem hosting your VM/VHD files. Keep this in mind if you decide to enable this…. This recovery will only run when the state change is detected… as a recovery to the condition, so any disks that are already in a warning state will not run this recovery should you enable it. This defrag has a timeout of 1 hour…. so it should kill the defrag if it cannot complete within an hour.

Another cool thing to do – is to use the recovery action as a single run-time task. You can do this right from health explorer, to fix the disks on your own schedule:

Just click the link, and run the task:

Minimize this…. and just let it run – you can come back in 1 hour – and see if it completed, or timed out.

You can also monitor for task status in the Task Status list in the console:

On the agent – you will see the following events logged in the OpsMgr event log:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 10:50:04 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Perform Defragmentation (disk: C:; computer: OMDW.opsmgr.net).

And when completed:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 11:03:44 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Defragmentation completed (disk: C:; computer: OMDW.opsmgr.net): FilePercentFragmentation = 0.

New KMS MP released version 6.0.7234.0 and some things you should know

kevinhol — Wed, 23 Sep 2009 22:37:33 GMT

A new KMS (Key Management Services) MP is released. I don't expect this is a widely used MP out there…. but there are some things you should know if you ever imported the previous versions.

First – the new MP is available on the catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

DO NOT download this MP directly via the R2 direct download mechanism. You need to get this one from the web catalog. As in all MP’s – you need to read the guide FIRST… because the “Before you import” section is critical.

The new MP is not backward compatible with the previous MP. You MUST delete any previous version of the KMS MP you have installed.

If you used the previous version of the KMS MP – you might have noticed some issues. For instance, the old KMS MP added a unique dataset type to the warehouse, with a grooming procedure just for the KMS dataset. These didn't work. When you download the new MP, it comes with instructions and a cleanup script to fix this. If you ever imported the old MP – you are affected. You will see the following events on your RMS every 10-11 minutes:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          9/23/2009 4:20:53 PM
Event ID:      31552
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Failed to store data in the Data Warehouse.
Exception 'SqlException': Sql execution failed. Error 777971002, Level 16, State 1, Procedure StandardDatasetGroom, Line 303, Message: Sql execution failed. Error 2812, Level 16, State 62, Procedure StandardDatasetGroom, Line 145, Message: Could not find stored procedure 'KMS_EventGroom'.

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.StandardDataSetMaintenance
Instance name: KMS Activation Event Data Set
Instance ID: {35CE8740-3FCF-FA7F-D733-1ECD0FBE0D7F}
Management group: PROD1

So – here are the necessary steps to get this new MP working (from reading the MP guide):

Delete the old MP (typically going to be version 6.0.6278.9)
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.
Import the new MP.

If you had previously imported the old KMS MP – and are getting this event on your RMS, and you no longer wish to use the KMS MP anymore….. you can clean this up using the script that is included with the new MP. So for a cleanup ONLY – where you do not wish to use the KMS MP:

Make sure the old KMS MP is deleted.
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.

The script output simply displays this when complete:

A cleaned up environment will no longer have any KMS datasets, or stored procedures for grooming this dataset. You can inspect for these with the following queries against the warehouse:

select * from standarddataset WHERE SchemaName = 'KMSEvent'

select * from sysobjects where name like ('%kms%')

Once the cleanup script has been run – these should return no results.

***Note – the new MP does not contain any reports. The old MP had reports the leveraged the KMS Dataset, but the new MP does not.

New Base OS MP released 9-18-2009 version 6.0.6667.0 to add support for Windows Server 2008R2 and other fixes

kevinhol — Tue, 22 Sep 2009 15:40:59 GMT

As you can see from the “Changes in this update section” of the guide – this isn't just about supporting Server 2008 R2 – there are many fixes and updates, so this would be a good one to get through your testing cycles and plan for production.

Available on the catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Changes in This Update

The September 2009 release (version 6.0.6667.0) of the Windows Server Operating System Management Pack includes the following changes:

Support for Windows Server 2008 R2, including new classes to represent Windows Server 2008 R2 systems specifically.
The intervals for some discoveries have been changed to reduce CPU usage. For details, see Objects the Windows Server Operating System Management Pack Discovers.
Added Logical Disk Fragmentation Level monitors for Windows Server 2003 and Windows Server 2008.
Disabled the following rules and monitors for Windows Server 2008 because the events needed are in Windows Server 2003 only.
- Duplicate computer name was detected
- Windows Activation State
- Disk Group Failed
- Disk Group Auto Import Failed
- Volume Not Started
- Run WMIAdap
Fixed the summary details for the Performance History (Percent Processor Time) report to indicate that the object type is Windows Server 2003 operating system rather than Windows Server 2003 processor.
Fixed the Performance History (Percent Interrupt Time) report to use a rule that targets the operating system rather than the processor.
Changed the data source for discoveries from System.Scheduler to System.Discovery.Scheduler to avoid performance issues.
Changed the Logical Disk Availability monitor to use a Run As profile with administrative privileges to enable it to run in a low-privilege environment.
Modified logical disk discovery to exclude mount points, because numerous rules use performance counters that are not applicable to mount points.
Fixed an issue in which an alert could be generated for Windows Activation too early.
Addressed an issue that was causing some discoveries to not show up correctly in the authoring section of the console.
Improved the product knowledge for the alert generated by the “A Service is Mis-configured” rule.
Fixed issue in which logical disk health check could have performance problems when run against clustered servers.
Fixed links to views in product knowledge articles.
Added Active Alerts views for Windows Server 2003 and Windows Server 2008.
Added retry logic to the "Probe Module: Is Feature Installed" module to avoid failures due to periodic time-outs.
Added a new integer property named "Size (MBytes) (Numeric)" to the Logical Disk class and updated the existing property "Size (Bytes) (String)" to clarify that it is a string. The new property can now be used in formulas and views as expected.

Make sure you also check out the new supported config statement and known issues regarding Server 2008 R2 and Windows 7 support:

http://support.microsoft.com/kb/974722

Community Cisco Network Hardware Management pack available

kevinhol — Wed, 16 Sep 2009 01:08:43 GMT

This guy has some serious talent.

http://operatingquadrant.com/2009/09/14/scom-advanced-snmp-monitoring-part-iii-the-completed-cisco-management-pack/

Make sure you check out all of Kris’s blog posts – amazingly good stuff there on using SNMP to accomplish monitoring that is otherwise very difficult, among other good posts.

New SQL MP released – version 6.0.6648.0 and why you NEED it.

kevinhol — Sat, 25 Jul 2009 00:53:11 GMT

Get it from the MP catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

This is considered a “minor” update. However, there are some SIGNIFICANT changes you should consider. Not all of the changes are documented in the guide.

I did a XML diff of the files to see what is new:

Some of the changes:

In previous versions of the SQL MP, event based alerts generated from Windows 2003 based clusters did not work. This has been fixed in this version of the MP.
SQL 2005 “Database Status” Monitor inspects DB status (online, offline) every hour, from every 2 minutes UnitMonitor ID="Microsoft.SQLServer.2005.Database.DBStatusMonitor"
SQL 2005 “Disk Free Space” DB File freespace monitor now runs every hour, from every 2 minutes. UnitMonitor ID="Microsoft.SQLServer.2005.DBFile.DiskFreeSpace"
SQL 2008 “Database Status” Monitor inspects DB status (online, offline) every hour, from every 2 minutes UnitMonitor ID="Microsoft.SQLServer.2008.Database.DBStatusMonitor"
SQL 2008 “Disk Free Space” DB File freespace monitor now runs every hour, from every 2 minutes. UnitMonitor ID="Microsoft.SQLServer.2008.DBFile.DiskFreeSpace"

For these reasons alone, anyone running clustered SQL servers should consider rapidly moving to this MP.

I say that – with some caveats: In order to use this MP effectively – your agents must be EITHER:

OpsMgr 2007 R2

<OR>

OpsMgr 2007 SP1 with http://support.microsoft.com/kb/959865 applied.

The guide is a bit misleading – it states:

In order for this fix to work fully on Windows Server 2003 and Windows Server 2008 operating systems, all agents on clustered nodes need to be running Operations Manager 2007 SP1 or Operations Manager 2007 R2 with the update installed from Knowledge Base article 959865

This is technically misleading – in that it leads you to think you need to apply 959865 to your R2 environment to support this MP. You don't. 959865 is an OpsMgr 2007 SP1 ONLY hotfix , and does NOT apply to R2.

Now – what if you don't want to apply this hotfix to SP1 – can you still run this MP? Technically – Yes, however it will not work on Server 2008 failover clusters hosting a SQL instance. There is a change in the way Server 2008 failover clusters log events… and a module was changed in R2, and in SP1-hotfix-959865, to address this. That said – I strongly recommend you move to R2. If you cannot move to R2, and will remain on SP1 for some time, I strongly recommend you apply 959865 to your management group.

Getting lots of Script Failed To Run alerts? WMI Probe Failed Execution? Backward Compatibility Script Error?

kevinhol — Mon, 29 Jun 2009 17:24:31 GMT

In OpsMgr 2007, it is likely that your most common alert is not really a MP based alert from a technology management pack…. it could be a built-in alert that a script failed, or WMI could not be accessed. This is because when WMI is broken on a machine, almost EVERYTHING fails to execute properly on that agent.

At a recent health check at a customer site, we found the top 5 alerts in his environment (by cumulative repeat count) were:

WMI Probe Module Failed Execution
Service Check Data Source Module Failed Execution
Backward Compatibility Script Error
Script or Executable Failed to run
Service Check Probe Module Failed Execution

Sometimes – these alerts are normal…. the server is busy, or someone rebooted it without putting it into maintenance mode and allowing the workflows to unload gracefully.

However, if you have a high repeat count on these, it is typically indicative of something seriously broken on that agent(s). Most of the time – the failure is in WMI. Many customers get frustrated with these script errors, because they see them as “false alerts” because they don't know how to resolve the root cause, and we just tell you “this action broke”, we don't tell you why. It is critical that you examine these alerts, however, because these alerts will indicate something seriously wrong with an agent, such as broken WMI/cscript/OS issue. If you ignore them, or disable them – you will never know that monitoring is not functioning 100%.

Generally – here is how I attack script/WMI failures.

1. If the repeat count is 0 or 1, I ignore these as random failures, and close the alerts from time to time.

2. If the repeat count is very high, then something is wrong with the agent, and needs remediation on the agent OS. Investigate the OpsMgr event log on the agent for Warning/Critical events – to see if a lot of workflows are failing due to this issue.

The FIRST thing I do – is to see if WMI is responsive. I run WBEMTEST, and connect to “root\cimv2”. I then hit “query” and execute a “select * from win32_operatingsystem” to see if it returns results, or an error. Next – I look at the namespace from the alert in SCOM…. perhaps it is “root\MicrosoftDNS”, or “root\CCM”. Then – I try and run the query that is failing from the alert.

If EITHER of the above connections/queries fail…. then I know what's wrong. WMI has a core issue, and I punt this to my platform or application team to fix it. Sometimes it needs a MOF recompile, sometimes it needs WMI service bounced or the OS bounced.

If these all appear to work correctly, or, the problem is resolved after a WMI service bounce, then re-appears later – check out the following:

There are many things you can do to resolve/remediate these issues. Here is a list of the most common fixes:

1. Apply http://support.microsoft.com/kb/933061 This resolves a LOT of issues on the Windows 2003 OS with WMI. This should be one of your first steps. This applies to x86 or x64 Windows Server 2003 SP1 or SP2.

2. Registry modification for WMI buffer thresholds (see below)

“HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\Low Threshold On Events (B)" to 35000000 (default is 10000000)
”HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\High Threshold On Events (B)" to 70000000 (default is 20000000)

The registry modification to WMI buffers increases the amount of objects that WMI can hold before injecting sleep delays to the WMI service.

3. Apply http://support.microsoft.com/kb/955360 This updates the Windows Scripting Host (cscript) to version 5.7. This resolves script timeouts, and scripts consuming a LOT of CPU during execution, and problems with multiple scripts running at the same time. This applies to x86 or x64 Windows Server 2003 SP1 or SP2. This is a very good hotfix for DNS servers, DHCP servers, and Domain Controllers. This has been seen to lessen the impact of VBscripts consuming a large amount of CPU during runtime.

Making these three modifications should resolve the majority of systemic issues out there, unless WMI is completely corrupt/unresponsive and needs repair. Sometimes, rebooting a server, or bouncing WMI will temporarily resolve these issues as well, if you cannot apply the fixes immediately.

If you have applied all three of these above, and are still experiencing a systemic repeat of a WMI query/script failure…. the next step would be to try running the query directly, accessing the namespace in WBEMtest. I’d like to hear about any experiences here.

Creating custom dynamic computer groups based on registry keys on agents

kevinhol — Wed, 10 Jun 2009 07:09:48 GMT

I have had a few requests now for this, so I thought I would take the time to write up the process.

Lets say I have three support levels of servers:

Level 1 – servers critical to business operations (ex: customer facing web applications, SQL back-ends)

Level 2 – important servers (ex: messaging, internal apps)

Level 3 – non-essential servers (ex: non-critical or highly redundant internal apps)

Lets say we want to create overrides for certain rules… where we will page on anything in Level 1 group, email notify on Level 2 group, and simply alert for Level 3. Possibly we want to create views, and only see alerts for Level 1 servers. Perhaps we wish to scope users so they only see Level 1 and Level 2 servers in the console?

Well – the first step is to place these servers into groups.

Sure – we can do this manually, with explicit assignments to the group. But that is resource intensive over time, and we might miss one down the road. I’d prefer to dynamically create the groups of Windows Computers based on a name…. but this can be difficult sometimes – where we don't have a solid naming scheme, or other criteria to group by.

I will demonstrate another way to accomplish this… by coming up with a business process to use a registry key on your managed servers, and collect this registry attribute with SCOM. Then – use this Registry attribute for dynamic group memberships.

Ultimately – there are three simple steps to this process:

1. Create registry keys on agents.

2. Extended a class with an attribute, to discover the registry keys and values.

3. Create dynamic groups based on the attribute values from the registry.

It is just that simple.

To get started – lets talk about our custom registry key. For this example, I am going to create a new Key at HKLM\Software\ and call it “CompanyName”

Next – in that key – I will create a new DWORD Value, named “SupportLevel”

Lastly – I will assign a numeric value to “SupportLevel” on each server, either 1, 2, or 3.

In my environment…. my Hyper-V servers are critical. They host all of my VM’s, including many business critical applications. Therefore – they will get Level 1.

My Exchange 2007 servers handle all my mail traffic and notifications, so I will set their registry value to Level 2.

My Exchange 2003 servers have been retired – for MP testing only… so we will set those to Level 3.

Here is a table that shows what I am planning:

ServerName	SupportLevel
VS1	1
VS2	1
VS3	1
EX1CLN1	2
EX1CLN2	2
EXCAS	2
EX2CLN1	3
EX2CLN2	3
OWA	3

So – I get all my registry values set on all computers. This is a big job at first, but it is a one time deal, and you can even script it if you are handy.

Next… we need to discover these registry entries in SCOM, as attributes of a class. Then were can use that attribute to group objects. Since I want Windows Computer objects in my groups (Windows Computer is a good object for most overrides, scoping, notifications…etc..) we would like to have these attributes added to the Windows Computer class.

However – there is a problem. The Windows Computer object is in a sealed MP. We cannot just add information to that class as we would like. Therefore – OpsMgr allows us to “Extend” an existing class… and add our custom attributes to it. This “Extended” class is basically a copy of the existing class… it will have all the built in attributes of Windows Computer, and will also have our custom attribute properties. It’s is easier to see it than to talk about it.

First – in the Ops console – authoring pane – go to Attributes. Create a new attribute. I am going to call this one “SupportLevel”

Next – choose “Registry” for the discovery type.

Next – We need to pick the Target class. We want Windows Computer. Note – this will create a new class, named “Windows Computer_Extended” by default. We can use this name, or you can rename this whatever you want. It is your class. I will leave it at the default.

Most important! Management Pack location.

This is CRITICAL. Spend some time making sure you are creating these attributes in the correct location. If you leave this MP unsealed XML…. then any groups you create that use these attributes, will have to be placed in this same MP. Then – if you use these groups for Overrides – those overrides will be force to go in this same MP. There is a “cardinal rule” in SCOM… objects in one unsealed MP cannot reference another unsealed MP. So – we cannot have a group in one unsealed MP, and then use that group for an override in another unsealed override MP.

So – we have two choices.

1. Keep an unsealed MP… and live with the fact that attribute, group, and override will all have to be placed here.

2. Create the attribute and the dynamic group in the MP, then seal it. Then – you can use this group in ANY of your override MP’s… for Exchange, SQL, etc…

I strongly recommend option #2 for this exercise… but you can make this decision for yourself.

Ok…. I will choose Option #2 (seal the MP), so I will create a new MP just for this extended class, and groups.

On the next screen – we can put in our registry information:

In this example – I am looking for a registry Value (1, 2, or 3), and my attribute type is “Int” for integer.

For the frequency, set this to a reasonable frequency to discover you machines as they come on to you network. Typically, once per day is sufficient (86400 seconds) Remember – this will run against ALL your Windows Computers… so never set this more frequent than once per hour… that creates unnecessary overhead.

Ok – lets examine our work!

Go to Monitoring, Discovered Inventory, and change target type to our new class “Windows Computer_Extended”

If you do this quickly – you may find it is empty. This is what is happening behind the scenes: All Windows Computers are now downloading our newly created MP. They are going to run the registry attribute discovery, and submit their discovery data to the management server. The Management Server will insert this discovery data in the database. Over time, you will start to see all your Windows Computers pop into this class membership. You will notice a new attribute now, in addition to all the existing Windows Computer attributes. This attribute is “SupportLevel” and will be 1, 2, 3, or empty… depending on what each agent find in the registry.

Now – I set my registry discovery to once per day…. so I will need to wait 24 hours before I can expect all my healthy agents to show up in this list. To speed things up – I am going to bounce the HealthService on these example agents. (Agents run all discoveries when a HealthService restarts, and then on their frequency schedule)

Here is an example a few minutes after bouncing the HealthService on some agents:

Next on the list – create the groups. I will create these in the same MP that the attributes exist in.

I will call my first group “CompanyName – Support Level 1 Servers Group”. I like to append the word “Group” to all groups I create as a best practice. This helps us determine this group class is actually a group when we see it in the list of classes in the UI. I sure wish all MP authors would take this to heart, since every group is actually a singleton class.

On the dynamic members screen – I will fins my “Windows Computer_Extended” class – and click Add. What we now see – is that we have a new attribute to use, “Support Level”

I will set this group to “SupportLevel Equals 1” and click OK.

Now – I can right-click my new group – and choose “View Group Members”

Yee-haw! It works! Now – I simply repeat this above step – creating groups for SupportLevel 2, and 3.

Now – that is done. This is the area, that I recommend we stop… take a breather…. then seal the MP. If you seal the MP – we will be able to use the groups for overrides in any other override MP. If you choose not to seal the MP now… any overrides you use the groups for – will be forced into this same MP. Please keep that in mind.

Since I am harping on sealing the MP…. I am going to do a quick example of just that. Jonathan Almquist has an excellent tutorial on sealing MP’s HERE and we will use his example.

**Note – when running the sn.exe commands to create our key…. we only need to do this one… not every time we want to seal an MP.

***Critical note – you need to keep a backup of this key… because it will be required for making updates to this MP in the future, re-sealing, and keeping the ability to upgrade the existing MP in production.

So, I create the folders, create the key using sn.exe, copy over the referenced MP’s from the RMS, and now I am ready to seal.

MPSeal.exe c:\mpseal\input\CompanyName.SupportLevel.MP.xml /I "c:\mpseal\mp" /Keyfile "c:\mpseal\key\PairKey.snk" /Company "CompanyName" /Outdir "c:\mpseal\output"

Works great.

Now – I can delete my unsealed MP from the management group, and import my sealed MP.

Phew. All the heavy lifting is done. Now… I have my groups… I can start setting up overrides using these groups, or scoping notifications.

On my Support Level 1 group – I will use this to set up my pager Notification subscriptions to only page based on specific classes, and this group.

On my Support Level 2 group – I will use this to override important alerts to High Priority… because I am using High Priority as a filter for email notifications, per my previous blog post here: http://blogs.technet.com/kevinholman/archive/2008/06/26/using-opsmgr-notifications-in-the-real-world-part-1.aspx

On my Support Level 3 group – I will use this group for tweaking/disabling rules and monitors for the group… turning off discoveries so they don't discover lab servers, scoping views, etc.

Maybe in my next post…. I will build on this MP… and show a really simple way to add the Health Service Watcher objects to these dynamic groups… for each Windows Computer object that is in the group – so we can use these groups for Heartbeat failure notifications.