Kevin Holman's OpsMgr Blog

Error alerts from the DNS MP – script failures, WMI Probe failed?

2009-06-29T19:45:25Z

I have seen this at several customer sites, and even in my own lab.

You might find the following alerts (below) stemming from the DNS MP.

This appears to be a random issue with WMI that causes this when trying to access the WMI based DNS namespace, and query it.

To resolve – see my previous post: Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

WMI Probe Module Failed Execution
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 10409
Description:
Object enumeration failed
Query: 'Select EventLogLevel from MicrosoftDNS_Server'
HRESULT: 0x80041001
Details: Generic failure
One or more workflows were affected by this.
Workflow name: Microsoft.Windows.DNSServer.2003.Monitor.ServerLoggingLevel
Instance name: dc01.opsmgr.net
Instance ID: {11056C4C-B933-98ED-3DC5-4B9AAE232B23}
Management group: PROD1

WMI Probe Module Failed Execution
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 10409
Description:
Object enumeration failed
Query: 'Select Name, Shutdown, Paused from MicrosoftDNS_Zone'
HRESULT: 0x80041001
Details: Generic failure
One or more workflows were affected by this.
Workflow name: Microsoft.Windows.DNSServer.2003.Monitor.ZoneRunning
Instance name: test.opsmgr.net (dc01.opsmgr.net)
Instance ID: {E0A3BD98-04B7-0C44-B26D-F8E6175456D1}
Management group: PROD1

Script or Executable Failed to run
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 21406
Description:
The process started at 6:26:59 AM failed to create System.Discovery.Data. Errors found in output:
C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 10\8675\DNS2003ComponentDiscovery.vbs(123, 9) SWbemServicesEx: Generic failure
Command executed: "C:\WINDOWS\system32\cscript.exe" /nologo "DNS2003ComponentDiscovery.vbs" {C984657D-0255-F11B-2C76-1542793A684D} {11056C4C-B933-98ED-3DC5-4B9AAE232B23} dc01.opsmgr.net true true true "" false 700 1 Working Directory: C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 10\8675\
One or more workflows were affected by this.
Workflow name: Microsoft.Windows.DNSServer.2003.Discovery.Components
Instance name: dc01.opsmgr.net
Instance ID: {11056C4C-B933-98ED-3DC5-4B9AAE232B23}
Management group: PROD1

Script or Executable Failed to run
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 21405
Description:
The process started at 3:58:21 AM failed to create System.Discovery.Data, no errors detected in the output. The process exited with 0
Command executed: "C:\WINDOWS\system32\cscript.exe" /nologo "DNS2003Discovery.vbs" {C8655A28-E27E-C6ED-B158-8569219A71A6} {89AC2E61-9144-4B94-9028-5A25F547213E} dc01.opsmgr.net false
Working Directory: C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 10\8515\
One or more workflows were affected by this.
Workflow name: Microsoft.Windows.DNSServer.2003.ServerDiscovery
Instance name: dc01.opsmgr.net
Instance ID: {89AC2E61-9144-4B94-9028-5A25F547213E}
Management group: PROD1

Event Type:              Error
Event Source:           Health Service Script
Event Category:        None
Event ID:                  1152
Date:                        5/19/2009
Time:                        11:18:48 AM
User:                        N/A
Computer:                DC01
Description:
DNS2003Discovery.vbs : The Query 'select * from MicrosoftDNS_Server' did not return any valid instances.
Please check to see if this is a valid WMI Query.. Generic failure

Getting lots of Script Failed To Run alerts? WMI Probe Failed Execution? Backward Compatibility Script Error?

2009-06-29T14:24:31Z

In OpsMgr 2007, it is likely that your most common alert is not really a MP based alert from a technology management pack…. it could be a built-in alert that a script failed, or WMI could not be accessed. This is because when WMI is broken on a machine, almost EVERYTHING fails to execute properly on that agent.

At a recent health check at a customer site, we found the top 5 alerts in his environment (by cumulative repeat count) were:

WMI Probe Module Failed Execution
Service Check Data Source Module Failed Execution
Backward Compatibility Script Error
Script or Executable Failed to run
Service Check Probe Module Failed Execution

Sometimes – these alerts are normal…. the server is busy, or someone rebooted it without putting it into maintenance mode and allowing the workflows to unload gracefully.

However, if you have a high repeat count on these, it is typically indicative of something seriously broken on that agent(s). Most of the time – the failure is in WMI. Many customers get frustrated with these script errors, because they see them as “false alerts” because they don't know how to resolve the root cause, and we just tell you “this action broke”, we don't tell you why. It is critical that you examine these alerts, however, because these alerts will indicate something seriously wrong with an agent, such as broken WMI/cscript/OS issue. If you ignore them, or disable them – you will never know that monitoring is not functioning 100%.

Generally – here is how I attack script/WMI failures.

1. If the repeat count is 0 or 1, I ignore these as random failures, and close the alerts from time to time.

2. If the repeat count is very high, then something is wrong with the agent, and needs remediation on the agent OS. Investigate the OpsMgr event log on the agent for Warning/Critical events – to see if a lot of workflows are failing due to this issue.

The FIRST thing I do – is to see if WMI is responsive. I run WBEMTEST, and connect to “root\cimv2”. I then hit “query” and execute a “select * from win32_operatingsystem” to see if it returns results, or an error. Next – I look at the namespace from the alert in SCOM…. perhaps it is “root\MicrosoftDNS”, or “root\CCM”. Then – I try and run the query that is failing from the alert.

If EITHER of the above connections/queries fail…. then I know what's wrong. WMI has a core issue, and I punt this to my platform or application team to fix it. Sometimes it needs a MOF recompile, sometimes it needs WMI service bounced or the OS bounced.

If these all appear to work correctly, or, the problem is resolved after a WMI service bounce, then re-appears later – check out the following:

There are many things you can do to resolve/remediate these issues. Here is a list of the most common fixes:

1. Apply http://support.microsoft.com/kb/933061 This resolves a LOT of issues on the Windows 2003 OS with WMI. This should be one of your first steps. This applies to x86 or x64 Windows Server 2003 SP1 or SP2.

2. Registry modification for WMI buffer thresholds (see below)

“HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\Low Threshold On Events (B)" to 35000000 (default is 10000000)
”HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\High Threshold On Events (B)" to 70000000 (default is 20000000)

The registry modification to WMI buffers increases the amount of objects that WMI can hold before injecting sleep delays to the WMI service.

3. Apply http://support.microsoft.com/kb/955360 This updates the Windows Scripting Host (cscript) to version 5.7. This resolves script timeouts, and scripts consuming a LOT of CPU during execution, and problems with multiple scripts running at the same time. This applies to x86 or x64 Windows Server 2003 SP1 or SP2. This is a very good hotfix for DNS servers, DHCP servers, and Domain Controllers. This has been seen to lessen the impact of VBscripts consuming a large amount of CPU during runtime.

Making these three modifications should resolve the majority of systemic issues out there, unless WMI is completely corrupt/unresponsive and needs repair. Sometimes, rebooting a server, or bouncing WMI will temporarily resolve these issues as well, if you cannot apply the fixes immediately.

If you have applied all three of these above, and are still experiencing a systemic repeat of a WMI query/script failure…. the next step would be to try running the query directly, accessing the namespace in WBEMtest. I’d like to hear about any experiences here.

Health Service and MonitoringHost thresholds in R2 – how this has changed and what you should know

2009-06-22T22:52:29Z

In THIS post – I described the way the agent HealthService will bounce on a regular basis, and how to alert on that, and change the thresholds. Please see that post for details on SP1.

Now – in R2 – much of this has changed in how it works. That said – the core challenge in SP1 still exists in R2:

1. The default agent threshold is 100MB for the HealthService and MonitoringHost process. That is too low for many of the typical agents in production environments.

2. When we bounce the agent for using more than 100MB, we do this silently, and do not alert. If your agents are constantly restarting in a loop, you will never know.

Lets take a look at R2:

First off – lets examine the HealthService class. There are two Monitors located at Health Service > Entity Health > Performance > Health Service Performance > Health Service State. They are for Handle Count threshold, and Private Bytes Threshold.

Health Service Handle Count Threshold: The default threshold is 2000 handles for an agent. There are built-in overrides for the Management servers – bumping this number up to 10,000 handles. Also, the new Native Exchange 2007 MP, bumps the threshold up to 5000, for Exchange 2007 computers. It is common that you *might* have to bump up this threshold for SOME agents. It is also common to bump the Management Server threshold up to 20,000 or even 50,000 if yours is constantly using more, but stable.

Health Service Private Bytes Threshold: The default is 100MB for all agents. There is a threshold override for Management servers to use up to 1.6GB. The new Native Exchange 2007 MP bumps the threshold to 600MB for Exchange 2007 servers. This is the monitor that needs the most attention! The default of 100MB is not enough for many server roles, especially if hosted on Server 2008 OS. You will likely need to override this monitor, for groups of Windows Computer objects, that are affected. Your agents will potentially be a in perpetual restart loop until this is done. Here is an example of mine – with a few overrides in place for SQL computers and DNS servers:

Now…. in R2 – the MonitoringHost workflows have changed, from rules, to Monitors. These are located under the “Agent” class. You will find them under Agent > Entity Health > Performance > Health Service Performance > Health Service State. They are named Monitoring Host Handle Count Threshold, and Monitoring Host Private Bytes Threshold.

***Note: In this view - you will also see the HealthService monitors, but note – these are inherited from the Health Service class. This is because there is a dependency rollup that rolls up the Health Service State of the HealthService, to the Agent. I will explain why in a moment.

Monitoring Host Handle Count Threshold: The default threshold is 2000 handles for an agent. There are built-in overrides for the Management servers – bumping this number up to 10,000 handles. Also, the new Native Exchange 2007 MP, bumps the threshold up to 5000, for Exchange 2007 computers. It is common that you *might* have to bump up this threshold for SOME agents. It is also common to bump the Management Server threshold up to 20,000 or even 50,000 if yours is constantly using more, but stable.

Monitoring Host Private Bytes Threshold: The default is 100MB for all agents. There is a threshold override for Management servers to use up to 1.6GB. The new Native Exchange 2007 MP bumps the threshold to 600MB for Exchange 2007 servers.

Now, in R2 – all four of these monitors roll their health state up to the Aggregate Roll-up monitor under the agent class, named “Health Service State”.

If we look at the properties of that aggregate monitor, we can see the recovery action to restart the HealthService is now on this monitor. Therefore – if ANY of the 4 monitors below it are in a critical state – they will roll up to this monitor, which will launch a script to bounce the HealthService on the agents.

This script, when it executes, will launch an event 6024 in the OpsMgr event log on the agent, that is is restarting the HealthService.

***NOTE – the text used in the event log is not technically accurate, in that it always states “Health Service exceeded Process\Handle Count or Private Bytes threshold.” It could be an issue with the Monitoring Host – NOT the HealthService, and this event might mislead you in troubleshooting. So just know that a 6024 event is a generic restart event – you need to look at the individual monitor state change history in Health Explorer to properly investigate.

So – to summarize the changes from SP1 to R2:

The MonitoringHost threshold rules are now standard monitors.
The Health Service monitors roll up to Agent - Health Service State rollup monitor.
The Health Service State rollup monitor has a recovery which runs a script to bounce the HealthService when it is in a critical state.
We still do not alert by default when the script bounces your agent, and you need to create a rule to look for this, or, alert on the state-change of the monitor.
You still will likely need to adjust the threshold of the Health Service Private Bytes monitor for many of your agents.

Lets talk about #4 above: You need to know when your agents are getting bounced, especially if they are caught in a loop of bouncing.

You have a few choices here…. but I like to either:

Create an alert rule, target “Agent”, Event ID 6024, in the OpsMgr event log.
Override the “Health Service State” rollup monitor, to “Generates Alert = True”.

Either one of those will give you a solution, that will detect the monitor state change, which results in bouncing the agent’s Health Service.

With regard to #5 above: You will likely need to adjust this default threshold for many agents. From my previous blog post on this topic – I have been seeing that mostly on the following types of servers:

Large SQL database servers
Server 2008 domain controllers
DHCP servers with large scope counts
DNS servers with large zone counts
Exchange 2007 servers
Large Exchange 2003 Mailbox servers
IIS7 (Server 2008) Servers
Proxy agents that perform special agent-less monitoring (Nworks/Vmware, etc…)

Create groups for these server types, and override the default threshold for this monitor for those groups. In general, I have found bumping to 250MB resolves most agent issues, but some special cases could need much more.

Using a Generic Text Log rule to monitor an ASCII text file – even when the file is a UNC path

2009-06-19T23:23:11Z

There are several examples in blogs on how to create a generic text log rule to monitor for a local text file (Unicode, ASCII, or UTF8).

This will be a step-by-step example of doing the same, however, using this to monitor the log file on a remote UNC path instead of a local drive. This is useful when we want to monitor a file/files on a NAS or an a share that is hosted by a computer without an agent.

This is a bit unique… instead of applying this rule to ALL systems that might have a specific logfile present in a specific directly – we are going to target this rule to only ONE agent. This agent will monitor the remote fileshare similar to the concept of a “Watcher Node” for a synthetic transaction. Therefore we will be creating this rule disabled, and enabling it only for our “Watcher”.

In the Ops console – select the Authoring pane > Rules.

Right click Rules, and select Create a new rule. We will chose the Generic Text Log for this example:

Choose the appropriate MP to save this new customer rule to, and click Next.

For this rule name – I will be using “Company Name – Monitor remote logfile rule”

Set the Rule Category to “Alert”

For the target – I like to use “Windows Server Operating System” for generic rules and monitors.

UNCHECK the box for “Rule is enabled”

Click Next.

The directory will be the UNC path. Mine is “\\VS2\Software\Temp”

The pattern will be the logfile(s) you want to monitor. We can use a specific file, such as “logfile.log” or a wildcard, such as “*.log”.

You should not check the “UTF8” box unless you know the logfile to be UTF8 encoded.

Click Next.

On the event expression, click Insert for a new line. Essentially – log file monitors look at each new line in a logfile as one object to read, and this is represented by “Params/Param[1]” This “Parameter 1” is the entire line in the logfile, and is the only value that is valid for this type of monitor – so just type/paste that in the box for Parameter Name.

Since we want to search the logfile line for a specific word, the Operator will be “Contains”.

For the value – this can be the word you are looking for in the line, that you want to alert on. For my example, I will use the word “failed”.

Click Next.

On the alert screen – we can customize the alert name if desired, set the severity and priority, and build a better Alert Description. If you are using SP1 – the default alert description is blank. If you are using R2 – the default alert description is “Event Description: $Data/EventDescription$” HOWEVER – this is an invalid event variable for this type of event (logfile)…. so we need to change that right away. I keep a list of common alert description strings HERE

For this – I will recommend the following alert description. Feel free to customize to make good sense out of your alert:

Logfile Directory : $Data/EventData/DataItem/LogFileDirectory$
Logfile name: $Data/EventData/DataItem/LogFileName$
String: $Data/EventData/DataItem/Params/Param[1]$

Click “Create” to create the rule.

Find the rule you just created in the console – right click it and choose “Properties”. On the Configuration tab, under responses (to the right of “Alert”) click Edit.

Click the “Alert Suppression” button. You should consider adding in alert suppression on specific fields of an alert – in order to suppress a single alert for each match in the logfile. If you don't – should the monitored logfile ever get flooded with lines containing “failed” from the application writing the log – SCOM will generate one alert for each line written to the log. This has the potential to flood the SCOM database/Console with alerts. By setting alert suppression here – we will create one alert, and increment the repeat count for each subsequent line/alert. I am going to suppress on LoggingComputer and Parameter 1 for this example:

Click OK several times to accept and save these changes to the rule.

Now – we created this rule as disabled – so we need to enable it via an override. I will find the rule in the console – and override the rule “For a specific object of class: Windows Server Operating System”

Now – pick one of these machines to be the “watcher” for the logfile in the remote share.

**Note – the default agent action account will make the connection to the share and read the file. In my case – the default agent action account is “Local System” so this will be the domain computer account of the “Watcher” agent which connects to the remote share and reads the file. This account will need access to the share, folder, and files monitored. Keep that in mind.

Set the override to “Enabled = True” and click OK.

At this point, our Watcher machine will download the management pack again with the newly created override, and apply the new config. Once that is complete – it will begin monitoring this file. You can create a log file in the share path, and then write a new line with the word “failed” in it. You need a carriage return after writing the line for SCOM to pick up on the change.

You should see a new alert pop in the console, based on matching the criteria. Subsequent log file matches will only increment the repeat count. Customize the alert suppression as it makes sense for you.

Then – create additional rules just like this – for different UNC paths.

Web Application recorder R2 – the recorder bar missing in IE?

2009-06-19T15:15:34Z

Sometimes getting the web application recorder to capture a web session can be a little tricky. I have blogged about some typical issues you might run into HERE

Something I noticed today, with R2:

When running the R2 console on an x64 machine – the web recorder bar is not coming up.

On my x86 machine – it was working just fine, however. I notice – when I go into IE settings, Tools, Manage Add-ons….. I see this on a working machine:

However – these add-ons are missing on my x64 based consoles.

The problem turned out to be…. that when you install the console on an x64 machine, it registers the x64 version of these add-ons. However – the IE browser launched by default by hitting “Start Capture”, is the x86 version. You have to manually launch the x64 IE shortcut – in order to use the web recorder browser.

So here are some steps to make this work:

1. Open the web application editor

2. Hit “Start Capture”

3. This will launch Internet Explorer in 32 bit mode. Close this browser.

4. From the start menu, run Internet Explorer (64-bit)

5. The web recorder will appear. (If not – choose View > Explorer Bars > Web Recorder)

From there you can record your session, hit stop, and it will populate the web application tool as normal. Just a minor inconvenience of closing one browser, and opening another.

Outages and Maintenance Report

2009-06-11T04:48:00Z

This is a little report I put together on request.

This report will query the data warehouse, and show all the outages, and maintenance, for objects in specific groups.

The outages look at all the “Failed to Connect to Computer” alerts, and list the start and end time of the outage, based on the time the alert was created, to the time it was closed (assumes it is auto-closed by the agent coming back online)

The maintenance looks at the times that the Health Service Watcher objects are placed into maintenance mode.

There is a start and end time parameters for the report – and the report defaults to the last 30 days.

There is a group choice parameter – you should pick a group that contains Health Service Watcher objects.

It looks like this:

It is based on this query, which you can tune to meet your needs:

declare @startdate datetime
declare @enddate datetime
declare @computergroups varchar(50)
SET @startdate = '2009-06-01 00:00:00.000'
SET @enddate = getutcdate()
set @computergroups = 'Agent Managed Computer Group'

select
apv.ParameterValue as SystemName,
(DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),av.RaisedDateTime)) as DownDateTime,
(DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),arsv.StateSetDateTime)) as RestoredDateTime,
adv.CustomField2 as OutageType,
adv.CustomField3 as RootCause,
adv.CustomField4 as Reason,
adv.DBLastModifiedByUserId as UserID
FROM Alert.vAlert av
JOIN Alert.vAlertDetail adv on av.AlertGuid = adv.AlertGuid
JOIN Alert.vAlertResolutionState arsv on av.AlertGuid = arsv.AlertGuid
JOIN Alert.vAlertParameter apv on av.AlertGuid = apv.AlertGuid
WHERE AlertName = 'Failed to Connect to Computer'
AND arsv.ResolutionState = '255'
--AND adv.CustomField2 IS NOT NULL
AND (DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),av.RaisedDateTime)) between @startdate and @enddate
and apv.ParameterValue IN (
SELECT vManagedEntity.DisplayName
FROM vManagedEntity
INNER JOIN vRelationship ON vManagedEntity.ManagedEntityRowId = vRelationship.TargetManagedEntityRowId
INNER JOIN vManagedEntity AS ManagedEntity_1 ON vRelationship.SourceManagedEntityRowId = ManagedEntity_1.ManagedEntityRowId
WHERE (ManagedEntity_1.DisplayName = @computergroups)
)

UNION ALL

select
vme.displayname,
(DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),vmm.StartDateTime)) as DownDateTime,
(DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),vmm.EndDateTime)) as RestoredDateTime,
'OutageType' =
CASE
vmm.PlannedMaintenanceInd
WHEN '1' THEN 'Scheduled'
WHEN '0' THEN 'Unscheduled'
END,
'RootCause' =
CASE
vmmh.ReasonCode
WHEN '0' THEN 'Other (Planned)'
WHEN '1' THEN 'Other (Unplanned)'
WHEN '2' THEN 'Hardware: Maintenance (Planned)'
WHEN '3' THEN 'Hardware: Maintenance (Unplanned)'
WHEN '4' THEN 'Hardware: Installation (Planned)'
WHEN '5' THEN 'Hardware: Installation (Unplanned)'
WHEN '6' THEN 'Operating System: Reconfiguration (Planned)'
WHEN '7' THEN 'Operating System: Reconfiguration (Unplanned)'
WHEN '8' THEN 'Application: Maintenance (Planned)'
WHEN '9' THEN 'Application: Maintenance (Unplanned)'
WHEN '10' THEN 'Application: Installation (Planned)'
WHEN '11' THEN 'Application: Unresponsive'
WHEN '12' THEN 'Application: Unstable'
WHEN '13' THEN 'Security Issue'
WHEN '14' THEN 'Loss of network connectivity (Unplanned)'
END,
vmmh.Comment as Reason,
vmmh.userid as UserID
from vMaintenanceMode vmm
join vManagedEntity vme on vmm.managedentityrowid = vme.managedentityrowid
join vMaintenanceModeHistory vmmh on vmm.maintenancemoderowid = vmmh.maintenancemoderowid
where vme.FullName LIKE '%HealthServiceWatcher%'
and (DATEADD(hh,DATEDIFF(hh,getutcdate(),getdate()),vmm.StartDateTime)) between @startdate and @enddate
and vme.displayname IN (
SELECT vManagedEntity.DisplayName
FROM vManagedEntity
INNER JOIN vRelationship ON vManagedEntity.ManagedEntityRowId = vRelationship.TargetManagedEntityRowId
INNER JOIN vManagedEntity AS ManagedEntity_1 ON vRelationship.SourceManagedEntityRowId = ManagedEntity_1.ManagedEntityRowId
WHERE (ManagedEntity_1.DisplayName = @computergroups)
)
ORDER BY DownDateTime

The report is attached below.

Creating custom dynamic computer groups based on registry keys on agents

2009-06-10T04:09:48Z

I have had a few requests now for this, so I thought I would take the time to write up the process.

Lets say I have three support levels of servers:

Level 1 – servers critical to business operations (ex: customer facing web applications, SQL back-ends)

Level 2 – important servers (ex: messaging, internal apps)

Level 3 – non-essential servers (ex: non-critical or highly redundant internal apps)

Lets say we want to create overrides for certain rules… where we will page on anything in Level 1 group, email notify on Level 2 group, and simply alert for Level 3. Possibly we want to create views, and only see alerts for Level 1 servers. Perhaps we wish to scope users so they only see Level 1 and Level 2 servers in the console?

Well – the first step is to place these servers into groups.

Sure – we can do this manually, with explicit assignments to the group. But that is resource intensive over time, and we might miss one down the road. I’d prefer to dynamically create the groups of Windows Computers based on a name…. but this can be difficult sometimes – where we don't have a solid naming scheme, or other criteria to group by.

I will demonstrate another way to accomplish this… by coming up with a business process to use a registry key on your managed servers, and collect this registry attribute with SCOM. Then – use this Registry attribute for dynamic group memberships.

Ultimately – there are three simple steps to this process:

1. Create registry keys on agents.

2. Extended a class with an attribute, to discover the registry keys and values.

3. Create dynamic groups based on the attribute values from the registry.

It is just that simple.

To get started – lets talk about our custom registry key. For this example, I am going to create a new Key at HKLM\Software\ and call it “CompanyName”

Next – in that key – I will create a new DWORD Value, named “SupportLevel”

Lastly – I will assign a numeric value to “SupportLevel” on each server, either 1, 2, or 3.

In my environment…. my Hyper-V servers are critical. They host all of my VM’s, including many business critical applications. Therefore – they will get Level 1.

My Exchange 2007 servers handle all my mail traffic and notifications, so I will set their registry value to Level 2.

My Exchange 2003 servers have been retired – for MP testing only… so we will set those to Level 3.

Here is a table that shows what I am planning:

ServerName	SupportLevel
VS1	1
VS2	1
VS3	1
EX1CLN1	2
EX1CLN2	2
EXCAS	2
EX2CLN1	3
EX2CLN2	3
OWA	3

So – I get all my registry values set on all computers. This is a big job at first, but it is a one time deal, and you can even script it if you are handy.

Next… we need to discover these registry entries in SCOM, as attributes of a class. Then were can use that attribute to group objects. Since I want Windows Computer objects in my groups (Windows Computer is a good object for most overrides, scoping, notifications…etc..) we would like to have these attributes added to the Windows Computer class.

However – there is a problem. The Windows Computer object is in a sealed MP. We cannot just add information to that class as we would like. Therefore – OpsMgr allows us to “Extend” an existing class… and add our custom attributes to it. This “Extended” class is basically a copy of the existing class… it will have all the built in attributes of Windows Computer, and will also have our custom attribute properties. It’s is easier to see it than to talk about it.

First – in the Ops console – authoring pane – go to Attributes. Create a new attribute. I am going to call this one “SupportLevel”

Next – choose “Registry” for the discovery type.

Next – We need to pick the Target class. We want Windows Computer. Note – this will create a new class, named “Windows Computer_Extended” by default. We can use this name, or you can rename this whatever you want. It is your class. I will leave it at the default.

Most important! Management Pack location.

This is CRITICAL. Spend some time making sure you are creating these attributes in the correct location. If you leave this MP unsealed XML…. then any groups you create that use these attributes, will have to be placed in this same MP. Then – if you use these groups for Overrides – those overrides will be force to go in this same MP. There is a “cardinal rule” in SCOM… objects in one unsealed MP cannot reference another unsealed MP. So – we cannot have a group in one unsealed MP, and then use that group for an override in another unsealed override MP.

So – we have two choices.

1. Keep an unsealed MP… and live with the fact that attribute, group, and override will all have to be placed here.

2. Create the attribute and the dynamic group in the MP, then seal it. Then – you can use this group in ANY of your override MP’s… for Exchange, SQL, etc…

I strongly recommend option #2 for this exercise… but you can make this decision for yourself.

Ok…. I will choose Option #2 (seal the MP), so I will create a new MP just for this extended class, and groups.

On the next screen – we can put in our registry information:

In this example – I am looking for a registry Value (1, 2, or 3), and my attribute type is “Int” for integer.

For the frequency, set this to a reasonable frequency to discover you machines as they come on to you network. Typically, once per day is sufficient (86400 seconds) Remember – this will run against ALL your Windows Computers… so never set this more frequent than once per hour… that creates unnecessary overhead.

Ok – lets examine our work!

Go to Monitoring, Discovered Inventory, and change target type to our new class “Windows Computer_Extended”

If you do this quickly – you may find it is empty. This is what is happening behind the scenes: All Windows Computers are now downloading our newly created MP. They are going to run the registry attribute discovery, and submit their discovery data to the management server. The Management Server will insert this discovery data in the database. Over time, you will start to see all your Windows Computers pop into this class membership. You will notice a new attribute now, in addition to all the existing Windows Computer attributes. This attribute is “SupportLevel” and will be 1, 2, 3, or empty… depending on what each agent find in the registry.

Now – I set my registry discovery to once per day…. so I will need to wait 24 hours before I can expect all my healthy agents to show up in this list. To speed things up – I am going to bounce the HealthService on these example agents. (Agents run all discoveries when a HealthService restarts, and then on their frequency schedule)

Here is an example a few minutes after bouncing the HealthService on some agents:

Next on the list – create the groups. I will create these in the same MP that the attributes exist in.

I will call my first group “CompanyName – Support Level 1 Servers Group”. I like to append the word “Group” to all groups I create as a best practice. This helps us determine this group class is actually a group when we see it in the list of classes in the UI. I sure wish all MP authors would take this to heart, since every group is actually a singleton class.

On the dynamic members screen – I will fins my “Windows Computer_Extended” class – and click Add. What we now see – is that we have a new attribute to use, “Support Level”

I will set this group to “SupportLevel Equals 1” and click OK.

Now – I can right-click my new group – and choose “View Group Members”

Yee-haw! It works! Now – I simply repeat this above step – creating groups for SupportLevel 2, and 3.

Now – that is done. This is the area, that I recommend we stop… take a breather…. then seal the MP. If you seal the MP – we will be able to use the groups for overrides in any other override MP. If you choose not to seal the MP now… any overrides you use the groups for – will be forced into this same MP. Please keep that in mind.

Since I am harping on sealing the MP…. I am going to do a quick example of just that. Jonathan Almquist has an excellent tutorial on sealing MP’s HERE and we will use his example.

**Note – when running the sn.exe commands to create our key…. we only need to do this one… not every time we want to seal an MP.

***Critical note – you need to keep a backup of this key… because it will be required for making updates to this MP in the future, re-sealing, and keeping the ability to upgrade the existing MP in production.

So, I create the folders, create the key using sn.exe, copy over the referenced MP’s from the RMS, and now I am ready to seal.

MPSeal.exe c:\mpseal\input\CompanyName.SupportLevel.MP.xml /I "c:\mpseal\mp" /Keyfile "c:\mpseal\key\PairKey.snk" /Company "CompanyName" /Outdir "c:\mpseal\output"

Works great.

Now – I can delete my unsealed MP from the management group, and import my sealed MP.

Phew. All the heavy lifting is done. Now… I have my groups… I can start setting up overrides using these groups, or scoping notifications.

On my Support Level 1 group – I will use this to set up my pager Notification subscriptions to only page based on specific classes, and this group.

On my Support Level 2 group – I will use this to override important alerts to High Priority… because I am using High Priority as a filter for email notifications, per my previous blog post here: http://blogs.technet.com/kevinholman/archive/2008/06/26/using-opsmgr-notifications-in-the-real-world-part-1.aspx

On my Support Level 3 group – I will use this group for tweaking/disabling rules and monitors for the group… turning off discoveries so they don't discover lab servers, scoping views, etc.

Maybe in my next post…. I will build on this MP… and show a really simple way to add the Health Service Watcher objects to these dynamic groups… for each Windows Computer object that is in the group – so we can use these groups for Heartbeat failure notifications.

Maintenance mode – tying the text of the category to the database

2009-06-05T05:59:36Z

I havent seen this discussed before – so I figured I would post this.

In the OpsDB and DWDB – we keep some tables names MaintenanceMode and MaintenanceModeHistory.

When you place an object into maintenance mode – we will log a row in the database for this object. You could potentially write reports against this data in the data warehouse, and report on MM history.

From the following query:

select * from dbo.vMaintenanceModeHistory

There is a column named “ReasonCode” This has a numeric value. However – in the UI – this correlates to the “Category” that you must select when you place an object into MM:

Here is a table which sorts out the numeric reason code vs the text in the UI:

Other (Planned)	0
Other (Unplanned)	1
Hardware: Maintenance (Planned)	2
Hardware: Maintenance (Unplanned)	3
Hardware: Installation (Planned)	4
Hardware: Installation (Unplanned)	5
Operating System: Reconfiguration (Planned)	6
Operating System: Reconfiguration (Unplanned)	7
Application: Maintenance (Planned)	8
Application: Maintenance (Unplanned)	9
Application: Installation (Planned)	10
Application: Unresponsive	11
Application: Unstable	12
Security Issue	13
Loss of network connectivity (Unplanned)	14

Some Server 2008 Windows Firewall rules needed to install OpsMgr R2

2009-06-01T23:03:03Z

I recently rebuilt a POC environment all on Server 2008, and SQL 2008.

Out of the box – the R2 install is a bit rocky – because of the Windows Firewall enabled on Server 2008 by default.

These required custom firewall rules are documented on the supported configuration guide…. located at http://technet.microsoft.com/en-us/library/bb309428.aspx

I just want to provide a little context around them.

My three servers are OMRMS, OMDB, and OMDW.

OMRMS – Root Management Server

OMDB – OperationsManager database role with SQL 2008 Database Services.

OMDW – OperationsManagerDW database role, Reporting Role, with SQL 2008 Database Services and Reporting Services

First off – during the install of the RMS, it will fail to locate the SC database. This is because it tries to contact the SQL database server on OMDB over port 1433, and this is blocked by the Windows Firewall on OMDB. To resolve – I need to allow the SQL server program access through the firewall:

I create a new rule – called “Custom - SQL DB Engine Program” and choose rule type of program – and give it the path to SQLServer.exe – in my case: E:\Program Files\Microsoft SQL Server\MSSQL10.I01\MSSQL\Binn\sqlservr.exe

In addition – I am using a named instance of SQL. When we use a named instance of SQL, the DB Engine instance will not use port 1433 by default. It will pick a random port and assign it to the DB engine instance. When clients connect to this instance – they do not know this random port…. therefore – they can leverage the SQL broker service – which will communicate the random port to the SQL client for communication.

Therefore – I need another rule on the firewall of the SQL server: this time a port based rule – allowing 1434 UDP.

Here is a screenshot of my two access rules:

Now – the RMS install is able to locate the SC database – and we continue.

Next up – I perform the reporting install on OMDW. I install the DW and the Reporting server roles here. The install goes fine. However – the RMS starts logging the following events:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/1/2009 5:08:19 PM
Event ID:      31551
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Failed to store data in the Data Warehouse. The operation will be retried.
Exception 'SqlException': A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance name: OMRMS.opsmgr.net
Instance ID: {E61C0DD2-AB2A-4CFA-D252-37B84BCF9A83}
Management group: PROD1

This is because the RMS cannot contact SQL on the data warehouse SQL server. Therefore – I need to add the same two firewall rules on OMDW that I enabled on OMDB…. to allow remote SQL connections.

Now – I have installed the RMS, and I have installed reporting into the management group. However – when I click on “Reporting” in the SCOM console – I get the following error:

Date: 6/1/2009 5:16:31 PM
Application: System Center Operations Manager 2007 R2
Application Version: 6.1.7221.0
Severity: Error
Message: Loading reporting hierarchy failed.

System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.10.10.7:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.ReportingService.ReportingService2005.ListChildren(String Item, Boolean Recursive)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.ManagementGroupReportFolder.GetSubfolders(Boolean includeHidden)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.ManagementGroupReportFolder.GetSubfolders()
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.WunderBar.ReportingPage.LoadReportingSubtree(TreeNode node, ManagementGroupReportFolder folder)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.WunderBar.ReportingPage.LoadReportingTree(ManagementGroupReportFolder folder)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.WunderBar.ReportingPage.LoadReportingTreeJob(Object sender, ConsoleJobEventArgs args)
System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.10.10.7:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

See the bolded blue part above. The console is not able to establish a connection to the HTTP reporting server address.

Additionally – you will see the following in the RMS event log:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/1/2009 5:40:22 PM
Event ID:      31569
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Report deployment process failed to request management pack list from SQL RS Server. The operation will be retried.
Exception 'WebException': Unable to connect to the remote server

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Report
Instance name: OMRMS.opsmgr.net
Instance ID: {E61C0DD2-AB2A-4CFA-D252-37B84BCF9A83}
Management group: PROD1

This is all due to the fact that the Windows Firewall is blocking port 80 (HTTP) by default. We need to open another access rule – for TCP port 80 for this:

Now – I go ahead and open it for TCP 80, and TCP 443, since I might lock this down using SSL in the future.

Once the HTTP rule is enabled on the Reporting server role – the RMS almost immediately throws the following events:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/1/2009 5:51:42 PM
Event ID:      31570
Task Category: Data Warehouse
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Report deployment process successfully requested management pack list from SQL RS Server

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Report
Instance name: OMRMS.opsmgr.net
Instance ID: {E61C0DD2-AB2A-4CFA-D252-37B84BCF9A83}
Management group: PROD1

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/1/2009 5:52:27 PM
Event ID:      31568
Task Category: Data Warehouse
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Reporting components successfully deployed

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Report
Instance name: OMRMS.opsmgr.net
Instance ID: {E61C0DD2-AB2A-4CFA-D252-37B84BCF9A83}
Management group: PROD1

A quick check of the Reporting tab of the console now yields:

And I wait an hour or so – and go back to the RMS – and make sure the RMS OperationsManager event log is clean – before doing anything else to the management group.

My experience upgrading to OpsMgr R2 RTM

2009-05-23T17:01:23Z

I upgraded my test lab from SP1 to R2-RTM this weekend.

My current test lab consists of the following servers:

OMRMS – Server 2003 - RMS role

OMMS3 – Server 2008 - MS role, Web Console

OMMS – Server 2003 - MS role, ACS collector

OMDB – Server 2003/SQL 2005 - OperationsManager Database

OMDW – Server 2003/SQL 2005 - OperationsManagerDW database, Reporting, SRS, ACSDB roles

There are 18 agents reporting to this management group.

So – I start – with a little light reading.

I begin with the release notes. These are available from the R2 CD, and on the web at Operations Manager 2007 R2 Release Notes I dont see anything in there that is terribly applicable to me…. but these are good to commit to short term memory – in case we hit a snag during/after the upgrade.

Next – I move on to the Upgrade guide. This is available on the Technet Library – at Operations Manager 2007 Upgrade Guide I need to spend a little time on this one, mapping out the pre-upgrade steps, and then planning the order of my upgrade based on how my management group is deployed.

So – I start by running down the pre-upgrade checklist at: Preparing to Upgrade Operations Manager 2007

I record my service accounts, make sure my DB’s have plenty of free space, and my t-logs are sized big enough. I make sure the volume with TempDB has plenty of free disk space in case TempDB needs to auto-grow.

Next – I map out my plan – and order of operations, for my management group, and share the plan with my team:

Get most recent backup of Database, Encryption key and Export unsealed MP’s for safekeeping.
Go to pending actions – and reject/remove anything in there.
Verify free space on SQL database and validate log size is appropriate.
I need to uninstall the agent from OMTERM – my terminal server which has a console and an agent only. I decide to go ahead and uninstall the agent, the console, and the SP1 authoring console as well, since I will be replacing it with the R2 auth console. I will replace the agent and consoles when the upgrade is complete for the management group.
I need to disable all my notification subscriptions, and disabled my product connectors. I am running a custom internal product connector – which runs as a service and updates alert properties – so I will stop and disable that service for the duration of the upgrade.
I see a section on Improving Upgrade Performance so I will add that step here – right before I upgrade the first component.
I am now ready to establish the upgrade order for my management group – this is available at: Planning your Operations Manager 2007 Upgrade
RMS (OMRMS)
Reporting Server (OMDW)
Stand Alone Consoles (None – I uninstalled this already in my case)
Management Servers (OMMS3, OMMS)
Gateway Servers (None)
Agents
Web Console (on OMMS3 and OMMS)
Post-Upgrade validation steps

Ok – that's my plan. Time to get rolling.

The SP1 to R2 steps are outlined here: Upgrading from Operations Manager 2007 SP1 to R2

I know from experience with customers – the success of your upgrade HINGES on how well you read AND follow the upgrade steps – VERBATIM. The majority of issues we see (especially on clustered RMS) are when a customer does not follow the steps exactly as written, in the correct order.

I complete steps 1-7 in the plan above, and then start the RMS upgrade at step 8. I run “SetupOM.exe” and kick off the pre-req checker before starting the install, where I hit my first snag. I need to install WS-Management v1.1, because I do plan on monitoring Unix/Linux machines in the future with this management group. (This was documented in the release notes, and in the upgrade guide – so I was expecting this… I should have added this to my plan) So I install WS-man from the link provided in the pre-req, which just takes a few minutes. Now – it looks much better in the pre-req checker:

The install instructions provided on TechNet are very straightforward. The install took about 20 minutes for my small environment. It waited the longest on “Loading Management Packs” on the screen in my environment. It finally ended with an error:

The guide has a note on this – about the fact you might get a warning that a service failed to start – and to hit OK. However – this is a different error – this is a service failing to stop… I click OK, and then a few minutes later – setup completes. I uncheck the box to start the console and to backup the encryption key.

I then ran the RMS upgrade validation steps – checking the registry and the services. Registry setup version shows me all is good.

***Note: We have changed the service display names for R2. See below:

I moved on to Reporting. My SRS, Reporting, and DataWarehouse are all shared on a single server – OMDW.

As I read the guide at Upgrading from Operations Manager 2007 SP1 to R2 I notice this little tidbit – which needs to be given STRONG attention before I kick off the upgrade:

Prior to running the upgrade on the Reporting server, you must remove the Operations Manager 2007 agent; the upgrade will fail if this is not done.

So – I kick off the uninstall of the agent on the Reporting/SRS server (OMDW in my case) from Add/Remove programs – before I start the upgrade. Missing little steps like this will drive you nuts if you aren't methodical.

After the agent uninstall – I pick back up on the guide – and kick off “SetupOM.exe”. Since I am a freak – I go ahead and run a pre-req check just to make sure all is good:

Moving on…. I start the install according to the guide. The install goes without a hitch, and took about 10 minutes to complete.

Next up – Management servers. I start with OMMS3. I hit the pre-req check – and I notice I already have WS-Man installed – so away I go. The installer immediately failed with a pre-req failure. I realized – I have the web console installed on this management server, and I forgot to add that when running the pre-req check manually. When I do – I see:

So – I need to grab the ASP.NET Ajax extensions…. this is to support the new cool health explorer in the Web Console. I click “More” on the pre-req check – which gives me a link to the download.

After this little hurdle – the management servers upgraded very quickly. Once again – I got an expected error about a failure to stop a service.

Click ok and setup completes. I repeat this upgrade on the other management server (OMMS) and these are done. A quick check of the registry – and the setup version is indeed 6.1.7221.0

I don't have any gateways in this lab – so next up is agents.

Lucky me – all 18 agents show up in pending actions for an update. I will approve them all – and let the management server push the update down and upgrade them.

***Note – do not upgrade more than 299 agents in this manner at a time. This is documented in the Upgrade Guide.

All my agents upgraded successfully except for two. BOTH that failed happened to be the two servers that I manually removed the SP1 agent from – OMTERM and OMDW. (I forgot to delete their “agent managed” object from the management group) Both have a different error. OMTERM is failing to install with a push failure for MOMAgentInstaller. I have had trouble with this agent before – possibly because of the TS role - so I just do a manual agent install here. OMDW is different – the console push said it was a success – however – the System Center Management Service (HealthService) will not start – it gives an error:

Event Type:    Error
Event Source:    Service Control Manager
Event Category:    None
Event ID:    7024
Date:        5/23/2009
Time:        1:09:37 AM
User:        N/A
Computer:    OMDW
Description:
The System Center Management service terminated with service-specific error 2147500037 (0x80004005).

I ran a repair action from the console – but got the same error here. So – I manually uninstalled the broken agent – and deleted the agent from the Agent Managed section of the console – and re-pushed the agent. I had a little trouble getting these two to come into the management group… but eventually after a couple delete/reinstalls they finally appear to be working ok. I’d recommend uninstalling them from the console next time…. so this will remove both the agent and the computer object from the console.

Next on the list: Web Console

From the upgrade guide I see this note….

If your Web console server is on the same computer as a management server, the Web console server is upgraded when the management server is upgraded, rendering this upgrade procedure unnecessary. You can still run the verification procedure to ensure that the Web console server upgrade was successful.

Good – my web console is not a stand-alone – it was running on a management server (OMMS3) so that is already taken care of.

Aha – I found something we forgot on the plan…. the ACS Collector. This role is missing from the table at Planning your Operations Manager 2007 Upgrade so I completely missed this as a planning step. However the process is documented at Upgrading from Operations Manager 2007 SP1 to R2. So – we need to do this – I will assume last since it is last on the upgrade detailed steps. Following the guide…. I walked through the steps – no issues.

Looks like we are done! I will now start the post-upgrade validation steps to make sure my management group is actually working as it should without any major issues.

There is a list of post-upgrade checks at Completing the Post-Upgrade Tasks

I am going to walk through those here:

1. I open up discovered inventory – and change target to “Health Service Watcher” and compare this to the list I had before the upgrade. These are agents that have a problem from the management server perspective – which causes them to appear “grey” in all other views. My list is the same as before I started – I have 6 in this list as critical – 5 of them are agents that are VM’s that are currently down – so this is good. 1 of them is an old management server… for some reason we don't groom these out of the view/database – and these seem to stick around forever in this view.

2. I review the event logs on the RMS and all MS roles. I am seeing some errors like below:

Event Type:    Warning
Event Source:    HealthService
Event Category:    Health Service
Event ID:    2120
Date:        5/23/2009
Time:        10:02:15 AM
User:        N/A
Computer:    OMRMS
Description:
The Health Service has deleted one or more items for management group "OPS" which could not be sent in 1440 minutes.

This is normal – it happens when you have agents that are down in your environment.

Event Type:    Error
Event Source:    Health Service Modules
Event Category:    Data Warehouse
Event ID:    31552
Date:        5/23/2009
Time:        10:03:38 AM
User:        N/A
Computer:    OMRMS
Description:
Failed to store data in the Data Warehouse.
Exception 'SqlException': Sql execution failed. Error 777971002, Level 16, State 1, Procedure StandardDatasetGroom, Line 303, Message: Sql execution failed. Error 2812, Level 16, State 62, Procedure StandardDatasetGroom, Line 145, Message: Could not find stored procedure 'KMS_EventGroom'.

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.StandardDataSetMaintenance
Instance name: KMS Activation Event Data Set
Instance ID: {800D8126-6F72-CA84-A76B-A94F7E3C93CF}
Management group: OPS

This is not normal – this looks like an issue with the KMS MP – and R2’s advanced logging is picking up on an error that's been there all along, I just didn't know it.

That is all from the RMS – pretty clean. On the Management servers…. I found a bit more – but they were all due to the problems I was having with a handful of agents. Once I removed and fixed those agents – the MS logs are clean.

3. No cluster in this lab – so nothing to test there.

4. Review alerts in the console. I sort by Repeat Count and LastModified (I add these to all my alert views) and look for anything that stands out as repeating a LOT, or something new that looks like a problem. I dont see anything here – so that is good!

5. DB server in perfmon looks good. I examine % Processor Time, and Logical Disk Avg disk sec/read and Avg disk sec/write. Those are both avg under 15ms (.015) on the DB and log volumes - so that looks good. CPU is avg under 25%.

6. Check all the console views. Much snappier than in SP1. Nice.

7. I opened up reporting – and ran the “Microsoft ODR Report Library > Most Common Alerts” report – to test out reporting. It ran with no issues. I test a few of my saved custom and favorite reports – no errors – all good.

8. Authoring pane looks good – I can see my groups, monitors, rules – and wow – they open a LOT faster than before. Very nice.

9. I check out my MP versions. The install upgraded all my core MP’s to 6.1.7221.0. I was already pretty current on my MP’s – so not much to do here now that needs my immediate attention.

10. Re-enable notification subscriptions and product connectors. I turn my subscriptions back on – and fire off a test event that I use to generate an alert and email me a notification. Works great. Next – I got to my custom product connector – and enable the service and start it back up again. I run some test alerts – to make sure my product connector is taking all the necessary actions on the alerts – and forwarding them as appropriately. All good.

11. Review My Workspace. Yep – all my old custom views are there.

12. Re-deploy agents. I already did this. Perhaps I should have waited on this step…. because I spent so much time troubleshooting those last few pesky agents that seem to have trouble.

13. Oh – the BIG ONE. This step is a bit odd – we tell you to go run this SQL query. LET ME WARN YOU – this is not a “quick job”. This is the script that is documented and discussed at my blog post: Does your OpsDB keep growing- Is your localizedtext table using all the space- Dont take this step lightly – running this script could take several hours – so plan accordingly. Read the link above for the details – and consider skipping this step for now…. until you are sure you are ready to execute it. Take some calculations based on the blog post above – how long it will take – how severely you are impacted (row count of your localizedtext table) and make sure you have a LOT of free space for the tempDB and tempDBlog to grow if needed. My LT table was already really small – so no issues for me running this – it completed in less than a minute.

Done! (with the “official” steps)

Now – I just have a couple cleanup steps I need to do – like go back and install the Ops Console and the Auth Console back on my terminal server. Did that without issue. All looks good.

And then I realized – we are missing another step in our plan – under the post-upgrade tasks – make sure the web console is working! I saw lots of items in the release notes about how this might break…. and I imagine someone will complain rather quickly if it isnt working – so we better go check that out.

Sweet! I hit up the web console and it is all good. I check out several of the new views – and run health explorer from the web console. I have tasks, maintenance mode, and health explorer. Very cool. I event execute some of my favorite reports under “My Workspace” just to make sure those are good – ouch – not working. I will have to look into that one.

Ok – that’s enough for today. All in all – a successful upgrade. A good plan written out at the beginning, based on the upgrade guide - makes all the difference.

New web based forums for OpsMgr – no more NNTP newsgroups

2009-05-22T22:37:46Z

Getting help from the community for OpsMgr just got easier.

We released the new interface to OpsMgr forums – on the web. No more NNTP newsgroups to parse, with a hard to use web interface.

http://social.technet.microsoft.com/Forums/en-US/category/systemcenteroperationsmanager

Check it out!

SCOM R2 hits RTM!

2009-05-22T17:08:28Z

New Functionality - Operations Manager 2007 R2 introduces key new and enhanced functionality, including:

Enhanced application performance and availability across heterogeneous platforms

Delivers monitoring across Windows, Linux and UNIX servers–all through a single console.
Extends end to end monitoring of distributed applications to any workload running on Windows, UNIX and Linux platforms.
Maximize availability of virtual workloads with integration with System Center Virtual Machine Manager 2008.

Improved management of applications in the data center

Delivers on the scale requirements of URL monitoring of your business.
Meet agreed service levels with enhanced reporting showing application performance and availability.
More efficient problem identification and action to resolve issues.

Increased speed of access to information and functionality to drive management

Faster load times for views and results.
Improved and simplified management pack authoring experience

Where and when can I obtain the bits?

The RTM release is build 7221.

At General Availability (July 1)

Customers can obtain the bits from the MVLS website.
New customers can obtain the trial bits from the Operations Manager 2007 R2 website or directly @ http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=93ddf25b-1ef0-4851-81b0-5fb9a2f76181#tm

Where can I find collateral, training, and more on Operations Manager 2007 R2?

Released on the TechNet: Webcast Series on Operations Manager 2007 R2:

How else can I extend Operations Manager 2007 R2?

Service Level Dashboard v2 from the Solution Accelerators team lets you measure and report application or system performance & availability in near real time across your organization. Using the Dashboard, you can easily spot trends and head off problems—before they occur. The Dashboard also lets you create role-specific dashboards to support different departments, like HR, Finance, or Operations. Download it today from Microsoft Connect.
Operations Manager 2007 R2 Interoperability Connectors provide the ability to synchronize alerts and status between Operations Manager 2007 R2 and other management systems. Beta connectors for Tivoli Enterprise Console, HP OpenView Operations, and the new Universal Connector can be obtained from the Operations Manager R2 download on Connect. Download the Interop Connectors from the System Center Catalog.
Operations Manager 2007 R2 Visio Add-in delivers the ability to link status and health information gathered by Operations Manager 2007 R2 into normally-static Visio diagrams, adding life and interaction to those diagrams. Download it today from Microsoft Connect.
New Exchange Server 2007 Management Pack (MP) Beta, which provides enhancements over the current Exchange MP such as reducing alert noise and enhanced performance. Download it today from Microsoft Connect.
BridgeWays MP Beta Program, providing beta MPs for MySQL, Apache, and Oracle running on Windows, Linux or Solaris. For more information, and to register into the BridgeWays MP Beta Program, visit http://www.bridgeways.ca/bw_management-pack-beta-program-signup_form.php

Quick tip – using regular expressions in a dynamic group

2009-04-21T19:29:12Z

Here is a quick tip on using a regular expression when creating a group.

OpsMgr dynamic inclusion rules are case sensitive.

If I have a group that I want to contain all computers that START with “OM”…. I can use the following expression:

The “^” tells regex to start a new line… without this – it will search the entire computername for “OM” instead of only servers with a name that starts with “OM”. This shows the following group membership:

However – this is not all my “OM” servers. This is because when some were built, they used a lower case name. If I change the dynamic inclusion expression to “^om” I will only see the lower case servers (the NetBIOS Computer name is lower case… not the display name as shown):

Ok – how do I get them ALL in the same group?

There are a couple ways.

One way is to “OR” the two regex queries:

Bingo!

Another way – courtesy of Tim Helton – is to use the regex modifier (?i:) to make it case insensitive:

Which yields:

This should make dynamic groups a bit easier. We can expand this to add the HealthService Watcher objects, which is typical for groups that we will scope notifications to:

Which yields all my Windows Computers, and their Health Service Watcher components.

How to find a specific rule/monitor/discovery in the console – when all you have a more cryptic ID in an alert

2009-04-17T18:58:53Z

Sometimes – we will get a Script Error alert, or WMI failure Alert, or some generic alert, telling us that some OTHER workflow had a problem. These alerts are NOT the root cause… the root cause is the workflow that gets referenced in the alert.

The problem is – the alert often gives us a Rule/Monitor/Discovery name, that is not the same as the Common Display Name of the workflow in the console.

For instance – in R2 – we have a new rule that will temporarily disable a workflow that is flooding alerts. Here is an example of that alert:

Alert generation was temporarily suspended due to too many alerts
A rule has generated 50 alerts in the last 60 seconds.
Usually, when a rule generates this many alerts, it is because the rule definition is misconfigured.
Please examine the rule for errors.
In order to avoid excessive load, this rule will be temporarily suspended until 2009-04-17T10:54:33.2764287+01:00.
Rule: Microsoft.SystemCenter.GenericNTPerfMapperModule.FailedExecution.Alert
Instance: rms.opsmgr.net
Instance ID: {1219134EC-909D-VA37-1E06-679DD505D87C}
Management Group: OPS

Now – to tune, or investigate – we need to understand the problem workflow. In this case… the problem workflow is a rule, “Microsoft.SystemCenter.GenericNTPerfMapperModule.FailedExecution.Alert”

To troubleshoot – we need to find this rule in the console. However – the console lets us search on the Display Name, not this more cryptic ID. So here is a query that will help us map that:

Rules:

SQL:

select DisplayName from ruleview
where name = 'Microsoft.SystemCenter.GenericNTPerfMapperModule.FailedExecution.Alert'

Command Shell:

(Get-Rule | Where {$_.name -match 'Microsoft.SystemCenter.GenericNTPerfMapperModule.FailedExecution.Alert'}).DisplayName

Discoveries:

SQL:

select DisplayName from DiscoveryView
where name = 'Microsoft.Office.Sharepoint.Server.2007.MOSS.Server.Discovery'

Command Shell:

(Get-Discovery| Where {$_.name -match 'Microsoft.Office.Sharepoint.Server.2007.MOSS.Server.Discovery'}).DisplayName

Monitors:

SQL:

select DisplayName from monitorview
where Name = 'MAPI_logon_failure.Monitor'

Command Shell:

(Get-Monitor| Where {$_.name -match 'MAPI_logon_failure.Monitor'}).DisplayName

This will output the common display name of the rule/monitor/discovery:

Generic Performance Mapper Module Execution Failure

With this – finding the rule is pretty easy. One of the fastest ways – is to use search:

Tools > Search, then paste in the workflow common display name we got from the query:

Then – click “View Knowledge” This will bring up the rule properties. From there – you can view the data source, and get a better idea of what the rule/monitor/discovery does, and how to troubleshoot it.

R2 – Improved override screens

2009-04-10T20:55:57Z

Here are some changes in R2 that make Overrides a bit more intuitive, and help a LOT in understanding what has been done:

First up: We used to have a lot of names for “Class”. We used “Target”, “Instance Type”, “Object Type” and “Type” before… among others. In the Override screens previously – we called it “All Objects of Type”. Now – it is named “For all objects of class” (see image below) This make a lot more sense.

Unfortunately – in other screens – we still refer to the “Class” as a “Target”… so it still isnt 100% consistent. :-(

Next up – we have a Default Value added! So now – in any override – we can see the MP default, The override value, and (once applied) the effective value.

And now – in the same console – we immediately see all our overrides,

Whats REALLY cool about this? You can scope this new view – based on just about any criteria you would want:

This will really help, and reduce some of the dependency on external tools like Override Explorer.