JIMMY HARPER'S OPERATIONS MANAGER BLOG : Monitor

Using Integers and other “non-string” data types in Rules and Monitors

jimmyharper — Thu, 01 Oct 2009 07:19:19 GMT

If you need to use any non-string data type in the criteria for custom Rules and Monitors, you’ll need to edit the XML in order for it to work properly. By default, OpsMgr will treat everything as a String value and the Rule/Monitor will not work properly.

For example, I created a rule to watch for Event ID 1000 in the Application Log and throw an Alert if Parameter 1 is greater than 20. Here is the Rule criteria:

Using Event Log Explorer (awesome tool for testing, get it here), I generate Event 1000 with Parameter 1 set to 9:

I then received the following alert:

The reason I received this alert is that if OpsMgr is evaluating Parameter 1 as a String Value, then 9 would be greater than 20 (since 9 is greater than 2).

To correct, this I’ll need to edit the XML of the rule to change the data type to Integer.

So, I export the Management Pack that contains this Rule and look at the XML.

Here is the full XML of the Rule. The expression that we are concerned with is highlighted in green, and the part we need to change is in red.

<Rule ID="MomUIGeneratedRuleb80bc5a17ec4486185215843882c0046" Enabled="true" Target="MicrosoftWindowsLibrary6062780!Microsoft.Windows.Computer" ConfirmDelivery="true" Remotable="true" Priority="Normal" DiscardLevel="100">
<Category>Custom</Category>
<DataSources>
    <DataSource ID="DS" TypeID="MicrosoftWindowsLibrary6062780!Microsoft.Windows.EventProvider">
      <ComputerName>$Target/Property[Type="MicrosoftWindowsLibrary6062780!Microsoft.Windows.Computer"]/NetworkName$</ComputerName>
      <LogName>Application</LogName>
      <Expression>
        <And>
          <Expression>
            <SimpleExpression>
              <ValueExpression>
                <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>
              </ValueExpression>
              <Operator>Equal</Operator>
              <ValueExpression>
                <Value Type="UnsignedInteger">1000</Value>
              </ValueExpression>
            </SimpleExpression>
          </Expression>
          <Expression>
            <SimpleExpression>
              <ValueExpression>
                <XPathQuery Type="String">Params/Param[1]</XPathQuery>
              </ValueExpression>
              <Operator>Greater</Operator>
              <ValueExpression>
                <Value Type="String">20</Value>
              </ValueExpression>
            </SimpleExpression>
          </Expression>
        </And>
      </Expression>
    </DataSource>
</DataSources>
<WriteActions>
    <WriteAction ID="Alert" TypeID="Health!System.Health.GenerateAlert">
      <Priority>1</Priority>
      <Severity>2</Severity>
      <AlertOwner />
      <AlertMessageId>$MPElement[Name="MomUIGeneratedRuleb80bc5a17ec4486185215843882c0046.AlertMessage"]$</AlertMessageId>
      <AlertParameters>
        <AlertParameter1>$Data/Params/Param[1]$</AlertParameter1>
      </AlertParameters>
      <Suppression />
      <Custom1 />
      <Custom2 />
      <Custom3 />
      <Custom4 />
      <Custom5 />
      <Custom6 />
      <Custom7 />
      <Custom8 />
      <Custom9 />
      <Custom10 />
    </WriteAction>
</WriteActions>
</Rule>

To get this rule to work as expected, we’ll need to change “String” to “Integer”

          <Expression>
            <SimpleExpression>
              <ValueExpression>
                <XPathQuery Type="Integer">Params/Param[1]</XPathQuery>
              </ValueExpression>
              <Operator>Greater</Operator>
              <ValueExpression>
                <Value Type="Integer">20</Value>
              </ValueExpression>
            </SimpleExpression>
          </Expression>

So, I make the change to the XML, update the version number of the MP, and reimport it.

I create the same event on the agent an no longer get alerted on it.

The possible data types that can be used here are:

"Boolean"
"Integer"
"UnsignedInteger"
"Double"
"Duration"
"DateTime"
"String"

Service Monitors – What does the “State” value mean?

jimmyharper — Wed, 12 Aug 2009 02:00:19 GMT

When you create a Service Monitor in OpsMgr 2007, we get an alert / state change when the service is not running, but this does not necessarily mean that the service is “stopped”. The monitor attempts to get the “State” of the service, and alerts when the State is not “Running”. So, what other states are there? Here is a list of possible service states, copied from http://msdn.microsoft.com/en-us/library/ms685996(VS.85).aspx:

However, I recently ran into an issue where we got an alert from a Service Monitor and Health Explorer showed that State=9:

After checking with the OpsMgr product group, I found that State=9 means “Server not found”, and we get this when we fail to open SCManaged with “RPC_S_SERVER_UNAVAILABLE”. In this particular case, the problem was on a clustered server which had failed over to the second node, which did not have the OpsMgr Agent installed.

We also have two other states that are not listed in the above table. State=8 means “Service not found” (we’re trying to monitor a service that does not exist on the agent), and State=0 means “Unknown state”….not sure exactly when we would see this.

So, here’s the final list of State values that you may see on a service monitor:

0 = MOM_SERVICE_UNKNOWN_STATE
1 = MOM_SERVICE_STOPPED
2 = MOM_SERVICE_START_PENDING
3 = MOM_SERVICE_STOP_PENDING
4 = MOM_SERVICE_RUNNING
5 = MOM_SERVICE_CONTINUE_PENDING
6 = MOM_SERVICE_PAUSE_PENDING
7 = MOM_SERVICE_PAUSED
8 = MOM_SERVICE_NOT_FOUND
9 = MOM_SERVER_NOT_FOUND

SQL Server Full Text Search Service Monitor

jimmyharper — Thu, 02 Jul 2009 10:41:00 GMT

This issue is described in the SQL Server Management Pack Guide, but I wanted to blog it since I’ve seen a couple customers hit it.

In the current version of the SQL Server Management Pack (version 6.0.6559.0), we have a monitor for the SQL Server Full Text Search Service which is targeted at the SQL 2005/2008 DB Engine classes.

The problem is, this is an optional component in SQL Server and is not always installed. So, for servers where this service is not installed, we will see a lot of the following alerts:

Alert Name:

Service Check Probe Module Failed Execution

Service Check Data Source Module Failed Execution

Alert Description:

Error getting state of service Error: 0x8007007b Details: The filename, directory name, or volume label syntax is incorrect. One or more workflows were affected by this. Workflow name: Microsoft.SQLServer.2005.DBEngine.FullTextSearchServiceMonitor Instance name: MSSQLSERVER Instance ID: {625091EA-A1D9-1857-802C-0D908C93A5BB} Management group: jimmyh_mg1

To fix this, all we need to do is disable this monitor on any SQL Server that does not have the Full Text Search Service installed. The easiest way to do this is to create a group for all of the SQL Instances that do not have the service installed. The Full Text Search Service name is one of the discovered properties for the DB Engine class and will be blank if the service is not installed:

To create a group of SQL instances that do not have it installed, we can just use the criteria “Does not match regular expression . (dot)”, like this:

Then, just set an “Enabled=False” override on the monitor, targeted at this group:

Repeat the same steps to create the group and override for SQL 2008 DB Engines.

One more thing that you’ll want to do with this monitor is set the “Alert only if startup type is automatic” override to False for clustered SQL Instances…..since the service will always be in a Manual startup mode.

To do this, I create a group of Cluster SQL Instances where Full Text Search Service IS Installed:

And target the override at this group:

Again, repeat for SQL 2008 DB Engines.

Attached is a sample MP that contains the above groups and overrides for SQL 2005 DB Engines.

UPDATED:

I've removed the original attachment and attached a .zip file that contains these MPs for both SQL 2005 and SQL 2008.

AD Trust Monitor doesn’t reset to Healthy State

jimmyharper — Wed, 20 May 2009 09:33:25 GMT

The Active Directory Management Pack (ADMP) – version 6.0.6452.0 – contains a monitor named “AD Trust Monitoring”. This monitor runs a VBScript which queries WMI to get the status of the Domain Trusts on the Domain Controller. Is the trust has an error status, the Monitor should change to a critical state, when the error status goes away, it should change back to a Healthy State:

The problem is that, with default settings, once the Monitor goes into a Critical state, it will not change back to a Healthy state once the Trust problem is resolved. This is due to a bug in the script, where the value that is used to set the “Good” state is dependent on an override being set. Without getting into the details, just know that the only way to get the Monitor to work properly (so that state is changed from Critical to Healthy when a Trust problem is resolved), is to set the “LogSuccessEvent” override to “True”:

Setting this Override will also cause the script to log an event to the OpsMgr Event Log every time it completes successfully.

Configuring or Disabling Replication Monitoring in the Active directory Management pack

jimmyharper — Wed, 20 May 2009 08:30:10 GMT

The latest version of the Active Directory Management Pack (ADMP) – version 6.0.6452.0 – contains some significant changes to Replication Monitoring. The basic premise is the same, but the Rules and Monitors used have changed a bit.

Here’s a quick overview of how Replication Monitoring works:

Each Domain Controller runs the AD Replication Monitoring VBScript. The first time the script runs, it creates an object for the DC in the OpsMgrLatencyMonitors container in each Active Directory Naming Context that is monitored (the options are Domain, Configuration, and Application; these can be configured via overrides). By default, every 6th time the script runs (determined by the “Change Injection Frequency” override), the script will update the AdminDescription attribute on the DC’s objects in Active Directory with the current time (these objects can be seen in ADSIEdit.msc). The script will also look at the objects for all other DCs in its local copy of the Directory. To determine how long replication from each DCs is taking, the script will look at the whenCreated attribute (this tells the DC when that copy of the object arrived at this DC) and the AdminDescription attribute (this tells the DC when the object was updated). The time difference between when the object was updated and when it arrived at this DC tells us how long it takes to replicate an object from the given DC.

The script does a number of other things as well….more details on how all of the scripts in the ADMP work can be found in the old ADMP Technical Reference, found here. This technical reference was written for the original ADMP for MOM 2005, but much of the information about how the ADMP scripts work still applies today.

Back to the subject of this blog. The previous version of the ADMP used a Monitor named “AD Replication Monitoring” to run the Replication Monitoring script. It also had 4 rules that ran the script as well. In the new version of the ADMP, the monitor has been “deprecated” and is disabled by default. Several Rules have been created to run the script and alert on various issues. The purpose of this change was to avoid alert storms when one Domain Controller stops replicating (previously, we would get an alert from each DC, now we get just one). The downside of this change is that we now have fourteen (14) Rules that run the Replication Monitoring script. That’s 14 rules for each OS version….so, 14 for Windows 2000 DCs, 14 for Server 2003 DCs, and 14 for Server 2008 DCs. To confuse things a little more, some of the rules have the EXACT same display names.

So, if you need to set overrides to configure or disable Replication Monitoring, they must be set on all of the following Rules:

AD Replication is occurring slowly (there are three rules with this name)
One or more domain controllers may not be replicating (there are three rules with this name)
DC has failed to synchronize naming context with its replication partner (there are three rules with this name)
All of the replication partners failed to replicate.
AD Replication Performance Collection - Metric Replication Latency
AD Replication Performance Collection - Metric Replication Latency:Minimum
AD Replication Performance Collection - Metric Replication Latency:Maximum
AD Replication Performance Collection - Metric Replication Latency:Average

Why are some of these rules triplicated? Behind the scenes, these are written to distinguish between replication problems from different versions of Windows Domain Controllers. For example, if you look in the XML for the ADMP, you can see that the three “AD Replication is occurring slowly” rules have the following IDs:

Active_Directory_Latency_Alert_Rule_For_Windows2000

Active_Directory_Latency_Alert_Rule_For_Windows2003

Active_Directory_Latency_Alert_Rule_For_Windows2008

So, for example, each of these rules applies to a Windows Server 2003 Domain Controller, and watches for replication problems from the specified Domain Controller version.

Again, all of the above rules run the same Replication Monitoring script, so if you need to configure overrides for the script, you must set them on all of these rules.

Monitoring a service for State and StartMode

jimmyharper — Sun, 10 Aug 2008 04:45:00 GMT

I recently had a customer that wants to get an alert when a specific service is not Disabled and/or not Stopped. I used the following steps to accomplish this using a "Timed Script Three State Monitor". Even if you do not have this specific need, these steps can be used as a template for creating a monitor that uses a script to query WMI and change state or generate alerts based on the results. If you don't have a need for three states (Critical, Warning, Healthy), there is a Two State Monitor that can be used for this.

Create a new Monitor, select Scripting\Generic\Timed Script Three State Monitor

Technorati Tags: Operations Manager,monitor,Three-state,Two-state,service

Give it a name, target, etc. (I targeted the Windows Computer class, but Windows Operating System may be a better choice). I try to make a habit of unchecking "Monitor is enabled" and enabling it with an override later....at least while testing it:

Set the schedule...this just depends on how quickly you want to know if the service gets changed:

Next, I used a basic VB script which accepts a service name as a parameter, queries WMI for the service, and puts the Service Name, State (Running, Stopped, etc.), and StartMode (Disabled, Manual, Automatic) into property bag values. The full text of the script is below the screenshot:

---------------------------------------------------------------------------------------------------

Dim oAPI, oBag,strComputer
Set oAPI = CreateObject("MOM.ScriptAPI")
Set oBag = oAPI.CreatePropertyBag()
set oArgs=wscript.arguments
strComputer="."
ServName=oArgs(0)

Set namespace=GetObject("winmgmts:\\"& strComputer & "\root\cimv2")
set servinfo=namespace.ExecQuery("select * from win32_service where name =" & """" & servname & """")

for each objservice in servinfo

Call oBag.AddValue("ServiceName",ServName)
Call oBag.AddValue("State",objservice.State)
Call oBag.AddValue("StartMode",objservice.StartMode)
Call oAPI.Return(oBag)

---------------------------------------------------------------------------------------------------

For the script parameter, I just enter "ServiceName"....this will be replaced by an override later, or you can just enter your service name here:

Next, I set the "Unhealthy", "Degraded", and "Healthy" expressions for the monitor. My goal is to set the state to Warning when the service is Stopped but NOT Disabled , Critical when it is NOT Stopped, and Healthy when it is Stopped AND Disabled. I used the following expressions:

Unhealthy Expression:

Parameter Name: Property[@Name='State']

Operator: Does not equal

Value: Stopped

Degraded Expression:

Parameter Name: Property[@Name='StartMode']

Operator: Does not equal

Value: Disabled

AND

Parameter Name: Property[@Name='State']

Operator: Equals

Value: Stopped

Healthy Expression:

Parameter Name: Property[@Name='StartMode']

Operator: Equals

Value: Disabled

AND

Parameter Name: Property[@Name='State']

Operator: Equals

Value: Stopped

Next, I used the default settings for Health State, since they already match what I want to do:

Next, I configure the alert settings. The settings in the screen shot below will generate a Warning alert when the monitor is in a Warning state (service is not Disabled), and a Critical alert when the monitor is in the Critical state (service is not Stopped). The Alert Description will have the service name (using the ServiceName property created by the script):

Now that I have the monitor created, I need to enable it and set the Override for the Service Name:

I'm using the Alerter service for my test:

To test the monitor, I first set the Alerter service to Manual Startup and leave it stopped:

Then I verify that I get the Warning alert:

Health Explorer correctly shows the "Degraded" Warning state:

Now I want to test the Critical state, so I start the Alerter Service:

Now the alert is changed to Critical:

And Health Explorer shows the "Unhealthy" Critical state:

When I stop the service and disable it, the alert is auto-resolved and the state is changed back to Healthy:

I've attached my sample MP which includes the following monitors:

Service disabled and stopped - two-state monitor:

If the specified service is not Stopped AND Disabled, the computer will be put in a Warning state and a Warning alert will be generated. When the service is stopped and disabled, the computer will be put in a Healthy state.

Service disabled and stopped - three-state monitor:

If the specified service is Stopped and is not Disabled, the computer will be put in a Warning state and a Warning alert will be generated. If the specified service is not Stopped, the computer will be put in a critical state and a Critical alert will be generated. When the service is stopped and disabled, the computer will be put in a Healthy state.

Usage:

Both monitors are targeted at the Windows Computer class and roll up to the Configuration Health. Both monitors are disabled by default. They are configured to check the service every 1 minute. To enable one of the monitors, add an Override for the Computer or Group you wish to monitor and set the following Override parameters:

Enabled=True

Script Arguments = <Service Name>

Enjoy!!