Automating the Infrastructure : MOM

Manually Resetting a Monitor

Brian Wren — Tue, 07 Aug 2007 19:42:16 GMT

You may have noticed that all monitors in OpsMgr 2007 have an option called Reset Health. The problem you may have encountered with this option is that it doesn't do anything on most monitors. It will typically only work on monitors that were created with a Manual reset. This option actually does work in all cases, but it's up to the monitor (or more specifically the monitor type) to do something with it. Most of the monitor types that you're probably building monitors from don't implement the required option for Reset Health to have any effect, so it looks like it just doesn't work.

To understand the issue here, we need to drop down into the management pack XML. Every monitor is based on a monitor type that may be defined in the same management pack but is more commonly one of the standard monitor types in Microsoft.Windows.Library. One of the sections of the of the monitor type is RegularDetections that defines what action to take when the different events watched by the monitor type are detected.

Below is an example from the monitor type Microsoft.Windows.2SingleEventLog2StateMonitorType. This is the monitor type typically used when you create a monitor using a Windows event for an error state and a second Windows event for a healthy state. Note the bold section that defines the RegularDetections. The order of actions to take is defined from the innermost XML node out. In the case of FirstEventRaised, FirstDataSource is executed to access the event log and then FirstFilterCondition is executed to filter out the event in question.

<UnitMonitorType ID="Microsoft.Windows.2SingleEventLog2StateMonitorType" Accessibility="Public">
<MonitorTypeStates>
    <MonitorTypeState ID="FirstEventRaised" NoDetection="false" />
    <MonitorTypeState ID="SecondEventRaised" NoDetection="false" />
</MonitorTypeStates>
<Configuration>
    <IncludeSchemaTypes>
      <SchemaType>System!System.ExpressionEvaluatorSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstComputerName" type="xsd:string" />
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstLogName" type="xsd:string" />
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstExpression" type="ExpressionType" />
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondComputerName" type="xsd:string" />
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondLogName" type="xsd:string" />
    <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondExpression" type="ExpressionType" />
</Configuration>
<MonitorImplementation>
    <MemberModules>
      <DataSource ID="FirstDataSource" TypeID="Microsoft.Windows.BaseEventProvider">
        <ComputerName>$Config/FirstComputerName$</ComputerName>
        <LogName>$Config/FirstLogName$</LogName>
      </DataSource>
      <DataSource ID="SecondDataSource" TypeID="Microsoft.Windows.BaseEventProvider">
        <ComputerName>$Config/SecondComputerName$</ComputerName>
        <LogName>$Config/SecondLogName$</LogName>
      </DataSource>
      <ConditionDetection ID="FirstFilterCondition" TypeID="System!System.ExpressionFilter">
        <Expression>$Config/FirstExpression$</Expression>
      </ConditionDetection>
      <ConditionDetection ID="SecondFilterCondition" TypeID="System!System.ExpressionFilter">
        <Expression>$Config/SecondExpression$</Expression>
      </ConditionDetection>
    </MemberModules>
    <RegularDetections>
      <RegularDetection MonitorTypeStateID="FirstEventRaised">
        <Node ID="FirstFilterCondition">
          <Node ID="FirstDataSource" />
        </Node>
      </RegularDetection>
      <RegularDetection MonitorTypeStateID="SecondEventRaised">
        <Node ID="SecondFilterCondition">
          <Node ID="SecondDataSource" />
        </Node>
      </RegularDetection>
    </RegularDetections>
</MonitorImplementation>
</UnitMonitorType>

In order to enable the Reset Health option for the monitor, you need to add an OnDemandDetections section to the monitor type to define the action to be taken. In the case of the Microsoft.Windows.2SingleEventLog2StateMonitorType, we need to decide which event will define success since this is the one we want to execute with the OnDemandDetection. I'm going to use the first event for an error, and the second event for success. With that in mind, we can modify the monitor type to the following with the added XML in bold:

<UnitMonitorType ID="Microsoft.Windows.2SingleEventLog2StateMonitorTypeWithReset" Accessibility="Public">
    <MonitorTypeStates>
        <MonitorTypeState ID="FirstEventRaised" NoDetection="false"/>
        <MonitorTypeState ID="SecondEventRaised" NoDetection="false"/>
    </MonitorTypeStates>
    <Configuration>
        <IncludeSchemaTypes>
            <SchemaType>System!System.ExpressionEvaluatorSchema</SchemaType>
        </IncludeSchemaTypes>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstComputerName" type="xsd:string"/>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstLogName" type="xsd:string"/>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="FirstExpression" type="ExpressionType"/>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondComputerName" type="xsd:string"/>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondLogName" type="xsd:string"/>
        <xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="SecondExpression" type="ExpressionType"/>
    </Configuration>
    <MonitorImplementation>
        <MemberModules>
            <DataSource ID="FirstDataSource" TypeID="Windows!Microsoft.Windows.BaseEventProvider">
                <ComputerName>$Config/FirstComputerName$</ComputerName>
                <LogName>$Config/FirstLogName$</LogName>
            </DataSource>
            <DataSource ID="SecondDataSource" TypeID="Windows!Microsoft.Windows.BaseEventProvider">
                <ComputerName>$Config/SecondComputerName$</ComputerName>
                <LogName>$Config/SecondLogName$</LogName>
            </DataSource>
            <ProbeAction ID="OnDemandReset" TypeID="System!System.PassThroughProbe"/>
            <ConditionDetection ID="FirstFilterCondition" TypeID="System!System.ExpressionFilter">
                <Expression>$Config/FirstExpression$</Expression>
            </ConditionDetection>
            <ConditionDetection ID="SecondFilterCondition" TypeID="System!System.ExpressionFilter">
                <Expression>$Config/SecondExpression$</Expression>
            </ConditionDetection>
        </MemberModules>
        <RegularDetections>
            <RegularDetection MonitorTypeStateID="FirstEventRaised">
                <Node ID="FirstFilterCondition">
                    <Node ID="FirstDataSource"/>
                </Node>
            </RegularDetection>
            <RegularDetection MonitorTypeStateID="SecondEventRaised">
                <Node ID="SecondFilterCondition">
                    <Node ID="SecondDataSource"/>
                </Node>
            </RegularDetection>
        </RegularDetections>
        <OnDemandDetections>
            <OnDemandDetection MonitorTypeStateID="SecondEventRaised">
                <Node ID="OnDemandReset"/>
            </OnDemandDetection>
        </OnDemandDetections>
    </MonitorImplementation>
</UnitMonitorType>

Note that there are two nodes added to the XML. First, we need to add a new System.PassThroughProbe module so the monitor knows to even look for our reset. We next need an OnDemandDetections node that specifies what MonitorTypeStateID to set when the reset is performed. In this case, I'm assuming that the second event indicates a healthy state. The MonitorTypeStateID of SecondEventRaised directly corresponds to the same ID in the MonitorTypeStates node.

Exporting Management Packs

Brian Wren — Mon, 06 Aug 2007 20:43:00 GMT

Obviously, a great way to learn how to author a management pack is to look at existing ones. The problem is that most of the interesting ones are sealed so you can't read the file, and the Operations Console won't allow you to export a sealed management pack.

This is actually very easy using Command Shell. You can export a single management pack or export the entire set of installed management packs from a management group. This is my personal favorite as I like to have the entire set of management packs available for my reference.

Command Shell is an optional component of the Operations Manager User Interfaces, so it's probably something you already have installed.

Exporting a single management pack

To export a single management pack, use the following command:

get-managementPack -name <Name of MP> | export-managementPack -path <Directory to store exported file>

For example, to export the SQL Library MP to a directory called c:\mp:

get-managementPack -name Microsoft.SQLServer.Library | export-managementPack -path c:\mp

Exporting all management packs

It's actually easier to export all management packs since you don't have to specify the -name parameter on the get-managementPack CmdLet. Just use the following, and you will fill c:\mp with the entire set of management packs installed in your management group.

get-managementPack | export-managementPack -path c:\mp

Targeting Rules and Monitors

Brian Wren — Mon, 14 May 2007 17:40:00 GMT

The concept of what target to use for a particular rule or monitor in OpsMgr 2007 seems to be one that people are struggling with. The confusion is most prevelant among people who have had experience with MOM 2005 since the methods have completely in changed. In MOM 2005, you would apply rule groups to computer groups. If a computer was in the group, it would get the rule - it was that simple. In OpsMgr 2007, we do not use groups to target rules/monitors. What adds to the confusion though is that groups are available in the list of targets when creating a rule/monitor in the console. Let's see if I can explain this.

The main conceptual difference between MOM 2005 and OpsMgr 2007 in this respect is that in MOM 2005 we thought about which computer we would retrieve a particular piece of information from, and we targeted rules at groups containing those computers. We had to think about the individual agents and view the environment in terms of the physical computers being managed. We had formulas for the groups to identify which computers held different components, but we were ultimately targeting groups of agents. In OpsMgr 2007, we think about what component is generating the information. That’s a distinctly different concept and ultimately more powerful. OpsMgr will figure out which agents hold instances of that component, deliver the rule/monitor to the appropriate agents, and execute the rule/monitor for each instance.

In 2007, we also work against multiple instances of a class on a particular agent. In MOM 2005, we only had a single rule sent to the agent, and that rule had to do the enumeration. A good example is SQL databases. The SQL scripts in MOM 2005 were enormous because they had to enumerate the whole list of databases and other SQL objects every single time they executed. In 2007, we just apply those rules to the SQL Database class, and OpsMgr executes the rule for each database instance discovered.

How does a rule get to an agent?

For any particular rule/monitor, OpsMgr will enumerate all instances of the target class and apply the rule to each. If there are no instances of the target class on a particular agent, then the rule will do nothing. It's that simple.

If I can't target groups, why are they listed when I select a target for a rule?

Groups are classes just like any other. They’re singleton classes where the class and the instance are one and the same, but they are classes nonetheless which is why they show up in the list with all other classes. There are really very few circumstances where you will target a rule at a group though.

What if I do target a group?

You can apply a rule/monitor directly to a group, but it will execute against the group object itself. OpsMgr will not enumerate members of the group and apply the rule to each. Any rules targeted at groups will actually operate on the Root Management Server since groups have no host and unhosted objects are managed by the RMS.

How do I target some group of objects then?

To the specific question of how to get a particular rule/monitor to a subset of components, you have two basic options. Let’s say for example, you have a particular subset of web sites that you need a particular rule to apply. You could target that rule at the IIS 2003 Web Site class for example, but that would apply the rule to all instances of that class. It would probably apply to sites that you didn’t want.

Option 1 would be to create a new class and target the rule at the class. In the case of an IIS site, this would mean that you would need to go to the Authoring Console or raw XML and create a new class and discovery. That’s a more advanced solution that most customers will do and probably overkill anyway.

Option 2 is the create a rule target at the whole class and disable it. Create a group with the sites you want and create an override for that group to enable your rule. This might sound like a workaround, but it's a completely valid solution.

How do I know if I'm selecting the right target?

The easiest method to validate you are using a target that actually has instances is to use the Discovered Inventory view in the Operations Console prior to creating your rule/monitor. In the Actions pane is an option called "Change target type..." that will bring up the same Select a Target Type dialog box that you see when you select the target for a rule/monitor. This view will list all instances of the target class you select. You can validate which agents have an instance of that class and how many instances each has. If there are no instances listed, then the rule isn't going to do anything. If there are instances, then you not only be confident that the rule/monitor will execute on the agent, but you can also view the properties of the instance that will be accessible to any rules/monitors targeted at it.

WMI Events in OpsMgr 2007

Brian Wren — Sat, 07 Apr 2007 23:12:00 GMT

I've already explained how to work with WMI events in MOM 2005 in an article in TechNet Magazine. Most of the information in that article still applies since it covers the basics of how WMI events work and how to write a query to detect them. What has changed significantly though is how to use them in Operations Manager. I suppose a follow up article would be in order, but a simple blog post should be enough to give you the information you need.

First of all, WMI events in OpsMgr 2007 are used for exactly the same purpose as in MOM 2005. The one big difference is that we no longer have WMI providers for OpsMgr objects to monitor for things like the modification of an alert. We're working on such a different platform though, that those old scenarios don't really apply anymore anyway.

I'm going to speak completely in terms of the Operations Console since that's where most people work with Operations Manager - at least at this early stage of its life. If you're more comfortable working directory with the XML of a management pack, then you'll know exactly what pieces of this information to pull out.

You can create a monitor or rule for a WMI event using the appropriate wizard in the Operations Console. If I have to guide you to those or define the different between an rule and a monitor, then you probably want to start on some more basic monitoring before tackling this topic. The namespace and query are going to be exactly the same as the ones you created in MOM 2005. Again, check my TechNet article if you need some help there.

WMI Event Filter Expressions

As discussed in my article, you can specify criteria in the WMI query itself, or you can write a general query and specify the criteria separately. In MOM 2005, that meant specifying criteria in the rule. In OpsMgr 2007, you need to provide a filter expression. The question is how to build it.

The wizard for creating a WMI Event monitor/rule actually won't let you specify no criteria. The Next button won't be active until you provide some entry. If you have a query that doesn't need any filter (which is entirely reasonable), then just specify some bogus filter. Once the monitor/rule is created, open up its properties and delete the filter. It's entirely valid to have a monitor/rule with no filter - the wizard just doesn't let you do it.

If you are going to provide filter, the parameter name needs to be in following format:

Collection[@Name='TargetInstance']/Property[@Name=<property name>]

Example - No Filter

We can show this with a simple example. Let's fire the rule when Notepad starts. One method would be to use the following criteria with no filter:

SELECT * FROM __InstanceCreationEvent WITHIN 30 WHERE TargetInstance ISA 'win32_process' AND TargetInstance.Name = 'notepad.exe'

Example - Filter

The second option would be to use the following query:

SELECT * FROM __InstanceCreationEvent WITHIN 30 WHERE TargetInstance ISA 'win32_process'

with the following filter:

Parameter Name: Collection[@Name='TargetInstance']/Property[@Name='Name'] equals
Operator: Equals
Value: notepad.exe

Blank Description in SQL Server 2005 Block Analysis script

Brian Wren — Sun, 09 Apr 2006 08:46:00 GMT

I found that block analysis script for SQL 2005 was generating an appropriate alert, but the description is blank. That description is pretty important because it identifies the process blocking and being blocked.

After some investigation, I found that there is a problem on the line of the script building the description - specifically where we include the time that the black has been occuring. This is pulled from the waittime field of the sysprocesses table. It turns out that this field was an Int in SQL 2000 but changed to a BigInt in SQL 2005. The script uses a CInt to covert this value to an Integer which results in an overflow. Since we run into an error building the description, it's left blank in the alert.

Fortunately, this is trivial fix - just a matter of swapping out that CInt. I found that to get a correct calculation, I had to convert to a Double prior to performing the division required to convert the value from milliseconds into minutes.

The offending line is 2240 by my count. Just modify as follows:

Original line

""" has been blocked for " & CStr(CInt(rsBlockedSPIDS("waittime").Value / 1000 / 60)) & _

Changed Line

""" has been blocked for " & CStr(CInt(CDbl(rsBlockedSPIDS("waittime").Value) / 1000 / 60)) & _

Problem with SQL Server 2005 Long Running Agent Jobs script in MOM 2005

Brian Wren — Sat, 25 Mar 2006 19:01:00 GMT

I found a problem in the latest version of the SQL Server Management Pack (version 09.0.1399.0700) that I imagine many others have wrestled with as well. Easy fix though, so I wanted to post it out here for everyone.

The problem is with the script SQL Server 2005 Long Running Agent Jobs. If this script finds a long running job, you will most likely get an alert with a message similar to the following:

The SQL Server management pack script "SQL Server 2005 Long Running Agent Jobs" is unable to successfully connect to the SQL Server instance "MSSQLSERVER". The error message returned is "Invalid column name 'originating_server'."

If you're interested in the background, this script is based on the same script for SQL 2000, but one critical modification was missed in the conversion. The script retrieves a list of running jobs from the sysjobs system table in msdb. In SQL 2000, the name of the server is in the originating_server field, but in SQL 2005 this was changed to originating_server_id. We need to join with sys.servers to retrieve the server name.

The script can be fixed with a change to a single line. According to my count, line 2265 should be the following:

strQuery = "select originating_server, name from sysjobs where job_id = " & strGUID

Replace this line with the following line that includes a new query. I shouldn't have to state the obvious, but make sure you backup the original script first.

strQuery = "SELECT sys.servers.name as originating_server,sysjobs.name FROM sys.servers JOIN sysjobs ON sysjobs.originating_server_id = sys.servers.server_id WHERE sysjobs.job_id = " & strGUID

That should be the only change you need to make to get this script working properly. I expect that we will see it fixed in the next revision of the management pack, but this quick fix should get you by until then.

WMI Notifications in MOM

Brian Wren — Wed, 21 Dec 2005 23:34:00 GMT

Okay, I'm getting tired of answering this question but I have to admit that there is relatively little technical information out there on it. The most common requests I get for MOM scripts are for things like monitoring if a process is running, if a file gets created, if a registry key gets changed, etc. Essentially, these are to monitor some action on the computer that we can't detect through standard MOM providers.

Most of the time, monitoring these actions does not require a script. They can be handled with a relatively simple WMI provider. This is far more efficient than script - easier to create, less overhead on the agent, and way more responsive. Rather than running a script every few minutes to figure out if a process is running for example, I can have a WMI notification tell me when the process terminates. Rather then regularly checking if some file exists, I can have a WMI notification tell me when it gets created.

I have a document that I wrote on this that includes an overview of WMI, methods for writing and testing WMI queries, and a bunch of examples. Have a look.