JIMMY HARPER'S OPERATIONS MANAGER BLOG : Management Pack

Noisy DBCC Rule in the SQL Management Pack

jimmyharper — Thu, 01 Oct 2009 08:39:20 GMT

The SQL Server Management Pack has a rule, targeted at SQL 2000/2005/2008 DB Engine named “DBCC executed found and repaired errors”. This rule monitors for Event ID 8957 in the Application Log, logged by the SQL Service. Here is the criteria for the rule:

The problem with this is that you will see this Event logged, even when DBCC did not find or fix any errors…here’s an example:

So, this could result in a lot of alerts. Also, this rule generates an Informational alert by default, so if the event did indicate that errors were found, it would not generate a Warning or Critical alert.

We can change this by disabling the default rule and creating a new one. The new one should generate a Warning or Critical alert only if the DBCC event indicates that errors were found. We have two ways that we can do this:

The Easy Way

SQL 2005/2008:

Set an Override to disable the built-in “DBCC executed found and repaired errors” rules targeted at SQL 2005/2008 DB Engine
Create two separate rules, one targeted at SQL 2005 DB Engine and one targeted at SQL 2008 DB Engine. The rule will be exactly the same as the original, but with additional criteria for “Event Description does not contain “found 0 errors””:

Configure alerting as desired:

SQL 2000:

Set an Override to disable the built-in “DBCC executed found and repaired errors” rules targeted at SQL 2000 DB Engine
Create a new rule, targeted at SQL 2000 DB Engine. The criteria will be a little different than SQL 2005/2008:

The Harder, But Better, Way

I don’t like using Event Description in criteria if I don’t have to…using Parameters is less of a performance hit on the agent. So, instead of using the above options with EventDescription, we can use parameters. The parameters we are concerned with here are Parameter 8 (number of errors found) and Parameter 9 (number of errors repaired). This is for SQL 2005/2008 only, I haven’t looked at the parameters for SQL 2000. So, we can use the following criteria:

The reason that this is better is that the Agent doesn’t need to look through the full Event description of every 8957 event, it only needs to look at parameters 8 and 9.

The reason that is is harder is because we’ll also have to edit the XML of the rule, since we are using Integers in the criteria. See my previous blog post (from, like, an hour ago) on this here.

So, we’ll need to export our custom MP, open it in a text editor and find the XML for the OR expression for Parameter 8 and Parameter 9. Here is the XML:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

We’ll then need to change the “String” entries to “Integer”.

Before:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

After:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="Integer">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="Integer">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="Integer">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="Integer">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

Update the version of the MP, re-import it, and you should be good to go.

Exchange 2007 Synthetic Transactions against clustered Mailbox Servers may stop working

jimmyharper — Sat, 26 Sep 2009 08:30:17 GMT

Are you using the new Exchange 2007 Management Pack?

Are you configuring the CAS and/or Mailflow synthetic transcactions?

Are any of your target Mailbox servers clustered?

If so, then you’ll want to verify that these transactions are actually running…and be sure to check it at least 12 hours after setting it up.

Here’s a very quick synopsis of the issue:

Symptom:

You create either CAS or Mailflow synthetic transactions where the target mailbox server is clustered
The synthetic transactions are discovered and appear to be running fine…objects are healthy and we see perf data
Within 12 hours, you notice that we are no longer seeing performance data from the transaction.

Cause:

There is a discovery called “RMS Target Relationship Discovery” which runs every 12 hours on the RMS. The purpose of this discovery is to create a relationship between the Exchange Synthetic Transaction object and the target Mailbox server. The purpose of the relationship is so that when the Mailbox server is put into Maintenance Mode, the Transaction object will also go into MM (otherwise, the transactions would continue to run and would fail and go into an unhealthy state). The problem is that when the target mailbox server is clustered, some core OpsMgr rules relating to cluster monitoring end up getting applied to the CAS server and essentially disabling the workflows that run these transactions. I know that’s not an extremely technical explanation, but it’ll do for now. Regardless, if the target mailbox server is clustered, the transactions will completely stop running within 12 hours.

Resolution:

To resolve this, we need to disable the “RMS Target Relationship Discovery” discovery and use Powershell to remove the discovered relationship from OpsMgr.

Here’s a visual look at the problem:

Use the wizard to create the transactions:

EXCAS.OpsMgr.net is my CAS server:

I am going to enable the Active Sync and OWA tests:

EX07a is a standalone Exchange server, EX1V1 is a clustered mailbox server:

The discovery defaults to running every 24 hours, I changed it to 5 minutes for testing:

After giving the discoveries time to run (or just restarting the Health Service on the CAS server to force it), I go to the CAS Synthetic Transaction State view in the console and see that the objects are discovered. I have an object for each Transaction to each Mailbox server (pay no attention to the unhealthy states in the screen shot, they are unrelated to this issue).

Looking in the performance view, I can see that I am receiving performance data from the transactions to both Mailbox servers (I configured the transactions to run every 1 minute, default is every 15 minutes):

After the RMS Target Relationship discovery runs, we are no longer seeing data from the transaction to the clustered mailbox server (the red line). Normally you would see this within 12 hours of the transactions being created…I sped it up to run sooner to reproduce it easily:

Also, the state of the transaction object will remain the same as it was before the RMS Target Relationship discovery ran (until the problem is fixed)….if it was Healthy then, it will stay Healthy….if it was Unhealthy, it will stay Unhealthy.

To resolve this we must do the following two things (both MUST be done):

1. Create an Override to disable the “RMS Target Relationship Discovery” discovery

Go to Authoring\Object Discoveries

Select “View all targets"

Scope to “Exchange 2007 CAS Connectivity”

Select the “RMS Target Relationship Discovery” object – Overrides – Override the Object Discovery – For all objects of class: Root Management Server:

Set an Override for Enabled = False:

Now that the override is disabled, we will need to remove the relationships that were discovered by it. To do this, we’ll use the Powershell command Remove-DisabledMonitoringObject. Note that you MUST set the Enabled=False override mentioned above in order for this to work (if the discovery is not explicitly disabled, the Powershell command will not remove the objects discovered by it). For more info on the Remove-DisabledMonitoringObject command, see Jonathan’s blog on it here.

Just open the OpsMgr Command Shell and run Remove-DisabledMonitoringObject:

Once the relationship is removed, the Transaction starts running again:

Bottom Line:

If you use the Exchange 2007 Synthetic Transactions for CAS or Mailflow tests against a Clustered Mailbox server, the transactions will stop running within 12 hours.
To fix this, you must disable the “RMS Target Relationship Discovery” discovery and run the Remove-DisabledMonitoringObject Powershell cmdlet.

SQL Server Full Text Search Service Monitor

jimmyharper — Thu, 02 Jul 2009 10:41:00 GMT

This issue is described in the SQL Server Management Pack Guide, but I wanted to blog it since I’ve seen a couple customers hit it.

In the current version of the SQL Server Management Pack (version 6.0.6559.0), we have a monitor for the SQL Server Full Text Search Service which is targeted at the SQL 2005/2008 DB Engine classes.

The problem is, this is an optional component in SQL Server and is not always installed. So, for servers where this service is not installed, we will see a lot of the following alerts:

Alert Name:

Service Check Probe Module Failed Execution

Service Check Data Source Module Failed Execution

Alert Description:

Error getting state of service Error: 0x8007007b Details: The filename, directory name, or volume label syntax is incorrect. One or more workflows were affected by this. Workflow name: Microsoft.SQLServer.2005.DBEngine.FullTextSearchServiceMonitor Instance name: MSSQLSERVER Instance ID: {625091EA-A1D9-1857-802C-0D908C93A5BB} Management group: jimmyh_mg1

To fix this, all we need to do is disable this monitor on any SQL Server that does not have the Full Text Search Service installed. The easiest way to do this is to create a group for all of the SQL Instances that do not have the service installed. The Full Text Search Service name is one of the discovered properties for the DB Engine class and will be blank if the service is not installed:

To create a group of SQL instances that do not have it installed, we can just use the criteria “Does not match regular expression . (dot)”, like this:

Then, just set an “Enabled=False” override on the monitor, targeted at this group:

Repeat the same steps to create the group and override for SQL 2008 DB Engines.

One more thing that you’ll want to do with this monitor is set the “Alert only if startup type is automatic” override to False for clustered SQL Instances…..since the service will always be in a Manual startup mode.

To do this, I create a group of Cluster SQL Instances where Full Text Search Service IS Installed:

And target the override at this group:

Again, repeat for SQL 2008 DB Engines.

Attached is a sample MP that contains the above groups and overrides for SQL 2005 DB Engines.

UPDATED:

I've removed the original attachment and attached a .zip file that contains these MPs for both SQL 2005 and SQL 2008.

SQL Agent Job Discovery not working?

jimmyharper — Fri, 05 Jun 2009 05:36:00 GMT

The SQL Server Management Pack includes an option to discover and monitor SQL Server Agent Jobs for SQL 2000/2005/2008. The Discovery for this is disabled by default:

I ran into an issue recently where all Agent Jobs for specific SQL Servers were not being discovered. Examining the event logs on the SQL Server, we see the following in the OpsMgr Event Log:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/4/2009 8:36:19 PM
Event ID:      21406
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
The process started at 8:36:18 PM failed to create System.Discovery.Data. Errors found in output:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 1\4595\SQLAgentJobDiscovery.vbs(106, 5) Microsoft VBScript runtime error: Type mismatch

Command executed: "C:\Windows\system32\cscript.exe" /nologo "SQLAgentJobDiscovery.vbs" {974F57A5-5705-B6B2-B8DC-1CA0B433DCD4} {46913442-CAC1-7E38-89B4-1A6B462ED0D0} OMDW.opsmgr.net OMDW.opsmgr.net OMDW\I01 I01 SQLAgent$I01"
Working Directory: C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 1\4595\

One or more workflows were affected by this.

Workflow name: Microsoft.SQLServer.2008.AgentJobDiscoveryRule
Instance name: SQLAgent$I01
Instance ID: {46913442-CAC1-7E38-89B4-1A6B462ED0D0}
Management group: PROD1

The “Type mismatch” error typically means that some variable in the script is returning with an incorrect data type. After examining the Discovery script and doing to troubleshooting, we determined that the problem was happening because the “Description” field for the Agent Job was NULL. This can be confirmed by running the “sp_help_job” Stored Procedure against the MSDB database on the SQL Instance (which is exactly what the Discovery script does:

This will also happen if any of the following properties of the job are NULL:

job_id
originating_server
name
description
category
owner

We probably won't ever see this with the job_id, originating_server, category or name properties, but we've seen it with the description and owner properties.

To correct this, we can simply enter some text in the description field of the Agent Job:

Note that the problem described above (Agent Job discovery failing when properties are NULL) happens on SQL 2005 and 2008 Agent Jobs. SQL 2000 Agent Job Discovery does not use the VBScript, and does seem to work in this scenario, but the NULL values are populated with the values of other Agent Jobs, so it is not accurate.

So, what if you have a very large number of SQL Servers and Agent Jobs and do not want to worry about making sure that all of them have text in the Description field? To take care of this, I created a “workaround” version of the SQL Agent Job Discoveries that will discover these jobs and enters NULL for the NULL propeerty.

Import the “Microsoft.SQLServer.200x.Discovery.CustomAgentJobDiscovery.xml” management pack
Disable the original Agent Job Discovery and enable the new one (“SQL Server 200x Custom Agent Job Discovery”):

Verify that the Agent Jobs are discovered:

Configuring or Disabling Replication Monitoring in the Active directory Management pack

jimmyharper — Wed, 20 May 2009 08:30:10 GMT

The latest version of the Active Directory Management Pack (ADMP) – version 6.0.6452.0 – contains some significant changes to Replication Monitoring. The basic premise is the same, but the Rules and Monitors used have changed a bit.

Here’s a quick overview of how Replication Monitoring works:

Each Domain Controller runs the AD Replication Monitoring VBScript. The first time the script runs, it creates an object for the DC in the OpsMgrLatencyMonitors container in each Active Directory Naming Context that is monitored (the options are Domain, Configuration, and Application; these can be configured via overrides). By default, every 6th time the script runs (determined by the “Change Injection Frequency” override), the script will update the AdminDescription attribute on the DC’s objects in Active Directory with the current time (these objects can be seen in ADSIEdit.msc). The script will also look at the objects for all other DCs in its local copy of the Directory. To determine how long replication from each DCs is taking, the script will look at the whenCreated attribute (this tells the DC when that copy of the object arrived at this DC) and the AdminDescription attribute (this tells the DC when the object was updated). The time difference between when the object was updated and when it arrived at this DC tells us how long it takes to replicate an object from the given DC.

The script does a number of other things as well….more details on how all of the scripts in the ADMP work can be found in the old ADMP Technical Reference, found here. This technical reference was written for the original ADMP for MOM 2005, but much of the information about how the ADMP scripts work still applies today.

Back to the subject of this blog. The previous version of the ADMP used a Monitor named “AD Replication Monitoring” to run the Replication Monitoring script. It also had 4 rules that ran the script as well. In the new version of the ADMP, the monitor has been “deprecated” and is disabled by default. Several Rules have been created to run the script and alert on various issues. The purpose of this change was to avoid alert storms when one Domain Controller stops replicating (previously, we would get an alert from each DC, now we get just one). The downside of this change is that we now have fourteen (14) Rules that run the Replication Monitoring script. That’s 14 rules for each OS version….so, 14 for Windows 2000 DCs, 14 for Server 2003 DCs, and 14 for Server 2008 DCs. To confuse things a little more, some of the rules have the EXACT same display names.

So, if you need to set overrides to configure or disable Replication Monitoring, they must be set on all of the following Rules:

AD Replication is occurring slowly (there are three rules with this name)
One or more domain controllers may not be replicating (there are three rules with this name)
DC has failed to synchronize naming context with its replication partner (there are three rules with this name)
All of the replication partners failed to replicate.
AD Replication Performance Collection - Metric Replication Latency
AD Replication Performance Collection - Metric Replication Latency:Minimum
AD Replication Performance Collection - Metric Replication Latency:Maximum
AD Replication Performance Collection - Metric Replication Latency:Average

Why are some of these rules triplicated? Behind the scenes, these are written to distinguish between replication problems from different versions of Windows Domain Controllers. For example, if you look in the XML for the ADMP, you can see that the three “AD Replication is occurring slowly” rules have the following IDs:

Active_Directory_Latency_Alert_Rule_For_Windows2000

Active_Directory_Latency_Alert_Rule_For_Windows2003

Active_Directory_Latency_Alert_Rule_For_Windows2008

So, for example, each of these rules applies to a Windows Server 2003 Domain Controller, and watches for replication problems from the specified Domain Controller version.

Again, all of the above rules run the same Replication Monitoring script, so if you need to configure overrides for the script, you must set them on all of these rules.