JIMMY HARPER'S OPERATIONS MANAGER BLOG : SQL

Noisy DBCC Rule in the SQL Management Pack

jimmyharper — Thu, 01 Oct 2009 08:39:20 GMT

The SQL Server Management Pack has a rule, targeted at SQL 2000/2005/2008 DB Engine named “DBCC executed found and repaired errors”. This rule monitors for Event ID 8957 in the Application Log, logged by the SQL Service. Here is the criteria for the rule:

The problem with this is that you will see this Event logged, even when DBCC did not find or fix any errors…here’s an example:

So, this could result in a lot of alerts. Also, this rule generates an Informational alert by default, so if the event did indicate that errors were found, it would not generate a Warning or Critical alert.

We can change this by disabling the default rule and creating a new one. The new one should generate a Warning or Critical alert only if the DBCC event indicates that errors were found. We have two ways that we can do this:

The Easy Way

SQL 2005/2008:

Set an Override to disable the built-in “DBCC executed found and repaired errors” rules targeted at SQL 2005/2008 DB Engine
Create two separate rules, one targeted at SQL 2005 DB Engine and one targeted at SQL 2008 DB Engine. The rule will be exactly the same as the original, but with additional criteria for “Event Description does not contain “found 0 errors””:

Configure alerting as desired:

SQL 2000:

Set an Override to disable the built-in “DBCC executed found and repaired errors” rules targeted at SQL 2000 DB Engine
Create a new rule, targeted at SQL 2000 DB Engine. The criteria will be a little different than SQL 2005/2008:

The Harder, But Better, Way

I don’t like using Event Description in criteria if I don’t have to…using Parameters is less of a performance hit on the agent. So, instead of using the above options with EventDescription, we can use parameters. The parameters we are concerned with here are Parameter 8 (number of errors found) and Parameter 9 (number of errors repaired). This is for SQL 2005/2008 only, I haven’t looked at the parameters for SQL 2000. So, we can use the following criteria:

The reason that this is better is that the Agent doesn’t need to look through the full Event description of every 8957 event, it only needs to look at parameters 8 and 9.

The reason that is is harder is because we’ll also have to edit the XML of the rule, since we are using Integers in the criteria. See my previous blog post (from, like, an hour ago) on this here.

So, we’ll need to export our custom MP, open it in a text editor and find the XML for the OR expression for Parameter 8 and Parameter 9. Here is the XML:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

We’ll then need to change the “String” entries to “Integer”.

Before:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="String">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="String">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

After:

<Expression>
   <Or>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="Integer">Params/Param[8]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="Integer">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
     <Expression>
       <SimpleExpression>
         <ValueExpression>
           <XPathQuery Type="Integer">Params/Param[9]</XPathQuery>
         </ValueExpression>
         <Operator>Greater</Operator>
         <ValueExpression>
           <Value Type="Integer">0</Value>
         </ValueExpression>
       </SimpleExpression>
     </Expression>
   </Or>
</Expression>

Update the version of the MP, re-import it, and you should be good to go.

SQL Server Full Text Search Service Monitor

jimmyharper — Thu, 02 Jul 2009 10:41:00 GMT

This issue is described in the SQL Server Management Pack Guide, but I wanted to blog it since I’ve seen a couple customers hit it.

In the current version of the SQL Server Management Pack (version 6.0.6559.0), we have a monitor for the SQL Server Full Text Search Service which is targeted at the SQL 2005/2008 DB Engine classes.

The problem is, this is an optional component in SQL Server and is not always installed. So, for servers where this service is not installed, we will see a lot of the following alerts:

Alert Name:

Service Check Probe Module Failed Execution

Service Check Data Source Module Failed Execution

Alert Description:

Error getting state of service Error: 0x8007007b Details: The filename, directory name, or volume label syntax is incorrect. One or more workflows were affected by this. Workflow name: Microsoft.SQLServer.2005.DBEngine.FullTextSearchServiceMonitor Instance name: MSSQLSERVER Instance ID: {625091EA-A1D9-1857-802C-0D908C93A5BB} Management group: jimmyh_mg1

To fix this, all we need to do is disable this monitor on any SQL Server that does not have the Full Text Search Service installed. The easiest way to do this is to create a group for all of the SQL Instances that do not have the service installed. The Full Text Search Service name is one of the discovered properties for the DB Engine class and will be blank if the service is not installed:

To create a group of SQL instances that do not have it installed, we can just use the criteria “Does not match regular expression . (dot)”, like this:

Then, just set an “Enabled=False” override on the monitor, targeted at this group:

Repeat the same steps to create the group and override for SQL 2008 DB Engines.

One more thing that you’ll want to do with this monitor is set the “Alert only if startup type is automatic” override to False for clustered SQL Instances…..since the service will always be in a Manual startup mode.

To do this, I create a group of Cluster SQL Instances where Full Text Search Service IS Installed:

And target the override at this group:

Again, repeat for SQL 2008 DB Engines.

Attached is a sample MP that contains the above groups and overrides for SQL 2005 DB Engines.

UPDATED:

I've removed the original attachment and attached a .zip file that contains these MPs for both SQL 2005 and SQL 2008.

SQL Database Properties Not Discovered

jimmyharper — Thu, 11 Jun 2009 20:26:00 GMT

I ran into an issue recently where some SQL Databases were not showing any properties in OpsMgr, other than the database name:

To get these properties, the database discovery script runs the “sp_helpdb” stored procedure against the database. To test this, open SQL Server Managment Studio, connect to the SQL Instance in question, open a new query window and run “sp_helpdb <database name>”":

NOTE: You should run this under the same account that is used for the “SQL Server Discovery Account” RunAs Profile….if you haven’t defined an account for this profile, then use the Action Account.

If this doesn’t return any results (as shown below), then the problem is likely due to permissions. From the SQL MP guide, the requirements for DB discovery are:

· EXEC permissions for (sp_helpdb)

· Select from sys.databases table in the master database

Also, before running sp_helpdb, the discovery script will query to get a list of databases. In SQL 2005/2008, the query is:

SELECT name, state_desc FROM sys.databases WHERE source_database_id IS NULL

In SQL 2000, the query is:

SELECT name FROM sysdatabases

The difference is that in SQL 2005/2008, we have the “WHERE source_database_id IS NULL“ clause, which will eliminate snapshot databases…..so if the SQL instance has any snapshot databases, they will not be discovered. We also select the “state” column from sys.databases in SQL 2005/2008 DB discovery, and if the state is not “ONLINE”, then the discovery ends there….so this would be another reason why the database properties do not show up in OpsMgr.

Attached to this blog are debug version of the database discovery script:

DiscoverSQL2005DB_debug.txt – Use this for SQL 2005/2008

DiscoverSQL2000DB_debug.txt – Use this for SQL 2000

To run the script:

Rename to .vbs
Run the following command:

cscript DiscoverSQL2005DB_debug.vbs <fqdn> <Server\instance> "exclude:"

Replace the bold items with:

<fqdn> = Full Qualified Domain Name of the SQL Server (server.domain.com)

<Server\instance> = SQL Server instance that we want to discover DBs on. If it is the default instance, it will just be the server name (SERVER), otherwise it will be SERVER\INSTANCE

Sample output from my server:

Server name is jimmyhsql1.jimmyhdom.com

SQL instance is OpsDB

Command line is cscript discoversql2005db_debug.vbs jimmyhsql1.jimmyhdom.com jimmyhsql1\opsdb "exclude:"

Output (I only copied the output for the first couple DBs):

Entering DoDatabaseDiscovery function...
Connection string is Server=jimmyhsql1\opsdb;Database=master;Trusted_Connection=
yes
Error number is 0
Querying for list of non-snapshot databases...
Error number is 0
==================================
DatabaseName: master
DatabaseState ONLINE

Runing sp_helpdb master
ErrorNumber: 0
If no results are listed below, then sp_helpdb did not return anything....check
permissions

DatabaseSize: 4
DatabaseSizeNumeric: 4
LogSize: 0.5
LogSizeNumeric: 0
RecoveryModel: SIMPLE
Updateability: READ_WRITE
UserAccess: MULTI_USER
Collation: SQL_Latin1_General_CP1_CI_AS
DatabaseAutogrow: True
LogAutogrow: True
Owner: sa
==================================

==================================
DatabaseName: tempdb
DatabaseState ONLINE

Runing sp_helpdb tempdb
ErrorNumber: 0
If no results are listed below, then sp_helpdb did not return anything....check
permissions

DatabaseSize: 23.0625
DatabaseSizeNumeric: 23
LogSize: 1
LogSizeNumeric: 1
RecoveryModel: SIMPLE
Updateability: READ_WRITE
UserAccess: MULTI_USER
Collation: SQL_Latin1_General_CP1_CI_AS
DatabaseAutogrow: True
LogAutogrow: True
Owner: sa
==================================

SQL Agent Job Discovery not working?

jimmyharper — Fri, 05 Jun 2009 05:36:00 GMT

The SQL Server Management Pack includes an option to discover and monitor SQL Server Agent Jobs for SQL 2000/2005/2008. The Discovery for this is disabled by default:

I ran into an issue recently where all Agent Jobs for specific SQL Servers were not being discovered. Examining the event logs on the SQL Server, we see the following in the OpsMgr Event Log:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          6/4/2009 8:36:19 PM
Event ID:      21406
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
The process started at 8:36:18 PM failed to create System.Discovery.Data. Errors found in output:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 1\4595\SQLAgentJobDiscovery.vbs(106, 5) Microsoft VBScript runtime error: Type mismatch

Command executed: "C:\Windows\system32\cscript.exe" /nologo "SQLAgentJobDiscovery.vbs" {974F57A5-5705-B6B2-B8DC-1CA0B433DCD4} {46913442-CAC1-7E38-89B4-1A6B462ED0D0} OMDW.opsmgr.net OMDW.opsmgr.net OMDW\I01 I01 SQLAgent$I01"
Working Directory: C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 1\4595\

One or more workflows were affected by this.

Workflow name: Microsoft.SQLServer.2008.AgentJobDiscoveryRule
Instance name: SQLAgent$I01
Instance ID: {46913442-CAC1-7E38-89B4-1A6B462ED0D0}
Management group: PROD1

The “Type mismatch” error typically means that some variable in the script is returning with an incorrect data type. After examining the Discovery script and doing to troubleshooting, we determined that the problem was happening because the “Description” field for the Agent Job was NULL. This can be confirmed by running the “sp_help_job” Stored Procedure against the MSDB database on the SQL Instance (which is exactly what the Discovery script does:

This will also happen if any of the following properties of the job are NULL:

job_id
originating_server
name
description
category
owner

We probably won't ever see this with the job_id, originating_server, category or name properties, but we've seen it with the description and owner properties.

To correct this, we can simply enter some text in the description field of the Agent Job:

Note that the problem described above (Agent Job discovery failing when properties are NULL) happens on SQL 2005 and 2008 Agent Jobs. SQL 2000 Agent Job Discovery does not use the VBScript, and does seem to work in this scenario, but the NULL values are populated with the values of other Agent Jobs, so it is not accurate.

So, what if you have a very large number of SQL Servers and Agent Jobs and do not want to worry about making sure that all of them have text in the Description field? To take care of this, I created a “workaround” version of the SQL Agent Job Discoveries that will discover these jobs and enters NULL for the NULL propeerty.

Import the “Microsoft.SQLServer.200x.Discovery.CustomAgentJobDiscovery.xml” management pack
Disable the original Agent Job Discovery and enable the new one (“SQL Server 200x Custom Agent Job Discovery”):

Verify that the Agent Jobs are discovered: