MOM 2005: Alert when a percentage of machines in a computer group fail to respond to ping requests.

shinsley — Sat, 02 Dec 2006 00:39:00 GMT

I was recently asked if it would be possible to generate an alert when a certain number of machines in a computer group become unreachable. MOM already generates alerts when an agent fails to heartbeat and even notes when the heartbeat failure is accompanied by ping failures. This is great to the owner of an individual system, but what if we're interested in something a bit more pandemic?

Terminal or web farm can have hundreds or even thousands of individual servers that are part of a larger collective. Failure of an individual machine wouldn't cause the average person's blood pressure to go up. On the other hand, how can we tell when a large number of systems within a web farm goes down without watching all those individual heartbeat alerts?

Approach

Let use information that MOM already collects to avoid increasing network traffic or database size. For our purposes, the information we need comes in the form of active Heartbeat failures that include ping failures. We'll need to do a few things:

Create a computer group containing machines to be monitored for ping failures,
Create a script to monitor systems in your new group for ping failures,
Create a rule to run your new script.

1 - Create and Populate MOM Computer Groups

If you already have a computer group in mind, great move on to step two. If you don't have a computer group or you have one that has to be manually populated, open the MOM Administrator console and read on:

a) From the Admin console, right-click Computer Groups and create a new computer group.

b) Pause to appreciate the welcome screen then click Next,

c) Give your new computer group a meaningful name and description before clicking Next,

Example: Terminal Server Farm A

d) Pause once more to realize that laziness is indeed the mother of invention.

The next few pages offer several option to automate membership of your newly created computer group. You can use manual inclusion/exclusion, name searches using filters, wild-cards and regular expressions or my personal favorite; formulas.

e) Click next about 5 times until you come to the formulas page

f) Select Specify a formula for this computer group, click Attribute and New.

g) Create a new attribute by using the controls to browse the local or a remote registry. Select the key or value to be used to identify machines that could be used to identify systems that should belong in your new computer group. Really, it's that easy!

Tip: Do you have your own custom registry keys such as application service or server build markers? You can add your own custom registry keys to be used during attribute discovery or use any other key or value that is unique to this particular collection of computers that belong in the new group.

Example:

HKLM\Software\XYZSoftware

HKLM\System\CurrentControlSet\Services\MyCrashtasticCustomApplication

For additional information on creating computer groups see Creating Computer Groups in the MOM 2005 Management Pack Developers Guide.

2 - Create a script to monitor active ping failure for computer in a given computer group.

MOM does a lot of the work for us... The number of systems within a computer group and anything we could possible want to know about the alerts is in the OnePoint database. All we need is a little t-SQL and VBScript.

a) Locate the Scripts folder, right-click and choose New Script

b) Provide a name, description and language and click next. In this case, I'm using VBScript...

c) Insert your script and click next...

The following is provided as an example:

'*************************************************************************************************
'*percentFailed.vs
'*
'* Written by: Scott W. Hinsley
'* Date: 11/29/06
'*
'* Requirements:
'*
'* - MOM 2005
'* - Script Parameters:
'* MOM_COMPUTER_GROUP_NAME
'*  MOM Computer Group to be monitored for % ping failure.
'* PERCENT_FAILED_THRESHOLD
'*  Failure threshold expressed as a percentage of system in MOM_COMPUTER_GROUP at
'*  which an alert should be generated. Expressed in a range of 1 to 100.
'* ONEPOINT_SERVER
'*  Name of SQL Server hosting the OnePoint database.
'*
'* Provided "AS IS" without warranty.
'*
'*************************************************************************************************

Option Explicit
On Error Resume Next

Dim strMomComputerGroupName
Dim strPercentFailedThreshold

'Script Parameters
strMomComputerGroupName = ScriptContext.Parameters.Get("MOM_COMPUTER_GROUP_NAME")
strPercentFailedThreshold = ScriptContext.Parameters.Get("PERCENT_FAILED_THRESHOLD")
strOnePointServer = ScriptContext.Parameters.Get("ONEPOINT_SERVER")

' Alert Constants

CONST ALERT_DESCRIPTION_STRING = "'% does not respond %'"
CONST ALERT_RESOLVED = 255
CONST ALERT_INFORMATION = 20
CONST ALERT_WARNING = 30
CONST ALERT_ERRROR = 40
CONST ALERT_CRITICALERROR = 50
CONST ALERT_SERVICEUNAVAILABLE = 70

'Database Constants

CONST adOpenStatic = 3
CONST adLockOptimistic = 3

Dim strQuery
Dim dblPercentFailed
Dim strAlertDescription
Dim strAlertName
Dim strName
Dim intLevel
Dim strDescription

'ADO variables
Dim objConnection
Dim objRecordSet

'strQuery: Contains query used to get MOM_COMPUTER_GROUP member count
'     and the number of active heartbeat failures with ping
'       failures. Used to form percentage to be compared with
'         PERCENT_FAILED_THRESHOLD.
'
'    Consider making strQuery a stored procedure to enhance
'  performance and code effeciency.

strQuery = "DECLARE" & _
    " @vComputerGroupGUID NVARCHAR(39)," & _
    " @SearchString NVARCHAR(21)," & _
    " @vExcludeResolutionState NVARCHAR(3)," & _
    " @PingFailCount DECIMAL(4,0)," & _
    " @ComputersInGroup DECIMAL(4,0)," & _
    " @vComputerGroupName NVARCHAR(100)," & _
    " @PercentFailed DECIMAL(3,2)" & _
  " SET @vComputerGroupName = " & "'" & strMomComputerGroupName & "'" & _
  " SET @SearchString = " & ALERT_DESCRIPTION_STRING & _
  " SET @vExcludeResolutionState = " & ALERT_RESOLVED & _
  " SET @vComputerGroupGUID =" & _
     " (SELECT GUID" & _
   " FROM [OnePoint].[dbo].MSFT_ComputerGroup" & _
   " WHERE [OnePoint].[dbo].MSFT_ComputerGroup.Name = @vComputerGroupName)" & _
  " SET @PingFailCount =" & _
      " (SELECT COUNT([OnePoint].[dbo].[SDKAlertView].ComputerName) AS ActiveCount" & _
      " FROM [OnePoint].[dbo].[SDKAlertView]" & _
   " INNER JOIN [OnePoint].[dbo].[SDKComputerToComputerGroupView]" & _
    " ON [OnePoint].[dbo].[SDKAlertView].ComputerName =" & _
                " [OnePoint].[dbo].[SDKComputerToComputerGroupView].ComputerName" & _
   " WHERE [OnePoint].[dbo].[SDKComputerToComputerGroupView].ComputerGroupGUID =" & _
    " @vComputerGroupGUID" & _
    " AND [OnePoint].[dbo].[SDKAlertView].Description LIKE @SearchString" & _
    " AND [OnePoint].[dbo].[SDKAlertView].ResolutionState <> @vExcludeResolutionState)" & _
  " SET @ComputersInGroup = (SELECT NumberOfComputers" & _
     " FROM [OnePoint].[dbo].[ComputerGroupView]" & _
     " WHERE ComputerGroupName = @vComputerGroupName)" & _
  " SET @PercentFailed = (SELECT @PingFailCount/@ComputersInGroup)*100" & _
  " SELECT @PercentFailed AS PercentFailed"

'Creating the data connection...

Set objConnection = CreateObject("ADODB.Connection")
Set objRecordSet = CreateObject("ADODB.Recordset")

'The following connection string demonstrates using a non-trusted SQL account.
'Update credentials for your particular needs.

objConnection.Open _
    "Provider=SQLOLEDB;Data Source=" & ONEPOINT_SERVER & ";" & _
        "Trusted_Connection=No;Initial Catalog=OnePoint;" & _
         "User ID=<ReadOnlySQLAccount>;Password=<password>;"

objRecordSet.Open strQuery, objConnection, adOpenStatic, adLockOptimistic

objRecordSet.MoveFirst

dblPercentFailed = CDbl(objRecordSet.Fields.Item("PercentFailed"))

'Compare value retrieved with strQuery with PERCENT_FAILED_THRESHOLD
'Alert if PERCENT_FAILED_THRESHOLD is exceeded.

If dblPercentFailed > strPercentFailedThreshold Then
strAlertDescription = dblPercentFailed & "%" & " of systems in " & strMomComputerGroupName & _
    " failed to respond to ping attempts. This is greater than the" & _
    " defined threshold of: " & strPercentFailedThreshold & "%"

strAlertName = strMomComputerGroupName & ": Ping failure threshold exceeded."

AlertRaise strAlertName, ALERT_CRITICALERROR, strAlertDescription

Else
End If

'AlertRaise is Called if PERCENT_FAILED_THRESHOLD
'exceded.

Sub AlertRaise(strName, intLevel, strDescription)
Dim objAlert
Set objAlert = ScriptContext.CreateAlert()
objAlert.Name = strName
objAlert.AlertLevel = intLevel
objAlert.Owner = "[unassigned]"
objAlert.ResolutionState = 0
objAlert.Description = strDescription
ScriptContext.Submit(objAlert)

End Sub

'* End VBScript
'*************************************************************************************************

d) Add any necessary parameters to your script. In this case, I use the following parameters:

MOM_COMPUTER_GROUP_NAME
PERCENT_FAILED_THRESHOLD
ONEPOINT_SERVER

* See code comments for details.

3 - Create a rule that will run the script on a scheduled interval.

Find an appropriate spot for your new rule. You may want to create a new rule group... I'll leave that to you.

a) Right-click Event Rules and select Alert or Respond to an Event.

b) Select New | Timed Event | OK. Configure an interval texceededhat is appropriate for your needs and click OK.

c) Click Next until you come to the Reponses screen. This is where you will put your script to work.

d) On the reponses screen, select Add | Launch a script.

e) ResponsesresponsesSelect your newly created script and and configure the required parameters.

f) Last but not least, complete the company knowledge and provide a descriptive name for you new rule.

Enjoy!

Scott Hinsley's WebLog : MOM 2005

MOM 2005: Alert when a percentage of machines in a computer group fail to respond to ping requests.