Command Shell Examples
Useful SQL Queries
Alert Grooming - is it working or not? - Jonathan Almquist on Operations Manager - Site Home - TechNet Blogs

Alert Grooming - is it working or not?

Alert Grooming - is it working or not?

  • Comments 3
  • Likes

I've run across a couple instances where it appeared that alerts were not being groomed from the Operations Manager database.  Part of the confusion was due to querying alerts raised per day by resolution state.  A query such as this will give you a rundown of alerts generated per day, that have not yet been groomed from the OpsDB...along with their current resolution state.  The purpose of this query is not to evaluate whether or not alert grooming is working.

There is one important fact to keep in mind about alert grooming.  Alerts raised beyond, and resolved within, your resolved alerts retention period may mislead one into thinking some alerts should have been groomed.  In fact, they are just waiting for their day to come.

image 

Here's a one-liner to check if alert grooming is working as designed.  Just copy and paste into the Operations Manager Command Shell.

$Threshold=(Get-Date).AddDays(-(get-defaultsetting)[42].Value-1).ToUniversalTime().Date.AddMinutes(30);Get-Alert | Where {$_.TimeResolved -and $_.TimeResolved -lt $Threshold} | Measure-Object

If this command returns nothing, this is good.  This means there were no objects returned...hence, grooming has done it's job.  If it returns a result, this would indicate alert grooming is not working properly.

Example: Results returned one alert not groomed on it's last grooming interval.image

The results in the above image indicates a problem with alert grooming.  So, how do we figure out why grooming didn't do it's job?  Well, the alert grooming job is not complicated.  It simply selects alerts that have a resolution state = 255 and TimeResolved <> NULL.  Then calculates, if TimeResolved + Resolved Alert retention days = Today...groom alert.  So, if there is a problem I would first check the InternalJobHistory.

SELECT * FROM InternalJobHistory Order By InternalJobHistoryId DESC

Read more on this here.

I do not moderate this blog anymore. If you have a question regarding this post, send me a message.

Comments
  • Hi Jonathon,

    Thank you for this great post, it saved me hours of digging through Operations Manager.

    More feedback here : http://www.msexperts.org/blogs/mark/archive/2008/12/16/grooming-and-operation-manager.aspx

    Regards,

    Mark Rhodes

  • Hi Jonathan,

    I run the Operations Manager Command Shell.

    $Threshold=(Get-Date).AddDays(-(get-defaultsetting)[42].Value-1).ToUniversalTime().Date.AddMinutes(30);Get-Alert | Where {$_.TimeResolved -and $_.TimeResolved -lt $Threshold} | Measure-Object

    But I received the error:

    Get- Alert : Exception of Type 'System.OutOfMemoryExceptio' was throw.

    At line: 1 char:98

    + $ Treshold 0 <Get-Date>.ToUniversalTime<>.AddDays <-<get-defaultsetting> [42]. Value

    >.Date;Get-Alert <<<< ¦ Where {$_.TimeResolved -an $_.TImeResolved.Date -1t $Threshold} ¦ Measure-Object

     + CategoryInfo         : Not Specified: <:> [Get-Alert], OutOfMemoryException

     +FullyQualifiedErrorId : System.OutOfeMemoryException,Microsoft. EnterpriseManagement.OperationsManager.ClientShell.GetAlertCmdlet

    Any idea??

    Thank u

    Regards,

  • Good post, but error when trying to run the PS script in OpsMgr 2012. Replacing with:

    $Threshold=(Get-Date).AddDays(-(Get-SCOMAlertResolutionSetting)[42].Value-1).ToUniversalTime().Date.AddMinutes(30);Get-scomAlert | Where {$_.TimeResolved -and $_.TimeResolved -lt $Threshold} | Measure-Object

    worked, but with errors

    Unable to index into an object of type System.Management.Automation.PSObject.

    At line:1 char:65

    + $Threshold=(Get-Date).AddDays(-(Get-SCOMAlertResolutionSetting)[ <<<< 42].Value-1).ToUniversalTime().D

    30);Get-scomAlert | Where {$_.TimeResolved -and $_.TimeResolved -lt $Threshold} | Measure-Object

       + CategoryInfo          : InvalidOperation: (42:Int32) [], RuntimeException

       + FullyQualifiedErrorId : CannotIndex