Custom APM Rules for Granular Alerting

Custom APM Rules for Granular Alerting

  • Comments 10
  • Likes

Michael wrote a post in August, about working with Alerts.  One of the key takeaways from that post, is that – for each application and for each application component – we have FOUR Alerting rules, that can be turned on or off from the APM Template. Quoting that post:

Alerting Rules

There is a rule for each type of event we alert on: Performance, Connectivity, Security and Application Failure. We raise an individual alert when those types of events are detected in the monitored application. These alerts do not affect the health state of the monitored application since a single performance or exception event doesn’t mean your application is unhealthy.

The post above was written during Beta, and the UI has improved since then, but the rules are still there and work in the same way that was described. 

In the RC build the options in the Template UI look like this:

APM Template Options

While the Authoring Guide describes what the options are used for, here I want to show you a bit more of what settings they drive “under the hood”.

The two checkboxes on the top turn ON and OFF:

  • Alerting for “Performance” events (“Turn on performance event alerts”)
  • Alerting for “Exception” events (“Turn on exception event alerts”) – when this checkbox is enabled, it allows you to configure three more options (a breakdown of the type of exceptions):
    • Security alerts
    • Connectivity alerts
    • Application failure alerts

Together, these checkboxes essentially enable and disable the previously-mentioned Alerting rules. You can find them in the Authoring pane of the Operations Console, under “Rules”:

APM Alerting Rules

The names should be all pretty self-explanatory to understand which one maps to which option.

While this mechanism is flexible enough for the most common usage, I want to show you how the whole thing works end to end, and show you how the solution is powerful and flexible, and how you can do even more with APM and configure even more granular alerts than the UI allows you to – with little XML editing.

If you look at those rules (in the Operations Console, or by un-sealing the MP and watching its XML), they are all very similar: they have a Data Source looking for the incoming APM events, and a Write Action that turns them into Alerts.

APM Rule Default workflow

The Data Source has a configuration as follows:

APM Data Source Configuration

 

As you can see in the screenshot above, the same data source is used for all four rules, and the “AspectType” is used to tell apart Performance, Connectivity, Security and Application failure events.

This is great for most situations, and our default settings have been chosen with the assumption that Operations folks would be more interested in Performance, Connectivity and Security events – those where they might be able to operate – but not necessarily about “Application failure” events, since those are (often) a bug in the code, and (typically) only a developer can fix those exceptions.

Even if this model is great, I found that, in some situations, people might want to have even more fine-grained alerting rules defined. In particular, I think the connectivity and security aspects are quite well-defined in our APM default configuration and they are typically not noisy unless something is really wrong. The same is not necessarily always true for Performance events and Application failures. For example you might want to get:

  1. Performance event alerts only for a specific web page or method (this can also be achieved by defining a transaction, but depending on the situation one approach or the other might be preferred – I’ll explain transactions in a future post)
  2. Performance event alerts for all cases but excluding a particular page/method which is “well known” to be slow but can’t be fixed/optimized (this is something that cannot be achieved even with a transaction)
  3. Application failure event alerts only for certain type of exceptions and not for other ones
  4. Application failure events alerts for all exception but excluding a particular page which is known to throw an un-handled exception but doesn’t cause bad user experience or can’t be fixed
  5. Application failure events alerts for all exception but excluding a particular exception type which can’t be fixed by the developer
    • one specific situation where #5 is desirable is when someone, calling a page which is not present on an ASP.NET application, will result in throwing a “System.Web.HttpException” with an HTTP 404 Error (not found) code – this is by design in ASP.NET: if I call an .aspx page, the ASP.NET engine will try to retrieve it and will be throwing an HTTP error;  this could cause a lot of noise in case a crawler or vulnerability assessment tool hits the site searching for “well-known” but not-present pages (this is actually something that we observed on the production deployment monitoring parts of the microsoft.com website)

For all these situations (and more) there is a a fairly simple solution: writing new APM Alerting rules with an added Expression Filter. Basically we’ll have a workflow which looks like the following:

APM Rule Custom workflow

One such a sample rule is pasted below. It looks very similar to (and in fact, it is derived from) the “default” APM alerting rules described earlier – only the Condition Detection highlighted has been added. This one rule represents example #5 from the list above – essentially, it should filter out those “page does not exist” 404 errors, but still alert on every other exception.

<Rule ID="Apm.AlertAppFailureAspectRule.Exclude404.message.Sample" Enabled="false" ConfirmDelivery="true" Target="APM!Microsoft.SystemCenter.Apm.ApplicationInstance" Remotable="false"> <Category>Alert</Category> <DataSources> <DataSource ID="LOBProvider" TypeID="APM!Microsoft.SystemCenter.Apm.LobDataProvider"> <Name>$Target/Property[Type="APM!Microsoft.SystemCenter.Apm.ApplicationInstanceBase"]/ApplicationName$</Name> <AspectType>applicationfailure</AspectType> </DataSource> </DataSources> <ConditionDetection ID="FilterExceptionClass" TypeID="System!System.ExpressionFilter"> <Expression> <Not> <Expression> <And> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="String">EventData/exceptionclass</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="String">System.Web.HttpException</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <RegExExpression> <ValueExpression> <XPathQuery Type="String">EventData/message</XPathQuery> </ValueExpression> <Operator>ContainsSubstring</Operator> <Pattern>does not exist</Pattern> </RegExExpression> </Expression> </And> </Expression> </Not> </Expression> </ConditionDetection> <WriteActions> <WriteAction ID="AlertWriteAction" TypeID="Health!System.Health.GenerateAlert"> <Priority>1</Priority> <Severity>1</Severity> <AlertMessageId>$MPElement[Name='Apm.AlertAppFailureAspectRule.Exclude404.message.Sample.AlertMessage']$</AlertMessageId> <AlertParameters> <AlertParameter1>$Target/Property[Type="APM!Microsoft.SystemCenter.Apm.ApplicationInstanceBase"]/ApplicationName$</AlertParameter1> <AlertParameter2>$Target/Host/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/PrincipalName$</AlertParameter2> <AlertParameter3>$Data/EventData/exceptionclass$</AlertParameter3> <AlertParameter4>$Data/EventData/message$</AlertParameter4> <AlertParameter5>$Data/EventData/name$</AlertParameter5> <AlertParameter6>$Data/EventData/ViewDetail$</AlertParameter6> </AlertParameters> <Suppression> <SuppressionValue>$Data/EventData/eventConsolidationHash$</SuppressionValue> </Suppression> <Custom1>ApplicationFailure</Custom1> </WriteAction> </WriteActions> </Rule
> 

This rule still produces alerts for other exceptions that “look and feel” pretty much like the built-in rules, but will not raise an alert for those HTTP 404’s “file does not exist” errors. Although be aware that the example above will not work on localized .NET Framework/Windows version, because I am searching for a English string (“does not exist”) in the error message. This is not really meant as a production-quality MP considering all cases, just as a quick example of how you can build your own workflows by adding filtering criteria, and my goal it mostly to help you understand how the APM pieces fit together in Operations Manager 2012 so that, with that knowledge, you can get creative and adapt it to your needs.

Anyway, an alternative would be digging out the actual HTTP Error code, which is buried down in the DataItem as well. To do so, we can rewrite our Expression Filter as follows:

<ConditionDetection ID="FilterExceptionClass" TypeID="System!System.ExpressionFilter"> <Expression> <Not> <Expression> <And> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="String">EventData/exceptionclass</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="String">System.Web.HttpException</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="String">EventData/log/events/event[1]/variable[1]/variables/variable[name='_httpCode']/value</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="String">404</Value> </ValueExpression> </SimpleExpression> </Expression> </And> </Expression> </Not> </Expression> </ConditionDetection
> 

Since the code above is is not easily readable due to the blog layout (but should be possible to copy/paste it just fine), a Management Pack with both variations of this rule is attached at the end of this post. It also contains two more (fairly similar) examples, for a total of three rules based on “Application Failure” events and one on “Performance” events. All the rules are disabled by default to prevent duplicate alerts starting to appear in your environment as soon as you import the MP – if you use these rules, you might want to disable the checkboxes in the template for the “built-in” rules, first. You can then decide to turn these new rules on by default, or selectively thru overrides, as you would do with any other rule.

Please also note that some of the criteria contained in the condition detection filters cannot be edited from the Operations Console, and alert messages token replacement will also most likely break if edited thru the GUI. These things are best edited in XML.

In addition, these criteria might need to be revisited fairly often as part of your tuning – known problems could became worth considering again, and new issue might appear that you want to start filtering out, and so on.

These rules with their filters will only affect Alerting; all APM events will still be collected in the database and be visible thru the AppDiagnostics console – as the actual event insertion is driven by a different rule (using the same data source module, but a different write action).

Once events are stored and visible in AppDiagnostics, we also provide ways to automatically delete them from the database, or mark as “by design” those events that aren’t considered useful or interesting, or appear to add noise. This is the Problem Management feature in AppDiagnostics (Rules Management Wizard), which – while it doesn’t prevent events from being stored or alerts from being raised in the first place - it helps keeping your database “clean” and I like to consider it a sort of “intelligent” grooming. There would be a lot to be said about the Problem Management feature - I’ll try to come back to this feature and its rules in a future post.

 

Happy .Net Monitoring!

 

Disclaimer

This posting is provided "AS IS" with no warranties, and confers no rights. Use of included utilities are subject to the terms specified at http://www.microsoft.com/info/copyright.htm.

Attachment: APM.AddOn.Sample.zip
Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • It's possible to bind these rule to a specific application version?

  • Antonio, you can simply enable them thru overrides for specific groups. Ideally, when you run the APM template to configure that application, you would already be using a group to scope the configuration just to the right machine - you can use th same group here.

  • Gerat post as usual Daniele, can you share the EventData schema? Some XML samples would be welcome, as well :-).

    - Daniele

  • Daniele - I have examples from our unit tests but I can't share those, and they are not overly complex; *real* events will be much more talktative. An easy way to figure them out is simply by looking in the database. There are two places where you can see the XML - in the 'context' column of the 'dbo.Alert' table you can see the full EventData. If that gets truncated (some are fairly big) you can look in the 'apm.EVENT' table - the 'ROWDATA' contains the event, which should be the same as everything you see under 'EventData/log'. Does that help?

  • Lovely, thx a mil

    - Daniele