So, the APM Wizard is great. We put a lot of effort in simplifying the most common scenario for configuration, as described in earlier posts such as this one http://blogs.technet.com/b/momteam/archive/2011/07/30/operations-manager-and-application-monitoring.aspx ; anyhow, there are additional settings that are buried within overrides, in a few places. This post wants to bring some of these to your attention.
One of such places is the Discovery called "Discovery of APM Agent properties" – which you can find, as shown in the screenshot below, by looking for the “.NET Application Monitoring Agent” class. This discovery contains a few interesting overrides. Some of them are related to “global” monitoring settings, while one of them is particularly important in relation to event throttling.
The other overrides are about enabling “global” settings for monitoring applications. While this is not the recommended approach, it is possible to bypass the APM Template entirely, and use these overrides instead – with some caveats:
Now, let’s get to the topic of throttling specifically. Another useful override in the above discovery rule is the following one:
As mentioned earlier, the main goal of Event Throttling is to keep CPU Utilization Low. Therefore it is NOT recommended to switch it off completely, as it is a useful “guard” for system resources. Anyhow, even while leaving it on, there are options to tweak the behavior of throttling – to make it more or less aggressive.
All these options can be configured thru overriding the "Apply APM Agent configuration rule" – again, targeted to “.NET Application Monitoring Agent”, see screenshots below:
While running, the APM agent continuously compares the count of monitored events with limits, and will provide those events to OpsMgr workflow for delivery to the management group only if the limits are not exceeded. In case your application goes “crazy” and starts throwing a lot of exceptions all of a sudden, this allows the APM agent to not add too much load, while still reporting the first exceptions of any “group” to you (see below about event groups).
Please note that the performance counters “% Exception Events / sec”, “% Performance Events / sec” and “Average Request Time” are NOT affected by this: these are the counters used in the monitors in OM, and we have described them earlier here http://blogs.technet.com/b/momteam/archive/2011/08/23/application-monitoring-working-with-alerts.aspx . They’ll keep reporting the real count of events having been “seen” on the agent and will keep track of all requests – but, due to throttling, not all events with full code-level details will be reported.
As I mentioned earlier on here http://blogs.technet.com/b/momteam/archive/2012/06/18/event-to-alert-ratio-reviewing-problems-and-understanding-trends-for-apm-data-in-opsmgr-2012.aspx there are 15 (FIFTEEN!) overrides that govern this. These are basically divided in 5 categories, and for each category there is a limit per MINUTE, per HOUR, and per DAY. All requests are counted by the agent and evaluated against these thresholds. Every time a period (minute, hour, day) passes, the counters to match against these limits are reset.
If you want to customize these limits, you need to understand how these limits look like and what counters are being evaluated to make the throttling decision. Below I attempt such explanation. Anyhow, be aware that we do NOT recommend you to increase ANY of the following limits – as it might have a negative impact on your application’s performance.
No discussion of APM throttling would be complete without touching on those “light” events. Anyhow, this has already been described very well by a former AVIcode employee, Alex – so I’ll defer you to his explanation rather than make up a new one – see here http://gotchahunter.net/2011/10/scom-apm-avicode-what-are-those-light-events/ . Additional related info are also on VIAcode’s blog here http://www.viacode.com/blog/2011/12/01/light-requests-optimization-logic-how-does-it-work/
One important note is that this cannot currently be switched off in OM12 in supported ways. Also interesting to note that “light” events don’t get counted against “problems” counts in many Advisor reports.
Another place is a rule that executes against a special Data Transfer singleton object in OM – this is responsible for continuously transferring APM data from OpsDB to DW database, perform APM data grooming – the settings for retention driving the grooming mechanism are documented here: http://technet.microsoft.com/en-us/library/jj159297
Another place is the paragraph in the documentation that explains how to “work with sensitive data” - here http://technet.microsoft.com/en-us/library/hh543995.aspx that also uses an override to define expressions for clearing up sensitive data out of function parameters (a commonly asked feature).
Other useful overrides useful to discover your applications are documented here as part of the documentation for the “Operations Manager APM Web IIS 7” MP - http://technet.microsoft.com/en-us/library/hh916929.aspx ; keep in mind that for IIS8 in SP1 there is an equivalent MP “Operations Manager APM Web IIS 8” and equivalent discoveries/rules. As described in the documentation, those rules work by searching specific file extensions, and discovering “ASP.NET applications” or “ASP.NET webservices”, depending on what they find (or what you tell them to find, via those overrides).
Also, there is one more rule for IIS-Hosted WCF endpoints which is not documented, that works pretty much the same way: it looks for *.svc file and use that to find WCF web services (objects of type “IIS Hosted WCF Web Service Endpoint” - this is a separate discovery in another MP (that is why it isn’t currently documented), but essentially is the same idea… in this case there is an actual DISCOVERY (not a rule) with an override-able property to look for different file extensions than “.svc” (screenshot from SP1 build… in RTM I the display name was slightly different, but it can be found in the same MP, same target…). This will find WCF services in both IIS7 and IIS8. Here it is:
If you need more information about the various endpoints – look at the query and report here: http://blogs.technet.com/b/momteam/archive/2012/08/22/apm-configured-endpoint-report.aspx
Before we set off, I just want to leave with a disclaimer that a lot of the overrides above (especially those related to throttling) can be lead you into an “unsupported” or “not recommended” state – increasing throttling limits can have a performance impact on your application: we put them in place to avoid overhead, and we think these to be useful as they are. Also, it is not guarantee they will work as described in future releases, and might even prevent upgrading in some cases. Also, as usual, this posting is provided "AS IS" with no warranties, and confers no rights. Use of included utilities are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.
With this in mind, I hope this journey into APM overrides was useful to understand how things work! Happy customization, and ‘till next time!
Thanks! - Great article!