Kevin Holman's OpsMgr Blog : self tuning

A tidbit on tuning some Self-Tuning threshold monitors

kevinhol — Fri, 31 Jul 2009 20:48:57 GMT

I wrote up a blog post long ago on loving and hating STT (Self Tuning Threshold) Monitors HERE

This is something I noticed in R2…. I don't remember seeing this in SP1 before…

I was getting some alert noise on my Windows Server 2008 Terminal Server – from the “Total Number of Active Sessions” monitor. This is a self-tuning threshold monitor that examines the active session count from perfmon, and will alert if it goes outside the calculated baseline. This is cool – however – my Terminal Server goes from having nobody connected (zero) to as many at 8 connected. Well – any counter that is often, or regularly at “zero” is not a good candidate for a self-tuning threshold monitor – because the monitor will be noisy. So – I disabled this monitor… since I don't care. If I did care – I would just create my own static threshold monitor.

Well – as soon as I did that – I started seeing this event in the agent event logs:

Log Name:         Operations Manager
Source:              Health Service Modules
Date:                  7/31/2009 11:13:46 AM
Event ID:           10123
Task Category: None
Level:                 Error
Keywords:          Classic
User:                  N/A
Computer:          OMTERM.opsmgr.net
Description:
Duplicate Signature ID usage detected for STT Signature ID: 'SignatureID:[Microsoft.Windows.Server.2008.TerminalServicesRole.Service.TerminalServer.InactiveSessions.Sig]\{697C0377-22AB-4C10-F23D-A353A58DB35E}'. The STT Rule and monitor for this ID might differ in configuration - please reconcile the settings.
Alternately, this Signature ID may already be in use. Assign a previously unused Signature ID for this STT Rule/Monitor.
Module will be unloaded.
Signature ID: 'SignatureID:[Microsoft.Windows.Server.2008.TerminalServicesRole.Service.TerminalServer.InactiveSessions.Sig]\{697C0377-22AB-4C10-F23D-A353A58DB35E}'

Workflow name: Microsoft.Windows.Server.2008.TerminalServicesRole.Service.TerminalServer.InactiveSessions.BaselineCollection
Instance name: Terminal Server
Instance ID: {697C0377-22AB-4C10-F23D-A353A58DB35E}
Management group: PROD1

I was getting these every time there was a config update on the agent! (picture below filtered for this event ID only)

What this is saying – is basically that all Self-Tuning Threshold monitors have a corresponding rule that goes with them – and there is a mismatch between the configuration of rule and monitor.

So – in this case – I need to go find the corresponding rule (general will have the same/similar name) and set the overrids on that rule to be the same (disabled in this case) Most of the time you can search on “Baseline” or “Collection” to find them… and they will target the same class as the monitor:

Once I set this override – those events go away on my agents and all is well in OpsMgrLand again. Time to hit the Relax button.

Boosting OpsMgr performance - by reducing the OpsDB data retention

kevinhol — Wed, 05 Nov 2008 02:24:40 GMT

Here is a little tip I often advise my customers on.....

The default data retention in OpsMgr is 7 days for most data types:

These are default settings which work well for a large cross section of different agent counts. In MOM 2005 - we defaulted to 4 days. Many customers, especially with large agent counts, would have to reduce that in MOM 2005 down to 2 days to keep a manageable Onepoint DB size.

That being said - to boost UI performance, and reduce OpsDB database size - consider reducing these values down to your real business requirements. For a new, out of the box management group - I advise my customers to set these to 2 days. This will keep less noise in your database as you deploy, and tune, agents and management packs. This keeps a smaller DB, and a more responsive UI, in large agent count environments.

Essentially - set each value to "2" except for Performance Signature, which we will change to 1. Performance Signature is unique.... the setting here isnt actually "Days" of retention. It is "business cycles". This is for self-tuning threshold ONLY. This data is used for calculating business cycle based self-tuning thresholds. There is NO REASON for this ever to be larger than the default of "2" business cycles.... and large agent count environments can see a performance benefit by bumping this down to only keeping "1" business cycle.

Then - once your Management group is fully deployed, and you have tuned your alert, performance, event, and state data.... IF you have a business requirement to keep this data for longer - bump it up.

Keep in mind - this will NOT cause you to groom out Alerts that are open - only closed alerts, and still will keep your closed alerts around for a couple days.

These settings have no impact on the data that is being written to the data warehouse - so any alert, event, or perf data needed will always be there.

Self Tuning Thresholds - love and hate

kevinhol — Wed, 19 Mar 2008 07:20:00 GMT

****** The references in this post to the Exchange Management Pack have been changed... many of these issues below have been addressed in the updated MP: http://blogs.technet.com/kevinholman/archive/2008/08/22/updated-exchange-2003-mp-released-version-6-0-6387-0.aspx

Self tuning thresholds are a new concept for OpsMgr. They are awesome - because they will "learn" what is normal for a performance counter, and alert when the value is outside of the learned baseline. This is great when we have performance counters that will vary widely from company to company, and we don't know a good static setting.

Problem is.... if the counter being monitored varies widely on a regular basis... these monitors are extremely noisy... and generate the massive amount of alerts and state changes that they were designed to control.

There are a couple good blog posts on these already. Probably the best one I have read is here: http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!183.entry We will be referring to this several times.

One of the complaints about self-tuning thresholds.... is that the numbers reflected in the baseline don't tell us anything about the actual values. This is true.... these are based on an internal algorithm... so people see this "2.81" or "3.31" and don't understand what it has to do with anything about our performance counter.

First - lets take a look at the basic components of a STT: We will create a new unit monitor. Under windows performance, self tuning thresholds.... we have several types to choose from. The most common are going to be a 2-state or 3-state baselining.... depending on how many states we want. For this example - we will choose a 2-state baselining.

Let's give it a name, choose Windows Server as the target, and choose the performance Parent Monitor. To keep this simple - lets choose Processor\% Processor Time\_Total as our performance Object\Counter\Instance. Set the interval to 1 minute.

Now - we get to adjust the business cycle. I'm picking one day for this example. Typically - you would choose a week.... especially if your server behaves differently on different days of the week.

We can choose how many business cycles to wait before alerting.... most of the time 1 business cycle is fine.

On "Sensitivity" we have a nice slider from "Low" to "High". In general.... we will be choosing a low sensitivity for our custom rules. Low = lest alerts, wider baseline range. I will explain the numeric values for each setting later.

On the "Configure Health" screen.... within the envelope will be Healthy, and above the envelope will generate a state change and (optionally) an alert.

Groovy. So - what did we just really create? Well certainly - we created a monitor:

But if we look at rules, on the same target.... we also created some rules:

One of the rules is to simply collect the performance data. The other collects signature data. Both on the same frequency we specified earlier.

So now.... on to the most important thing - the numbers. When we created our 2-state baselining monitor - we pretty much accepted all defaults.... except we pick low sensitivity. To see these numbers - create an override for all objects of type, and you can see what defaults, and low equal:

So "inner" is 4.01 while "outer" is 4.51 We will look at these numbers more later. This is important - because we will use these to adjust and override other counters later.

Also - on the signature collection rule that was created - a sensitivity value was placed:

So.... lets try and find out how each setting affects these numbers - to better understand them.

I created 5 Self-Tuning 2-state baseline monitors.... each with a different sensitivity setting.... starting with low:

Low: Inner: 4.01 Outer: 4.51 Rule Sensitivity: 4.01

Low-Mid: Inner: 3.77 Outer: 4.27 Rule Sensitivity: 3.77

Mid: Inner: 3.29 Outer: 3.79 Rule Sensitivity: 3.29

Mid-High: Inner: 2.81 Outer: 3.31 Rule Sensitivity: 2.81

High: Inner: 2.57 Outer: 3.07 Rule Sensitivity: 2.57

That will give us a good baseline to use - when tuning these rules. We can see that default inner sensitivity ranges from 2.57 to 4.01, and outer ranges from 3.07 to 4.51. The larger the numbers.... the less sensitive the baseline range, and therefore fewer alerts. The difference between the numbers is always .5

To tune these self tuning alerts..... we simply need to adjust these values, for the Performance signature rule, and the corresponding baselining monitor.

Here is a list of some very common noisy STT's - taken from the link above:

ALERT=Information Store Transport Temp Table is outside the calculated baseline

RULE=Baseline Collection Rule for Information Store temp table number of entries (Rules, of type Exchange Queue)

MONITOR=IS Transport Temp Table Monitor (Exchange Queue, Entity Health, Performance)

ALERT= Mailbox Store Send Queue is outside the calculated baseline

RULE=Baseline Collection Rule for Mailbox Store Send Queue Length (Rules, of type Exchange Queue)

MONITOR=MB Store Send Queue Monitor (Exchange Queue, Entity Health, Performance)

ALERT=SMTP Local queue is outside calculated baseline

RULE=Baseline Collection Rule for SMTP Server Local Queue (Rules, of type Exchange Queue)

MONITOR=SMTP Local Queue Monitor (Exchange Queue, Entity Health, Performance)

ALERT=SMTP Messages in the Queue Directory is outside calculated baseline

RULE=Baseline Collection for SMTP Message Queue Directory (Rules, of type Exchange Queue)

MONITOR=SMTP Message Queue Directory Monitor (Exchange Queue, Entity Health, Performance)

ALERT=SMTP Remote Queue is outside the calculated baseline

RULE=Baseline Collection Rule for SMTP Server Remote Queue Length (Rules, of type Exchange Queue)

MONITOR= SMTP Remote Queue Monitor (Exchange Queue, Entity Health, Performance)

ALERT=SMTP Remote Retry Queue is outside the calculated baseline

RULE=Baseline Collection Rule for SMTP Server Remote Retry Queue Length (Rules, of type Exchange Queue)

MONITOR=SMTP Remote Retry Queue Monitor (Exchange Queue, Entity Health, Performance)

ALERT=IS Virtual Bytes is outside the calculated baseline

RULE=Baseline Collection Rule for IS Virtual Bytes (Rules, of type Exchange IS Service)

MONITOR=IS Virtual Bytes Monitor (Exchange IS Service, Entity Health, Performance)

ALERT= Number of RPC requests is outside the calculated baseline

RULE=Baseline Collection Rule for IS RPC Requests (Rule, of type Exchange IS Service)

MONITOR=IS RPC Requests Monitor (Exchange IS Service, Entity Health, Performance)

What we see - is that most of the default STT's in the management packs are set to "Medium-High" sensitivity.... or a Inner of 2.81 and outer of 3.31. This is likely too sensitive, and needs to be adjusted. Essentially... start by bumping up to the next set of numbers for both values, and adjusting them from Mid-High, to Mid, Mid-Low, or Low.

Here are the steps from the above blog post... with a few changes:

Steps to resolve: (perform all of these steps for each Alert in your environment which needs to be tuned)

Find the rule that applies to the alert. (To find the rules, it’s easiest to change the scope to filter by the two areas that we need - which are the Exchange Queue and Exchange IS Service. Both of these are available when you click on scope and choose the option to view all targets. Then find rules with “Baseline Collection” as the start. This scopes it down to about 17 rules versus over 6000.) Details on the names of each of the above rules are listed below. Disable the rule (Right-click on the rule, overrides, disable the rule for all objects of type: Exchange Queue, click yes to accept).
Change the rule sensitivity to 3.29 (Right-click on the rule, Overrides, Override the rule, For all Objects of type: Exchange Queue, check the Sensitivity parameter and set it to 3.29 (or higher if needed), click OK).
Find the monitor that applies to the alert. This can be found by searching or scoping to the type of object identified for the rule. Disable the monitor (Right-click on the monitor, Overrides, Disable the monitor for all objects of type: Exchange Queue, click yes to accept).
Change the monitor inner sensitivity to 3.29 (Right-click on the monitor, Overrides, Overrides the monitor, For all Objects of type: Exchange Queue, check the Inner Sensitivity parameter and set it to 3.29 if it’s not already set to that value, click Ok).
Change the monitor outer sensitivity to 3.79 (Right-click on the monitor, overrides, Overrides the monitor, For all Objects of type: Exchange Queue, check the Outer Sensitivity parameter and set it to 3.79 if it’s not already set to that value, click Ok).
Re-enable the monitor. (Right-click on the monitor, click on Overrides Summary, delete the override that says Type, Exchange Queue, Enabled, False).
Go back to the rule identified in step #1 and re-enable the rule. (Right-click on the rule, click on Overrides Summary, delete the override that says Type, Exchange Queue, Enabled, False).

NOTE:

The "outer" sensitivity does not matter. It is an early design leftover, and does not have an impact. Only the inner sensitivity makes a difference in tuning. There has been some conflicting information in the newsgroups, but this information has been verified with the dev team.

The only requirements... on the outer, is that it be a larger number than the inner. So when adjusting - focus on bumping the inner in .5 increments, and just make sure the outer is any number higher than the inner.... such as .1 higher than inner.

In Summary:

1. Not all counters are good candidates for STT’s based on the performance counter pattern.

2. Some of our built in STT’s are a bit on the sensitive side and should be tuned. If the alert noise is high - start by tuning - lower the sensitivity.

3. Some of our built in STT’s are targeting a perf counter that is not a good candidate for an STT. (eg… STMP queue, or any perf counter that is often “zero value” when healthy).

4. There is no simple way to view the learned baseline of an STT…. the “show baseline” in graph view does not display a range.

5. Any time a customer is not happy with the results of a STT monitor – they should simply create a static threshold monitor. This is very basic and provides the best solution. If you cant tune noise out of a STT, or you NEED to know at what threshold an alert will be generated.... then simply turn off the STT, and create an identical static threshold monitor, of the average, or consecutive samples above, type.