Kevin Holman's OpsMgr Blog : database

Maintenance mode – tying the text of the category to the database

kevinhol — Fri, 05 Jun 2009 08:59:36 GMT

I havent seen this discussed before – so I figured I would post this.

In the OpsDB and DWDB – we keep some tables names MaintenanceMode and MaintenanceModeHistory.

When you place an object into maintenance mode – we will log a row in the database for this object. You could potentially write reports against this data in the data warehouse, and report on MM history.

From the following query:

select * from dbo.vMaintenanceModeHistory

There is a column named “ReasonCode” This has a numeric value. However – in the UI – this correlates to the “Category” that you must select when you place an object into MM:

Here is a table which sorts out the numeric reason code vs the text in the UI:

Other (Planned)	0
Other (Unplanned)	1
Hardware: Maintenance (Planned)	2
Hardware: Maintenance (Unplanned)	3
Hardware: Installation (Planned)	4
Hardware: Installation (Unplanned)	5
Operating System: Reconfiguration (Planned)	6
Operating System: Reconfiguration (Unplanned)	7
Application: Maintenance (Planned)	8
Application: Maintenance (Unplanned)	9
Application: Installation (Planned)	10
Application: Unresponsive	11
Application: Unstable	12
Security Issue	13
Loss of network connectivity (Unplanned)	14

Populating groups from a SQL server CMDB – step by step

kevinhol — Tue, 27 Jan 2009 21:54:00 GMT

Boris wrote a cool article HERE on how to populate a group of computers in OpsMgr, from an external source…. such as active directory. In his published example – you run an LDAP query to AD, to return a recordset list if computers, in order to populate them into a group.

This post will extend that, by showing how to do the same thing – but using a SQL database as the CMDB source for populating groups, instead of AD. I had a customer who wanted to do this – to dynamically create groups for the purpose of scoping console views and notifications, to the teams that “owned” the different servers. The CMDB contained this data, but it changes often, so manually controlling this proved to be a pain.

Here is a very simple example of the CMDB, which contains the ServerName, and the team that owns the server, in the “ServerList” table:

As you can see… I can easily write a SQL query to show ONLY servers owned by TEAM 1:

Let’s use this data source… to populate three groups. Team 1 Group, Team 2 Group, and Team 3 group.

First – I will post my finished XML example at the bottom – go grab that and open it – it will help you follow along with the XML requirements.

In the XML… we basically need 4 components:

1. We need to define the name of the MP, and add references to other MP’s we will depend on. (<Manifest> Section)

2. We need to define the groups themselves, and then define the relationships (stating that they will contain Windows Computer Objects) (<TypeDefinitions> section)

3. We need to run a discovery to populate the groups… this will be a script based discovery that runs only on the RMS, queries the CMDB, matches on the servername FQDN, and populates the group with a windows computer object. (<Monitoring> section)

4. We need to modify the display strings in the XML MP – in order to show friendly display names for each of the above, in the UI. (<LanguagePacks> Section)

You can simply take the XML posted below, and just modify each section with your custom group names… or add new groups by adding a new class, relationship, discovery/script, and presentation section to each.

Here we go:

Section 1: <Manifest>

Simply modify the <ID>, <Version>, and <Name> sections based on your custom MP naming standard.

<ManagementPack ContentReadable="true" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<Manifest>
    <Identity>
      <ID>SQLBasedGroupDemo</ID>
      <Version>1.0.0.0</Version>
    </Identity>
    <Name>SQLBasedGroupDemo</Name>
    <References>
      <Reference Alias="SC">
        <ID>Microsoft.SystemCenter.Library</ID>
        <Version>6.0.6278.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
      <Reference Alias="Windows">
        <ID>Microsoft.Windows.Library</ID>
        <Version>6.0.6278.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
      <Reference Alias="Health">
        <ID>System.Health.Library</ID>
        <Version>6.0.6278.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
      <Reference Alias="System">
        <ID>System.Library</ID>
        <Version>6.0.6278.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
    </References>
</Manifest>

Section 2: <TypeDefinitions>

The example below defines 3 groups, and the relationships for those groups (contains windows computer objects). Simply replace the bolded red “Team1Group” example with a short name for your custom groups. (We will define the UI friendly name later, in the <Presentation> section.

<TypeDefinitions>
<EntityTypes>
    <ClassTypes>
      <ClassType ID="GroupPopulation.Team1Group" Accessibility="Internal" Abstract="false" Base="System!System.Group" Hosted="false" Singleton="true" />
      <ClassType ID="GroupPopulation.Team2Group" Accessibility="Internal" Abstract="false" Base="System!System.Group" Hosted="false" Singleton="true" />
      <ClassType ID="GroupPopulation.Team3Group" Accessibility="Internal" Abstract="false" Base="System!System.Group" Hosted="false" Singleton="true" />
    </ClassTypes>
    <RelationshipTypes>
      <RelationshipType ID="GroupPopulation.Team1GroupContainsWindowsComputers" Accessibility="Internal" Abstract="false" Base="System!System.Containment">
        <Source>GroupPopulation.Team1Group</Source>
        <Target>Windows!Microsoft.Windows.Computer</Target>
      </RelationshipType>
      <RelationshipType ID="GroupPopulation.Team2GroupContainsWindowsComputers" Accessibility="Internal" Abstract="false" Base="System!System.Containment">
        <Source>GroupPopulation.Team2Group</Source>
        <Target>Windows!Microsoft.Windows.Computer</Target>
      </RelationshipType>
      <RelationshipType ID="GroupPopulation.Team3GroupContainsWindowsComputers" Accessibility="Internal" Abstract="false" Base="System!System.Containment">
        <Source>GroupPopulation.Team3Group</Source>
        <Target>Windows!Microsoft.Windows.Computer</Target>
      </RelationshipType>
    </RelationshipTypes>
</EntityTypes>
</TypeDefinitions>

Section 3: <Monitoring>

In this section – we define the discovery, and add the script to run. In this example – I am running a VBscript with a SQL query to the CMDB. You will need to modify the group name – just like you did above. This is where we create a discovery to populate each group – so we will need one of these sections for each group in the MP. Each of these sections will run a distinct script, with a different query, depending on which computers you want populated.

I bolded in RED the group name sections you will need to modify, just like we did above… and you will need to modify the SQL DB information, and the script name.

I set the discovery time to every 3600 seconds in this example…. you should probably set this to once or twice a day max…. no need to keep re-running it for groups that wont change that often.

<Monitoring>
    <Discoveries>
      <Discovery ID="Team1Group.Discovery" Enabled="true" Target="SC!Microsoft.SystemCenter.RootManagementServer" ConfirmDelivery="true" Remotable="true" Priority="Normal">
        <Category>Discovery</Category>
        <DiscoveryTypes>
          <DiscoveryClass TypeID="GroupPopulation.Team1Group" />
        </DiscoveryTypes>
        <DataSource ID="DS" TypeID="Windows!Microsoft.Windows.TimedScript.DiscoveryProvider">
          <IntervalSeconds>3600</IntervalSeconds>
          <SyncTime />
          <ScriptName>Team1GroupDiscovery.vbs</ScriptName>
          <Arguments>$MPElement$ $Target/Id$</Arguments>
          <ScriptBody><![CDATA[Dim SourceId
Dim objConnection
Dim oRS
Dim sConnectString
Dim ManagedEntityID
Dim oAPI
Dim oDiscoveryData
SourceId                = WScript.Arguments(0)
ManagedEntityId         = WScript.Arguments(1)
Set oAPI                = CreateObject("MOM.ScriptAPI")
Set oDiscoveryData      = oAPI.CreateDiscoveryData(0,SourceId,ManagedEntityId)
sConnectString = "Driver={SQL Server}; Server=OMDW; Database=CMDB;"
Set objConnection = CreateObject("ADODB.Connection")
objConnection.Open sConnectString
Set oRS = CreateObject("ADODB.Recordset")
oRS.Open "select ServerName from ServerList where MonitorGroup = 'Team 1'", objConnection
Set groupInstance = oDiscoveryData.CreateClassInstance("$MPElement[Name='GroupPopulation.Team1Group']$")
While Not oRS.EOF
Set serverInstance = oDiscoveryData.CreateClassInstance("$MPElement[Name='Windows!Microsoft.Windows.Computer']$")
serverInstance.AddProperty "$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$",oRS.Fields("ServerName")
Set relationshipInstance = oDiscoveryData.CreateRelationshipInstance("$MPElement[Name='GroupPopulation.Team1GroupContainsWindowsComputers']$")
relationshipInstance.Source = groupInstance
relationshipInstance.Target = serverInstance
oDiscoveryData.AddInstance relationshipInstance
oRS.MoveNext
Wend
objConnection.Close
Call oAPI.Return(oDiscoveryData)
                     ]]></ScriptBody>
          <TimeoutSeconds>120</TimeoutSeconds>
        </DataSource>
      </Discovery>

Section 4: <LanguagePacks>

Here – we will take ALL of the modifications we made in the prior three steps, and match them up with friendly names to show in the UI:

Essentially – just modify the MP name, group name, relationship name, and discovery name, to match what you did above…. and assign each a friendly name that you want to see in the UI. I will bold in red the sections to modify, for the “Team 1 servers”. You would continue this for as many groups as you used:

<LanguagePacks>
<LanguagePack ID="ENU" IsDefault="true">
    <DisplayStrings>
      <DisplayString ElementID="SQLBasedGroupDemo">
        <Name>SQL Based Group Population Demo MP</Name>
      </DisplayString>

      <DisplayString ElementID="GroupPopulation.Team1Group">
        <Name>Team 1 Servers Group</Name>
      </DisplayString>
      <DisplayString ElementID="GroupPopulation.Team1GroupContainsWindowsComputers">
        <Name>Team 1 SQL Based Group Contains Windows Computers</Name>
      </DisplayString>
      <DisplayString ElementID="Team1Group.Discovery">
        <Name>Team 1 SQL Based Group Discovery</Name>
        <Description />
      </DisplayString>

      <DisplayString ElementID="GroupPopulation.Team2Group">
        <Name>Team 2 Servers Group</Name>
      </DisplayString>
      <DisplayString ElementID="GroupPopulation.Team2GroupContainsWindowsComputers">
        <Name>Team 2 SQL Based Group Contains Windows Computers</Name>
      </DisplayString>
      <DisplayString ElementID="Team2Group.Discovery">
        <Name>Team 2 SQL Based Group Discovery</Name>
        <Description />
      </DisplayString>

      <DisplayString ElementID="GroupPopulation.Team3Group">
        <Name>Team 3 Servers Group</Name>
      </DisplayString>
      <DisplayString ElementID="GroupPopulation.Team3GroupContainsWindowsComputers">
        <Name>Team 3 SQL Based Group Contains Windows Computers</Name>
      </DisplayString>
      <DisplayString ElementID="Team3Group.Discovery">
        <Name>Team 3 SQL Based Group Discovery</Name>
        <Description />
      </DisplayString>

</DisplayStrings>
</LanguagePack>
</LanguagePacks>

That’s it! If you get errors trying to import – you most likely modified a definition incompletely…. the import error should help you figure out what's wrong.

Now I can go to my groups – find “Team 1 Group” and see if it is populated: SUCCESS!

I am attaching my working sample MP below

Boosting OpsMgr performance - by reducing the OpsDB data retention

kevinhol — Wed, 05 Nov 2008 02:24:40 GMT

Here is a little tip I often advise my customers on.....

The default data retention in OpsMgr is 7 days for most data types:

These are default settings which work well for a large cross section of different agent counts. In MOM 2005 - we defaulted to 4 days. Many customers, especially with large agent counts, would have to reduce that in MOM 2005 down to 2 days to keep a manageable Onepoint DB size.

That being said - to boost UI performance, and reduce OpsDB database size - consider reducing these values down to your real business requirements. For a new, out of the box management group - I advise my customers to set these to 2 days. This will keep less noise in your database as you deploy, and tune, agents and management packs. This keeps a smaller DB, and a more responsive UI, in large agent count environments.

Essentially - set each value to "2" except for Performance Signature, which we will change to 1. Performance Signature is unique.... the setting here isnt actually "Days" of retention. It is "business cycles". This is for self-tuning threshold ONLY. This data is used for calculating business cycle based self-tuning thresholds. There is NO REASON for this ever to be larger than the default of "2" business cycles.... and large agent count environments can see a performance benefit by bumping this down to only keeping "1" business cycle.

Then - once your Management group is fully deployed, and you have tuned your alert, performance, event, and state data.... IF you have a business requirement to keep this data for longer - bump it up.

Keep in mind - this will NOT cause you to groom out Alerts that are open - only closed alerts, and still will keep your closed alerts around for a couple days.

These settings have no impact on the data that is being written to the data warehouse - so any alert, event, or perf data needed will always be there.

Agent Pending Actions can get out of synch between the Console, and the database

kevinhol — Mon, 29 Sep 2008 21:57:59 GMT

When you look at your agent pending actions in the Administration pane of the console.... you will see pending actions for things like approving a manual agent install, agent installation in progress, approving agent updates, like from a hotfix, etc.

This pending action information is also contained in the SQL table in the OpsDB - agentpendingaction

It is possible for the agentpendingaction table to get out of synch with the console, for instance, if the server was in the middle of updating/installing an agent - and the management server Healthservice process crashed or was killed.

In this case, you might have a lingering pending action, that blocks you from doing something in the future. For instance - if you had a pending action to install an agent, that did not show up in the pending actions view of the console. What might happen, is that when you attempt to discover and push the agent to this same server, you get an error message:

"One or more computers you are trying to manage are already in the process of being managed. Please resolve these issues via the Pending Management view in Administration, prior to attempting to manage them again"

The problem is - they don't show up in this view!

To view the database information on pending actions:

select * from agentpendingaction

You should be able to find your pending action there - that does not show up in the Pending Action view in the console, if you are affected by this.

To resolve - we should first try and reject these "ghost" pending actions via the SDK... using powershell. Open a command shell, and run the following:

get-agentpendingaction

To see a prettier view:

get-agentpendingaction | ft agentname,agentpendingactiontype

To see a specific pending action for a specific agent:

get-agentPendingAction | where {$_.AgentName -eq "servername.domain.com"}

To reject the specific pending action:

get-agentPendingAction | where {$_.AgentName -eq "servername.domain.com"}|Reject-agentPendingAction

We can use the last line - to reject the specific pending action we are interested in.

You might get an exception running this:

Reject-AgentPendingAction : Microsoft.EnterpriseManagement.Common.UnknownServiceE
xception: The service threw an unknown exception. See inner exception for details
. ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:
Exception of type 'Microsoft.EnterpriseManagement.Common.DataItemDoesNotExistExc
eption' was thrown.

If this fails, such as gives an exception, or if our problem pending action doesn't even show up in Powershell.... we have to drop down to the SQL database level. This is a LAST resort and NOT SUPPORTED.... run at your own risk.

There is a stored procedure to delete pending actions.... here is an example, to run in a SQL query window:

exec p_AgentPendingActionDeleteByAgentName 'agentname.domain.com'

Change 'agentname.domain.com' to the agent name that is showing up in the SQL table, but not in the console view.

DBcreatewizard or just run good old SetupOM.exe - which should I use to install the Database component of OpsMgr?

kevinhol — Sat, 03 May 2008 14:04:50 GMT

There has always been a bit of confusion on when to run the DBCreateWizard.exe tool, or when to just use SetupOM.exe to create the Operational DB or Data Warehouse DB.

Historically.... in MOM 2005, we used the DBcreate Wizard in order to create the Onepoint database on Active/Active clusters..... or when SQL DBA teams refused to run a MSI based setup on one of their SQL servers. The DB create wizard was a better option for them.... since it did not have to install any binaries on a SQL server. In practice.... it was pretty rare to see this in widespread use.

In OpsMgr 2007, we haven't really documented all the scenarios for when you should run the DBcreate Wizard.... and I will try and do that here.

The DB create wizard is located on the CD - In the \SupportTools folder. It does require some additional files to run it - these don't have to be "installed", just need to be copied over to the SQL DB server where you will run the wizard. Follow: http://support.microsoft.com/kb/938997/en-us

*** Note - the additional files required to run DBCreateWizard.exe are documented in the KB article above. They were also provided on the SP1 Select CD. However - the files provided on CD are for 32bit x86 only. If you are using the DBCreateWizard on a x64 platform - you MUST copy these files listed in the KB article from an x64 server.... any x64 server with the console installed will have them.

Note - there were some significant issues with the RTM version of this tool... in detecting the correct SQL instance on a multi-instance cluster, and leaving some table information blank (http://support.microsoft.com/kb/942865/en-us). When deploying SP1 - Use the SP1 version of this tool. If you MUST deploy the RTM version - I would recommend using SetupOM.exe for all installs.

Ok.... first, you will notice in the OpsMgr Deployment guide, they instruct to use the DBcreateWizard when installing the database on an Active/Passive cluster. That's pretty much our first introduction to this tool. While this isn't required (you can simply run SetupOM.exe on the Active node) it is recommended to use DBCreateWizard. Essentially, our recommendation is that anytime you have a dedicated SQL server for the OpsDB role... with no other OpsMgr role present, then you should use the DBcreateWizard to create the Operational database. The reason for this, from an internal discussion I have been involved in.... is because using SetupOM.exe will create some additional registry entries on the database server... and will change how updates are applied to the server from an OpsMgr perspective. Another scenario to leverage this tool, is anytime your SQL DBA teams refuse to allow you to run a MSI based setup on their SQL servers/clusters.

Below, I will just walk through some of the scenarios where using this stand-alone tool really makes good sense.

Scenarios:

1. All in one role/shared roles. This is where a single server hosts SQL Server 2005 and the Operational Database role, along with the RMS role. In this case.... you might as well just run SetupOM.exe and create the database while installing the management group. You potentially could run the DBcreatewizard first.... but this would be an additional step and provides no value.

2. Split roles: Dedicated SQL server (Server A) and dedicated RMS (Server B). In this scenario - we recommend using DBcreatewizard.exe instead of just running SetupOM.exe on the SQL server. However - you certainly can do either one.... both are fully supported.

3. Split roles - clustered DB: Dedicated cluster for SQL (can be A/P or A/A or multi-instance or multi node.... doesn't matter) In this scenario - we recommend using DBcreatewizard.exe instead of just running SetupOM.exe on the SQL server. That said.... you can run SetupOM.exe on any node that owns the SQL instance you are creating the DB in.... we just favor using DBcreateWizard.

4. Draconian DBA's. In general.... DBA's are used to creating an empty database for an application, then granting permissions to the DB only.... then washing their hands of it. They don't like running setup's... or even running tools on their SQL servers.... If they must have an application create a database as part of that application install - they MUCH prefer that all the DB creation be handled remotely. Unfortunately.... MOM 2005 and OpsMgr 2007 do not support what DBA's would most like to see. We must run our setup or tool on the database server/node in order to install that component. I suppose we could install the OpsDB using the DBcreatewizard in a test lab SQL box, then detach it.... then hand the files to a SQL team and have them drop in into a production environment to make them happier.... but I haven't really done much testing there. Anyway.... the DBcreateWizard is the best option when working with a rigid DBA team. Just follow the KB article listed above... and have the SQL team run the tool to create the database.... then they can delete to tool from the server. We will still require SA priv over the instance to complete the RMS setup.... but once that is done, they can remove these advanced rights, per my previous post: http://blogs.technet.com/kevinholman/archive/2008/04/15/opsmgr-security-account-rights-mapping-what-accounts-need-what-privileges.aspx

5. Multiple Operational Databases in the same SQL instance. It is possible, if you have multiple management groups, that you could place all the Operational DB's into a single SQL instance. Now - these had better be small environments (test/dev) or a beefy SQL server to handle all that I/O.... but just for grins.... lets say you are doing it. If you tried to run SetupOM.exe and install the database component multiple times.... it would detect it was already installed and ask you if you wish to repair or remove OpsMgr. No good. In comes the DBcreateWizard. This tool is the supported method for creating multiple OpsDB's in a single SQL instance.

Event ID 2115 A Bind Data Source in Management Group

kevinhol — Tue, 22 Apr 2008 01:12:00 GMT

I see this event a lot in customer environments. I am not an expert on troubleshooting this here... but saw this post in the MS newsgroups and felt it was worth capturing....

My experience has been that it is MUCH more common to see these when there is a management pack that collects way too much discovery data.... than any real performance problem with the data warehouse. In most cases.... if the issue just started after bringing in a new MP.... deleting that MP solves the problem. I have seen this repeatedly after importing the Cluster MP, Or Exchange 2007 MP.... but haven't been able to fully investigate the root cause yet:

In a nutshell.... if they are happening just a couple times an hour.... and the time in seconds is fairly low (under a few minutes) then this is normal.

If they are happening very frequently - like every minute, and the times are increasing - then there is an issue that needs to be resolved.

Taken from the newsgroups:

-------------------------------------------

In OpsMgr 2007 one of the performance concerns is DB/DW data insertion performance. Here is a description of how to identify and troubleshoot problems with DB/DW data insertion.

Symptoms:

DB/DW write action workflows run on a Management Server, they first keep data received from Agent / Gateway in an internal buffer, then they create a batch of data from the buffer and insert the data batch to DB / DW, when the insertion of the first batch finished, they will create another batch and insert it to DB / DW. The size of the batch depends on how much data is available in the buffer when the batch is created, but there is a maximum limit on the size of the batch, a batch can contain up to 5000 data items. If data item incoming (from Agent / Gateway) throughput becomes larger, or the data item insertion (to DB/DW) throughput becomes smaller, then the buffer will tend to accumulate more data and the batch size will tend to become larger. There are different write action workflows running on a MS, they handle data insertion to DB / DW for different type of data:

Microsoft.SystemCenter.DataWarehouse.CollectEntityHealthStateChange
Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Microsoft.SystemCenter.DataWarehouse.CollectEventData
Microsoft.SystemCenter.CollectAlerts
Microsoft.SystemCenter.CollectEntityState
Microsoft.SystemCenter.CollectPublishedEntityState
Microsoft.SystemCenter.CollectDiscoveryData
Microsoft.SystemCenter.CollectSignatureData
Microsoft.SystemCenter.CollectEventData

When a DB/DW write action workflow on Management Server notices that the insertion of a single data batch is slow (ie. slower than 1 minute), it will start to log a 2115 NT event to OpsMgr NT event log once every minute until the batch is inserted to DB / DW or is dropped by DB / DW write action module. So you will see 2115 events in management server's "Operations Manager" NT event log when it is slow to insert data to DB /DW. You might also see 2115 events when there is a burst of data items coming to
Management server and the number of data items in a batch is large. (This can happen during a large amount of discovery data being inserted - from a freshly imported or noisy management pack.)

2115 events have 2 import pieces of information: the name of the workflow that has insertion problem, and the pending time since the workflow started inserting last data batch. Here is an example of a 2115 event:

------------------------------------

A Bind Data Source in Management Group OpsMgr07PREMT01 has posted items to the workflow, but has not received a response in 3600 seconds. This indicates a performance or functional problem with the workflow.

Workflow Id : Microsoft.SystemCenter.CollectSignatureData

Instance : MOMPREMSMT02.redmond.corp.microsoft.com

Instance Id : {6D52A6BB-9535-9136-0EF2-128511F264C4}

------------------------------------------

This 2115 event is saying DB write action workflow "Microsoft.SystemCenter.CollectSignatureData" (which writes performance
signature data to DB) is trying to insert a batch of signature data to DB and it started inserting 3600 seconds ago but the insertion has not finished yet. Normally inserting of a batch should finish within 1 minutes.

Normally, there should not be much 2115 events happening on Management server, if it happens less than 1 or 2 times every hour (per write action workflow), then it is not a big concern, but if it happens more than that, there is a DB /DW insertion problem.

The following performance counters on Management Server gives information of DB / DW write action insertion batch size and insertion time, if batch size is becoming larger (by default maximum batch size is 5000), it means management server is either slow in inserting data to DB/DW or is getting a burst of data items from Agent/Gateway. From the DB / DW write action's Avg. Processing Time, you will see how much time it takes to write a batch of data to DB / DW.

OpsMgr DB Write Action Modules(*)\Avg. Batch Size
OpsMgr DB Write Action Modules(*)\Avg. Processing Time
OpsMgr DW Writer Module(*)\Avg. Batch Processing Time, ms
OpsMgr DW Writer Module(*)\Avg. Batch Size

Possible root causes:

In OpsMgr, discovery data insertion is relatively expensive, so a discovery burst (a discovery burst is referring to a short period of time when a lot of discovery data is received by management server) could cause 2115 event (complaining about slow insertion of discovery data), since discovery insertion should not happen frequently. So if you see consistently 2115 events for discovery data collection. That means you either have DB /DW insertion problem or some discovery rules in a MP is collecting too much
discovery data.
OpsMgr Config update caused by instance space change or MP import will impact the CPU utilization on DB and will have impact on DB data insertion. After importing a new MP or after a big instance space change in a large environment, you will probably see more than normal 2115 events.
Expensive UI queries can impact the resource utilization on DB and could have impact on DB data insertion. When user is doing expensive UI operation, you will probably see more than normal 2115 events.
When DB / DW is out of space / offline you will find Management server keeps logging 2115 events to NT event log and the pending time is becoming higher and higher.
Sometimes invalid data item sent from agent /Gateway will cause DB / DW insertion error which will end up with 2115 event complaining about DB /DW slow insertion. In this case please check the OpsMgr event log for relevant error events. It's more common in DW write action workflows.
If DB / DW hardware is not configured properly, there could be performance issue, and it could cause slow data insertion to DB / DW. The problem could be:
- The network link between DB / DW to MS is slow (either bandwidth is small / latency is large, as a best practice we recommend MS to be in the same LAN as DB/DW).
- The data / log / tempdb disk used by DB / DW is slow, (we recommend separating data, log and tempdb to different disks, we recommend using RAID 10 instead of using RAID 5, we also recommend turning on write cache of the array controllers).
- The OpsDB tables are too fragmented (this is a common cause of DB performance issues). Reindex affected tables will solve this issue.
- The DB / DW does not have enough memory.

Now - that is the GENERAL synopsis and how to attack them. Next - we will cover a specific issue we are seeing with a specific type of 2115 Event:

-----------------------------------------------

It appears we may be hitting cache resolution error we were trying to catch for a while. This is about CollectEventData workflow. Error is very hard to catch and we're including a fix in SP2 to avoid it. There are two ways to resolve the problem in the meantime. Since the error happens very rarely, you can just restart Health Service on the Management Server that is affected. Or you can prevent it from blocking the workflow by creating overrides in the following way:

-----------------------------------------------

1) Launch Console, switch to Authoring space and click "Rules"
2) In the right top hand side of the screen click "Change Scope"
3) Select "Data Warehouse Connection Server" in the list of types,. click "Ok"
4) Find "Event data collector" rule in the list of rules;
5) Right click "Event data collector" rule, select Overrides/Override the Rule/For all objects of type...
6) Set Max Execution Attempt Count to 10
7) Set Execution Attempt Timeout Interval Seconds to 6

That way if DW event writer fails to process event batch for ~ a minute, it will discard the batch. 2115 events related to
Datawarehouse.CollectEventData should go away after you apply these overrides. BTW, while you're at it you may want to override "Max Batches To Process Before Maintenance Count" to 50 if you have a relatively large environment. We think 50 is better default setting then SP1's 20 in this case and we'll switch default to 50 in SP2.

-------------------------------------------------

Essentially - to know if you are affected by the specific 2115 issue describe above - here is the criteria:

1. You are seeing 2115 bind events in the OpsMgr event log of the RMS or MS, and they are recurring every minute.

2. The events have a Workflow ID of: Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData

3. The "has not received a response" time is increasing, and growing to be a very large number over time.

Here is an example of a MS with the problem: Note consecutive events, from the CollectEventData workflow, occurring every minute, with the time being a large number and increasing:

Event Type:      Warning
Event Source:   HealthService
Event Category:            None
Event ID:          2115
Date:                5/5/2008
Time:                2:37:06 PM
User:                N/A
Computer:         MS1
Description:
A Bind Data Source in Management Group MG1 has posted items to the workflow, but has not received a response in 706594 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance    : MS1.domain.com
Instance Id : {646486D0-E366-03CA-38E7-79A0D6F34F82}

Event Type:      Warning
Event Source:   HealthService
Event Category:            None
Event ID:          2115
Date:                5/5/2008
Time:                2:36:05 PM
User:                N/A
Computer:         MS1
Description:
A Bind Data Source in Management Group MG1 has posted items to the workflow, but has not received a response in 706533 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance    : MS1.domain.com
Instance Id : {646486D0-E366-03CA-38E7-79A0D6F34F82}

Event Type:      Warning
Event Source:   HealthService
Event Category:            None
Event ID:          2115
Date:                5/5/2008
Time:                2:35:03 PM
User:                N/A
Computer:         MS1
Description:
A Bind Data Source in Management Group MG1 has posted items to the workflow, but has not received a response in 706471 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance    : MS1.domain.com
Instance Id : {646486D0-E366-03CA-38E7-79A0D6F34F82}

What SQL maintenance should I perform on my OpsMgr databases?

kevinhol — Sat, 12 Apr 2008 02:38:00 GMT

This question comes up a lot. The answer is really - not what maintenance you should be performing... but what maintenance you should be *excluding*.... or when. Here is why:

Most SQL DBA's will set up some pretty basic default maintenance on all SQL DB's they support. This often includes, but is not limited to:

DBCC CHECKDB (to look for DB errors and report on them)

UPDATE STATISTICS (to boost query performance)

DBCC DBREINDEX (to rebuild the table indexes to boost performance)

BACKUP

SQL DBA's might schedule these to run via the SQL Agent to execute nightly, weekly, or some combination of the above depending on DB size and requirements.

On the other side of the coin.... in some companies, the MOM/OpsMgr team installs and owns the SQL server.... and they dont do ANY default maintenance to SQL. Because of this - a focus in OpsMgr was to have the Ops DB and Datawarehouse DB to be fully self-maintaining.... providing a good level of SQL performance whether or not any default maintenance was being done.

Operational Database:

Reindexing is already taking place against the OperationsManager database for some of the tables. This is built into the product. What we need to ensure - is that any default DBA maintenance tasks are not redundant nor conflicting with our built-in maintenance, and our built-in schedules:

There is a rule in OpsMgr that is targeted at the Root Management Server:

The rule executes the "p_OptimizeIndexes" stored procedure, every day at 2:30AM:

This rule cannot be changed or modified. Therefore - we need to ensure there is not other SQL maintenance (including backups) running at 2:30AM, or performance will be impacted.

If you want to view the built-in UPDATE STATISTICS and DBCC DBREINDEX jobs history - just run the following queries:

select *
from DomainTable dt
inner join DomainTableIndexOptimizationHistory dti
on dt.domaintablerowID = dti.domaintableindexrowID
ORDER BY optimizationdurationseconds DESC

select *
from DomainTable dt
inner join DomainTableStatisticsUpdateHistory dti
on dt.domaintablerowID = dti.domaintablerowID
ORDER BY UpdateDurationSeconds DESC

Take note of the update/optimization duration seconds column. This will show you how long your maintenance is typically running. In a healthy environment these should not take very long.

If you want to view the fragmentation levels of the current tables in the database, run:

DBCC SHOWCONTIG WITH FAST

Here is some sample output:

----------------------------------------------------------------------------------------------

DBCC SHOWCONTIG scanning 'Alert' table...
Table: 'Alert' (1771153355); index ID: 1, database ID: 5
TABLE level scan performed.
- Pages Scanned................................: 936
- Extent Switches..............................: 427
- Scan Density [Best Count:Actual Count].......: 27.34% [117:428]
- Logical Scan Fragmentation ..................: 60.90%

----------------------------------------------------------------------------------------------

In general - we would like the "Scan density" to be high (Above 80%), and the "Logical Scan Fragmentation" to be low (below 30%). What you might find... is that *some* of the tables are more fragmented than others, because our built-in maintenance does not reindex all tables. Especially tables like the raw perf, event, and localizedtext tables.

That said - there is nothing wrong with running a DBA's default maintenance against the Operational database..... reindexing these tables in the database might also help console performance. We just dont want to run any DBA maintenance during the same time that we run our own internal maintenance, so try not to conflict with this schedule. Care should also be taken in any default DBA maintenance, that it does not run too long, or impact normal operations of OpsMgr. Maintenance jobs should be monitored, and should not conflict with the backup schedules as well.

Here is a reindex job you can schedule with SQL agent.... for the OpsDB:

USE OperationsManager
go
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
SET ARITHABORT ON
SET CONCAT_NULL_YIELDS_NULL ON
SET QUOTED_IDENTIFIER ON
SET NUMERIC_ROUNDABORT OFF
EXEC SP_MSForEachTable "Print 'Reindexing '+'?' DBCC DBREINDEX ('?')"

Data Warehouse Database:

The data warehouse DB is also fully self maintaining. This is called out by a rule "Standard Data Warehouse Data Set maintenance rule" which is targeted to the "Standard Data Set" object type. This stored procedure is called on the data warehouse every 60 seconds. It performs many, many tasks, of which Index optimization is but one.

This SP calls the StandardDatasetOptimize stored procedure, which handles any index operations.

To examine the index and statistics history - run the following query for the Alert, Event, Perf, and State tables:

select basetablename, optimizationstartdatetime, optimizationdurationseconds,
beforeavgfragmentationinpercent, afteravgfragmentationinpercent,
optimizationmethod, onlinerebuildlastperformeddatetime
from StandardDatasetOptimizationHistory sdoh
inner join StandardDatasetAggregationStorageIndex sdasi
on sdoh.StandardDatasetAggregationStorageIndexRowId = sdasi.StandardDatasetAggregationStorageIndexRowId
inner join StandardDatasetAggregationStorage sdas
on sdasi.StandardDatasetAggregationStorageRowId = sdas.StandardDatasetAggregationStorageRowId
ORDER BY optimizationdurationseconds DESC

Then examine the default domain tables optimization history.... run the same two queries as listed above for the OperationsDB.

In the data warehouse - we can see that all the necessary tables are being updated and reindexed as needed. When a table is 10% fragmented - we reorganize. When it is 30% or more, we rebuild the index.

Therefore - there is no need for a DBA to execute any UPDATE STATISTICS or DBCC DBREINDEX maintenance against this database. Furthermore, since we run our maintenance every 60 seconds, and only execute maintenance when necessary, there is no "set window" where we will run our maintenance jobs. This means that if a DBA team also sets up a UPDATE STATISTICS or DBCC DBREINDEX job - it can conflict with our jobs and execute concurrently. This should not be performed.

For the above reasons, I would recommend against any maintenance jobs on the Data Warehouse DB, beyond a CHECKDB (only if DBA's mandate it) and a good backup schedule.

For the OpsDB: any standard maintenance is fine, as long as it does not conflict with the built-in maintenance, or impact production by taking too long, or having an impact on I/O.

Lastly - I'd like to discuss the recovery model of the SQL database. We default to "simple" for all our DB's. This should be left alone.... unless you have *very* specific reasons to change this. Some SQL teams automatically assume all databases should be set to "full" recovery model. This requires that they back up the transaction logs on a very regular basis, but give the added advantage of restoring up to the time of the last t-log backup. For OpsMgr, this is of very little value, as the data changing on an hourly basis is of little value compared to the complexity added by moving from simple to full. Also, changing to full will mean that your transaction logs will only checkpoint once a t-log backup is performed. What I have seen, is that many companies aren't prepared for the amount of data written to these databases.... and their standard transaction log backups (often hourly) are not frequent enough to keep them from filling. The only valid reason to change to FULL, in my opinion, is when you are using an advanced replication strategy, like log shipping, which requires full recovery model. When in doubt - keep it simple. :-)

P.S.... The Operations Database needs 50% free space at all times. This is for growth, and for re-index operations to be successful. This is a general supportability recommendation, but the OpsDB will alert when this falls below 40%.

For the Data warehouse.... we do not require the same 50% free space. This would be a temendous requireemnts if we had a multiple-terabyte database!

Think of the data warehouse to have 2 stages... a "growth" stage (while it is adding data and not yet grooming much (havent hit the default 400 days retention) and a "maturity stage" where agent count is steady, MP's are not changing, and the grooming is happening because we are at 400 days retention. During "growth" we need to watch and maintain free space, and monitor for available disk space. In "maturity" we only need enough free space to handle our index operations. when you start talking 1 Terabyte of data.... that means 500GB of free space, which is expensive, and. If you cannot allocate it.... then just allow auto-grow and monitor the database.... but always plan for it from a volume size perspective.

For transaction log sizing - we don't have any hard rules. A good rule of thumb for the OpsDB is ~20% to 50% of the database size.... this all depends on your environment. For the Data warehouse, it depends on how large the warehouse is - but you will probably find steady state to require somewhere around 10% to 20% of the warehouse size. Any time we are doing any additional grooming of an alert/event/perf storm.... or changin grooming from 400 days to 300 days - this will require a LOT more transaction log space - so keep that in mind as your databases grow.

Grooming process in the Operations Database

kevinhol — Wed, 13 Feb 2008 05:51:47 GMT

This is a continuation of my other post, on general alert grooming:

How grooming and auto-resolution work in the OpsMgr 2007 Operational database

Grooming of the OpsDB is called once per day at 12:00am…. by the rule: “Partitioning and Grooming” You can search for this rule in the Authoring space of the console, under Rules. It is targeted to the “Root Management Server” and is part of the System Center Internal Library.

It calls the “p_PartitioningAndGrooming” stored procedure. This SP calls two other SP's: p_Partitioning and then p_Grooming

p_Partitioning inspects the table PartitionAndGroomingSettings, and then calls the SP p_PartitionObject for each object in the PartitionAndGroomingSettings table where "IsPartitioned = 1" (note - we partition event and perf into 61 daily tables - just like MOM 2005)

The PartitionAndGroomingSettings table:

The p_PartitionObject SP first identifies the next partition in the sequence, truncates it to make sure it is empty, and then updates the PartitionTables table in the database, to update the IsCurrent field to the next numeric table for events and perf. Then it calls the p_PartitionAlterInsertView sproc, to make new data start writing to the current event and perf table.

To review which tables you are writing to - execute the following query: select * from partitiontables where IsCurrent = '1'

A select * from partitiontables will show you all 61 event and perf tables, and when they were used. You should see a PartitionStartTime updated every day - around midnight (time is stored in UTC in the database). If partitioning is failing to run, then we wont see this date changing every day.

Ok - that's the first step of the p_PartitioningAndGrooming sproc - Partitioning. Now - if that is all successful, we will start grooming!

The p_Grooming is called after partitioning is successful. One of the first things it does - is to update the InternalJobHistory table. In this able - we keep a record of all partitioning and grooming jobs. It is a good spot check to see what's going on with grooming. To have a peek at this table - execute a select * from InternalJobHistory order by InternalJobHistoryId

The p_Grooming sproc then calls p_GroomPartitionedObjects This sproc will first examine the PartitionAndGroomingSettings and compare the days to keep column, against the current date, to figure out how many partitions to groom. It will then inspect the partitions to ensure they have data, and then truncate the partition, by calling p_PartitionTruncate. The p_GroomPartitionedObjects sproc will then update the PartitionAndGroomingSettings table with the current time, under the GroomingRunTime column.

Next - the p_Grooming sproc continues, by calling p_GroomNonPartitionedObjects. p_GroomNonPartitionedObjects is a short, but complex sproc - in that is calls all the individual sprocs listed in the PartitionAndGroomingSettings table where IsPartitioned = 0. (see my other post at the link above to follow the logic of one of these non-partitioned sprocs)

Next - the p_Grooming sproc continues, by updating the InternalJobHistory table, to give it a status of success (StatusCode of 1 = success, 2= failed, 0 appears to be never completed?)

If you ever have a problem with grooming - or need to get your OpsDB database size under control - simply reduce the data retention days, in the console, under Administration, Settings, Database Grooming. To start with - I recommend setting all these to just 2 days, fromt he default of 7. This keeps your OpsDB under control until you have time to tune all the noise fromt he MP's you import. So just reduce this number, then open up query analyzer, and execute p_PartitioningAndGrooming When it is done, check the job status by executing select * from InternalJobHistory order by InternalJobHistoryId The last groom job should be present, and successful. The OpsDB size should be smaller, with more free space. And to validate, you can always run my large table query, found at: Useful Operations Manager 2007 SQL queries

Print Server management pack fills the Operational DB with TONS of perf data

kevinhol — Tue, 12 Feb 2008 03:19:51 GMT

This is something I have noticed in MOM 2005, and seems to be the same in the conversion MP for OpsMgr 2007. (Version 6.0.5000.0 of the Microsoft.Windows.Server.Printserver (Converted) MP). When you import this MP, it will fill the Operational and reporting databases with performance data about print jobs and queues, if you have a large number of print servers/queues in your environment.

If reporting on this perf data is not critical to your environment, you should disable these rules:

Failed tasks aren't groomed from the Operational Database

kevinhol — Tue, 05 Feb 2008 20:02:59 GMT

This appears to be present up to RC-SP1 version, build 6.0.6246.0

In the Task Status console view - I noticed an old failed task from 2 months ago..... however, my task grooming is set to 7 days.

To view the grooming process:

http://blogs.technet.com/kevinholman/archive/2007/12/13/how-grooming-and-auto-resolution-work-in-the-opsmgr-2007-operational-database.aspx

Basically – select * from PartitionAndGroomingSettings will show you all grooming going on.

Tasks are kept in the jobstatus table.

Select * from jobstatus will show all tasks.

p_jobstatusgrooming is called to groom this table.

Here is the text of that SP:

--------------------------------

USE [OperationsManager]

/****** Object: StoredProcedure [dbo].[p_JobStatusGrooming] Script Date: 02/05/2008 10:49:32 ******/

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

ALTER PROCEDURE [dbo].[p_JobStatusGrooming]

BEGIN

SET NOCOUNT ON

DECLARE @Err int

DECLARE @Ret int

DECLARE @RowCount int

DECLARE @SaveTranCount int

DECLARE @GroomingThresholdLocal datetime

DECLARE @GroomingThresholdUTC datetime

DECLARE @TimeGroomingRan datetime

DECLARE @MaxTimeGroomed datetime

SET @SaveTranCount = @@TRANCOUNT

SET @TimeGroomingRan = getutcdate()

SELECT @GroomingThresholdLocal = dbo.fn_GroomingThreshold(DaysToKeep, getdate())

FROM dbo.PartitionAndGroomingSettings

WHERE ObjectName = 'JobStatus'

EXEC dbo.p_ConvertLocalTimeToUTC @GroomingThresholdLocal, @GroomingThresholdUTC OUT

IF (@@ERROR <> 0)

BEGIN

GOTO Error_Exit

END

-- Selecting the max time to be groomed to update the table

SELECT @MaxTimeGroomed = MAX(LastModified)

FROM dbo.JobStatus

WHERE TimeFinished IS NOT NULL

AND LastModified < @GroomingThresholdUTC

IF @MaxTimeGroomed IS NULL

GOTO Success_Exit

BEGIN TRAN

-- Change the Statement below to reflect the new item

-- that needs to be groomed

DELETE FROM dbo.JobStatus

WHERE TimeFinished IS NOT NULL

AND LastModified < @GroomingThresholdUTC

SET @Err = @@ERROR

IF (@Err <> 0)

BEGIN

GOTO Error_Exit

END

UPDATE dbo.PartitionAndGroomingSettings

SET GroomingRunTime = @TimeGroomingRan,

DataGroomedMaxTime = @MaxTimeGroomed

WHERE ObjectName = 'JobStatus'

SELECT @Err = @@ERROR, @RowCount = @@ROWCOUNT

IF (@Err <> 0 OR @RowCount <> 1)

BEGIN

GOTO Error_Exit

END

COMMIT TRAN

Success_Exit:

RETURN 0

Error_Exit:

-- If there was an error and there is a transaction

-- pending, rollback.

IF (@@TRANCOUNT > @SaveTranCount)

ROLLBACK TRAN

RETURN 1

END

------------------------------------

Here is the problem in the SP:

DELETE FROM dbo.JobStatus

WHERE TimeFinished IS NOT NULL

AND LastModified < @GroomingThresholdUTC

We only delete (groom) tasks that have a timestamp in TimeFinished. If a failed task doesn’t finish – this field will be NULL and never gets groomed.

Moving the Data Warehouse and ACS databases in OpsMgr 2007

kevinhol — Fri, 14 Dec 2007 19:12:00 GMT

Moving the Data Warehouse and ACS databases in OpsMgr 2007

Instructions for how to move the data warehouse database, and ACS database – have been published.

The Operations Manager 2007 Backup and Recovery Guide has been updated as of December 2007, to include moving the DW and ACS DB's, in addition to the previously published move of the OpsDB.

Perhaps you deployed your databases to interim hardware? Perhaps you outgrew you current disk I/O? Perhaps you want to move to a high availability cluster?

Move ‘em!

http://technet.microsoft.com/en-us/opsmgr/bb498235.aspx

How grooming and auto-resolution work in the OpsMgr 2007 Operational database

kevinhol — Fri, 14 Dec 2007 02:22:00 GMT

How Grooming and Auto-Resolution works in the OpsMgr 2007 Operations DB

Warning – don’t read this if you are bored easily.

In a simplified view to groom alerts…..

Grooming of the ops DB is called once per day at 12:00am…. by the rule: “Partitioning and Grooming” You can search for this rule in the Authoring space of the console, under Rules. It is targeted to the “Root Management Server” and is part of the System Center Internal Library.

It calls the “p_PartitioningAndGrooming” stored procedure, which calls p_Grooming, which calls p_GroomNonPartitionedObjects (Alerts are not partitioned) which inspects the PartitionAndGroomingSettings table… and executes each stored procedure. The Alerts stored procedure in that table is referenced as p_AlertGrooming which has the following sql statement:

SELECT AlertId INTO #AlertsToGroom

FROM dbo.Alert

WHERE TimeResolved IS NOT NULL

AND TimeResolved < @GroomingThresholdUTC

AND ResolutionState = 255

So…. the criteria for what is groomed is pretty simple: In a resolution state of “Closed” (255) and older than the 7 day default setting (or your custom setting referenced in the table above)

We won’t groom any alerts that are in New (0), or any custom resolution-states (custom ID #). Those will have to be set to “Closed” (255)…. either by autoresolution of a monitor returning to healthy, direct user interaction, our built in autoresolution mechanism, or your own custom script.

Ok – that covers grooming.

However – I can see that brings up the question – how does auto-resolution work?

That specifically states “alerts in the new resolution state”. I don’t think that is completely correct:

That is called upon by the rule “Alert Auto Resolve Execute All” which runs p_AlertAutoResolveExecuteAll once per day at 4:00am. This calls p_AlertAutoResolve twice…. once with a variable of “0” and once with a “1”.

Here is the sql statement:

IF (@AutoResolveType = 0)

BEGIN

SELECT @AlertResolvePeriodInDays = [SettingValue]

FROM dbo.[GlobalSettings]

WHERE [ManagedTypePropertyId] = dbo.fn_ManagedTypePropertyId_MicrosoftSystemCenterManagementGroup_HealthyAlertAutoResolvePeriod()

SET @AutoResolveThreshold = DATEADD(dd, -@AlertResolvePeriodInDays, getutcdate())

SET @RootMonitorId = dbo.fn_ManagedTypeId_SystemHealthEntityState()

-- We will resolve all alerts that have green state and are un-resolved

-- and haven't been modified for N number of days.

INSERT INTO @AlertsToBeResolved

SELECT A.[AlertId]

FROM dbo.[Alert] A

JOIN dbo.[State] S

ON A.[BaseManagedEntityId] = S.[BaseManagedEntityId] AND S.[MonitorId] = @RootMonitorId

WHERE A.[LastModified] < @AutoResolveThreshold

AND A.[ResolutionState] <> 255

AND S.[HealthState] = 1

<snip>

ELSE IF (@AutoResolveType = 1)

BEGIN

SELECT @AlertResolvePeriodInDays = [SettingValue]

FROM dbo.[GlobalSettings]

WHERE [ManagedTypePropertyId] = dbo.fn_ManagedTypePropertyId_MicrosoftSystemCenterManagementGroup_AlertAutoResolvePeriod()

SET @AutoResolveThreshold = DATEADD(dd, -@AlertResolvePeriodInDays, getutcdate())

-- We will resolve all alerts that are un-resolved

-- and haven't been modified for N number of days.

INSERT INTO @AlertsToBeResolved

SELECT A.[AlertId]

FROM dbo.[Alert] A

WHERE A.[LastModified] < @AutoResolveThreshold

AND ResolutionState <> 255

So we are basically checking that Resolution state <> 255….. not specifically “New” (0) as we would lead you to believe by the wording in the interface. There are simply two types of auto-resolution: Resolve all alerts where the object has returned to a healthy state in “N” days….. and Resolve all alerts no matter what, as long as they haven’t been modified in “N” days.

Useful MOM 2005 SQL queries

kevinhol — Thu, 18 Oct 2007 21:44:00 GMT

SCDW/DTS:

Primary 6 tables that SCDW data is retained in:

SC_AlertFact_Table
SC_AlertHistoryFact_Table
SC_AlertToEventFact_Table
SC_EventFact_Table
SC_EventParameterFact_Table
SC_SampledNumericDataFact_Table

Commands to execute to modify the data retention age: Please run these commands changing the values from 385 to the number to the number of days of data to be retained:

Exec p_updategroomdays 'SC_AlertFact_Table', 385
Exec p_updategroomdays 'SC_AlertHistoryFact_Table', 385
Exec p_updategroomdays 'SC_AlertToEventFact_Table', 385
Exec p_updategroomdays 'SC_EventFact_Table', 385
Exec p_updategroomdays 'SC_EventParameterFact_Table', 385
Exec p_updategroomdays 'SC_SampledNumericDataFact_Table', 385

Queries to verify above setting:

For example, to verify the number of days that the SC_AlertFact_Table retains data, use the following command:

select cs.cs_tablename 'Table Name', wcs.wcs_groomdays 'Groom Days' from warehouseclassschema wcs
join classschemas cs
on cs.cs_classID = wcs.wcs_classID
where cs.cs_tablename = 'SC_AlertFact_Table'
and wcs.wcs_mustbegroomed = 1

Grooming the SCDW:
If you set the grooming data to attempt to groom too much data at a time, the job will likely fail due to the transaction log filling up. You should first ensure your transaction log has plenty of free space, or grow it manually, or set it to auto-grow (if ample disk space). Ensure your tempdb has ample room (disk space) to grow temporarily, and then attempt to run the SCDWgroomjob. Start with only grooming out 10 days at a time.

Queries to find out how old present data is:

SELECT DATEDIFF(d, MIN(DateTimeLastModified), GETDATE()) AS [Current] FROM SC_AlertFact_View
SELECT DATEDIFF(d, MIN(DateTimeLastModified), GETDATE()) AS [Current] FROM SC_AlertHistoryFact_View
SELECT DATEDIFF(d, MIN(DateTimeEventStored), GETDATE()) AS [Current] FROM SC_AlertToEventFact_View
SELECT DATEDIFF(d, MIN(DateTimeStored), GETDATE()) AS [Current] FROM SC_EventFact_View
SELECT DATEDIFF(d, MIN(DateTimeEventStored), GETDATE()) AS [Current] FROM SC_EventParameterFact_View
SELECT DATEDIFF(d, MIN(DateTimeAdded), GETDATE()) AS [Current] FROM SC_SampledNumericDataFact_View

Delete event data from SCDW (NOT SUPPORTED):

USE SystemCenterReporting
DELETE SC_EventFact_Table
FROM SC_EventFact_Table AS ft
INNER JOIN SC_EventDetailDimension_Table AS ed
ON ft.EventDetail_FK = ed.SMC_InstanceID
WHERE ed.EventID_PK = '1234'

Query to examine the last groom job and how long it took in the SCDW:

Use SystemCenterReporting
GO
Select MWCS.GroomDays, MCS.ClassName, MCS.TableName,MWGI.*, DATEDIFF(s,MWGI.StartTime, MWGI.EndTime) AS GroomTookSeconds FROM dbo.SMC_Meta_WarehouseGroomingInfo MWGI
INNER JOIN SMC_Meta_ClassSchemas MCS
ON MWGI.ClassID=MCS.ClassID
INNER JOIN SMC_Meta_WarehouseClassSchema MWCS
ON MWGI.ClassID=MWCS.ClassID
ORDER BY MCS.ClassName

Running the DTS job with latency:
Add the “/latency:90” switch to the scheduled task command line for the DTS job (MOM.Datawarehousing.DTSPackageGenerator.exe)
(The latency switch specifies the number of days to skip)

Setting query timeout value:
If a *remote* query or DTS job times out, you can change the value in Ent. Manager, on the properties of the server, connections tab.
Default is 600, set this to 0 for unlimited.

Query to look at oldest date of event records:

use SystemCenterReporting
select min(timestored) from sdkeventview

Given the event ID – a quick query to view the number of distinct events for each message, and view the text of the event message:

select distinct message, count(*) as number from sdkeventview where nteventid = '7036' group by message order by number desc

Faking the Onepoint database to think DTS is working in order to groom:
Changed the Date of "TimeDTSLastRan" to today's date in ReportingSettings table of Onepoint DB.
Then, run the MOMX Partitioning and grooming job. This will delete all data in Onepoint based on the global setting of data retention, even if it has not been transferred to the SCDW.

The GroomingSettings table also contains a “TimeDTSLastRan” value – but this will be an old date, as it is typically the time and date of the reporting server install and is apparently not used by the grooming stored procedure.

DTS Logging:

1. Setup Error Logging for the DTS Package. You can use info from the following MS KB to setup DTS Package Logging:
316043 HOW TO: Log Data Transformation Services Package Execution to a Text File
http://support.microsoft.com/?id=316043

2. SQL Log location: Zip up all the files in the SQL Server Log folder "..\program files\microsoft
sql server\MSSQL\LOG" Note: if you have a SQL Named Instance then the path will be as following for
the LOG Folder: "..\program files\microsoft sql server\SQLNamedInstance\MSSQL\LOG"

3. SQL profiler settings to capture for DTS logging:
- Errors and Warnings ---> Add All events under this
- Stored Proc ---> RPC/SP Starting/Completed, SP:Stmt Starting/Completed, and SP:
Recompile
- Transactions ---> All of these
- TSQL ---> SQL:Batch/SQL:Stmt Starting/Completed

Onepoint:

Alerts:

Most common alerts in a Onepoint DB:
This will tell us how many alerts we are generating per day, and

SELECT CONVERT(char(10), TimeRaised, 101) AS "Alert Date (by Day)", COUNT(*) AS "Number of Alerts"
FROM SDKAlertView
GROUP BY CONVERT(char(10), TimeRaised, 101)
ORDER BY "Alert Date (by Day)" DESC

SELECT Culprit, Name, SUM(1) AS AlertCount, SUM(RepeatCount+1) AS AlertCountWithRepeatCount
FROM Alert WITH (NOLOCK)
WHERE ResolutionState = (0|255)
GROUP BY Culprit, Name
ORDER BY AlertCount DESC
-- ORDER BY AlertCountWithRepeatCount DESC

Events:

Most common events in a Onepoint DB:

SELECT CONVERT(char(10), TimeGenerated, 101) AS "Events Date (by Day)", COUNT(*) AS "Number of Events"
FROM SDKEventView
GROUP BY CONVERT(char(10), TimeGenerated, 101)
ORDER BY "Events Date (by Day)" DESC

SELECT NTEventID, COUNT(*) AS "Numver of Events"
FROM SDKEventView
GROUP BY NTEventID
ORDER BY "Numver of Events" DESC

To view the oldest events in the database (this should mirror the “1” in Lastgroomed from above query):

select min(timestored) from sdkeventview

To count the number of events in Onepoint:

select count(*) from sdkeventview

To determine computers that are generating the most event data:

select distinct ComputerName, count(*) as NumberOfOccurences from SDKEventView
group by Computername order by numberofoccurences desc

To determine the noisiest computers and see the events they are generating:

select top 100 Computername, Message, NTEventID, Source, Count(*) AS TOTAL
from SDKEventView
Group by Computername, Message, NTEventID, Source
Order by TOTAL DESC

Selecting multiple events:

Select * from SDKeventview where NTeventID in (‘9980’,’9981’,’9982’,’9983’)

Given the event ID – a quick query to view the number of distinct events for each message, and view the text of the event message:

Select distinct message, count(*) as number from sdkeventview where nteventid = '7036' group by message order by number desc

Performance:

To display the most common perf insertions:

select performanceobjectname, performancecountername, count(performanceobjectname) as 'count' from sdkperformanceview
group by performanceobjectname, performancecountername
order by 'count' desc

To determine computers are generating the most perf insertions:

select distinct server, count(*) as NumberOfOccurences from SampledNumericDataPerformanceReportView
group by server order by numberofoccurences desc

Top 10% Computers with High Perf Volume, by Counter, by Object, by Day

SELECT top 10 percent CONVERT(char(10), TimeSampled, 101) AS "Perf Date (by Day)"
, Computername, PerformanceObjectName, PerformanceCounterName, COUNT(*) AS "Number of PerfObjects"
FROM SDKPerformanceView
GROUP BY CONVERT(char(10), TimeSampled, 101), computername, PerformanceObjectName, PerformanceCounterName
ORDER BY "Number of PerfObjects" DESC

Top 10% Computers with High Perf Volume, by Day

SELECT top 10 percent CONVERT(char(10), TimeSampled, 101) AS "Perf Date (by Day)", computername, COUNT(*) AS "Number of PerfObjects"
FROM SDKPerformanceView
GROUP BY CONVERT(char(10), TimeSampled, 101), computername
ORDER BY "Number of PerfObjects" DESC

Top 10% Computers with High Perf Volume, by Object, by Day

SELECT top 10 percent CONVERT(char(10), TimeSampled, 101) AS "Perf Date (by Day)", computername, performanceobjectname, COUNT(*) AS "Number of PerfObjects"
FROM SDKPerformanceView
GROUP BY CONVERT(char(10), TimeSampled, 101), computername, performanceobjectname
ORDER BY "Number of PerfObjects" DESC

To display the counters causing the most perf impact per computer:

select performanceobjectname, performancecountername, count(performanceobjectname) as 'count'
from sdkperformanceview
where computername = 'COMPUTERNAME'
group by performanceobjectname, performancecountername
order by 'count' desc

Misc Onepoint:

To view the number of Operations Console currently open:

SELECT program_name, count(*)
FROM Master..sysprocesses
WHERE ecid=0 and program_name='Microsoft Operations Manager - DAS Operations Console'
GROUP BY program_name
ORDER BY count(*) desc

To view the grooming information on tables in one point, especially with respect to the “Current” and “LastGroomed” columns =1
SELECT * FROM dbo.PartitionTables

To view or modify the TimeDTSLastRan field to allow grooming to occur:
select * from ReportingSettings

To display the rule name given a GUID:

select Name FROM onepoint.dbo.ProcessRule WHERE idProcessRule=’GUIDstringHERE’

To find all rules associated with a provider:

select pr.name 'Rule Name', pi.name 'Provider Name' from processrule pr
join ProviderInstance pi
on pr.idproviderinstance = pi.idproviderinstance
where pi.name like 'ISA Server%'

To display the Total number of agents (all agent types)

SELECT * FROM ManagedCountsView WHERE ManagedType = ‘-1’

To display the total number of Unmanaged agents

SELECT * FROM ManagedCountsView WHERE ManagedType = ‘0’

To display the total number of Managed agents

SELECT * FROM ManagedCountsView WHERE ManagedType = ‘2’

To display the total number of Windows Server Cluster computers

SELECT * FROM ManagedCountsView WHERE ManagedType = ‘3’

To display the total number of Agents reporting to a specific Mgmt Server

SELECT COUNT(*) FROM MOMv2_ComputerAllPropertiesView WHERE ConfigManagerName = ‘mgmt server name’

or to list the total number of Agents reporting to all Mgmt Servers (NULL are unmanaged)

SELECT configmanagername, COUNT(*) FROM MOMv2_ComputerAllPropertiesView group by ConfigManagerName

To display the agent action account for all servers:

use OnePoint
Select Name, Value
from Attribute INNER JOIN Computer ON DISCOVERYCOMPUTERID = IDCOMPUTER
WHERE ClassAttributeID IN (Select ClassAttributeID from ClassAttribute where
ClassAttributeName = 'Action Account Identity')
AND IDComputer IN (Select DiscoveryComputerID from Attribute WHERE
ClassAttributeID IN (Select ClassAttributeID from ClassAttribute where
ClassAttributeName = 'Action Account Identity'))
order by value, name

MISC:

Simple query to display large tables:

SELECT
so.name,
8 * Sum(CASE WHEN si.indid IN (0, 1) THEN si.reserved END) AS data_kb,
Coalesce(8 * Sum(CASE WHEN si.indid NOT IN (0, 1, 255) THEN si.reserved END), 0) AS index_kb,
Coalesce(8 * Sum(CASE WHEN si.indid IN (255) THEN si.reserved END), 0) AS blob_kb
FROM dbo.sysobjects AS so JOIN dbo.sysindexes AS si ON (si.id = so.id)
WHERE 'U' = so.type GROUP BY so.name ORDER BY data_kb DESC

Simple query for Onepoint or SCDW to dump a perfcounter to a table output. Modify Computername and timestamps:

DECLARE @BeginDate datetime
DECLARE @EndDate datetime
SET @BeginDate = '2006-10-18 05:45:00.287'
SET @EndDate = '2006-10-19 05:45:00.287'
SELECT * FROM dbo.SDKPerformanceView WHERE Computername = 'EXCH1' and PERFORMANCECOUNTERNAME = 'Local Queue Length' and TimeSampled BETWEEN @BeginDate AND @EndDate ORDER BY TimeSampled DESC

Database Performance:

The System Center DW does not come with any maintenance.
I really like the maintenance plan at: http://systemcenterforum.org/wp-content/uploads/scdw_reindex1.zip

To get better performance manually:

Update Statistics (will help speed up reports and takes less time than a full reindex):

EXEC sp_updatestats

Show index fragmentation (to determine how badly you need a reindex – logical scan frag > 10% = bad. Scan density below 80 = bad):

DBCC SHOWCONTIG
DBCC SHOWCONTIG WITH FAST (less data than above – in case you don’t have time)

Reindex the database:

Onepoint:

USE OnePoint
go
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
SET ARITHABORT ON
SET CONCAT_NULL_YIELDS_NULL ON
SET QUOTED_IDENTIFIER ON
SET NUMERIC_ROUNDABORT OFF
EXEC SP_MSForEachTable "Print 'Reindexing '+'?' DBCC DBREINDEX ('?')"
            SystemCenterReporting:
            DBCC REINDEX (‘TableName’)
            DBCC REINDEX (‘SC_SampledNumericDataFact_Table’)

Update usage (use when SQL reports a table size that you know not to be correct):

DBCC updateusage ('systemcenterreporting')
DBCC updateusage ('systemcenterreporting','sc_samplednumericdatafact_table')

How to identify your version of SQL server:

SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

SQL 2005:

SQL Server 2005 RTM                           2005.90.1399
SQL Server 2005 SP1                           2005.90.2047
SQL Server 2005 SP1 plus 918222         2005.90.2153
SQL Server 2005 SP2                           2005.90.3042

SQL 2000:

SQL Server 2000 RTM               2000.80.194.0
SQL Server 2000 SP1                2000.80.384.0
SQL Server 2000 SP2                2000.80.534.0
SQL Server 2000 SP3                2000.80.760.0
SQL Server 2000 SP3a              2000.80.760.0
SQL Server 2000 SP4                2000.8.00.2039

SQL2000 Reporting RTM             8.00.743.00
SQL2000 Reporting SP1              8.00.878.00
SQL2000 Reporting SP2              8.00.1038.00