Jimmy Harper's Operations Manager Blog

ACS “Event Count” reports

2013-04-18T14:47:00Z

Here are a couple of ACS reports to show the event count by User, Forwarder, and Event ID. Just import the .rdl files into a folder in SQL Server Reporting Services.

The first report “Event Count Overview” shows the top 10 Events, Forwarders, and User Names for the specified date range:

Click on any of the bars in the chart or the chart titles to launch the “Event Count Details” report. This report shows more detail about the item that you clicked on. For example, if you click on the bar for a specific event Id in the “Top 10 Events” section, it will show the top 10 User Names and Forwarders for this event:

If you click on the bar for a specific Forwarder in the “Top 10 Forwarders” section, it will show the top 10 User Names and Events IDs for this forwarder:

If you click on one of the chart titles it will show details for all events in the chart. For example, clicking on “Top 10 Events” shows the top 10 User Names and Forwarders for each of the top 10 events:

You can also run the “Event Count Details” report on its own to show the Top N User Name, Event or Forwarder based on a User, Event, or Computer Name filter:

CU6 for OpsMgr 2007 R2 is now available

2012-05-15T15:58:33Z

CU6 for OpsMgr 2007 R2 is now available at http://www.microsoft.com/en-us/download/details.aspx?id=29850. You can also find the corresponding KB Article at http://support.microsoft.com/kb/2626076.

Cumulative Update 6 for Operations Manager 2007 R2 resolves the following issues:

RMS promotion fails if NetworkName and PrincipalNames are not in sync for agents.
UI is limited to only 100 MB for the Memory Usage field in the wizard.
Additional OIDs in auth certificate are not processed correctly.
AEM creates duplicate computer objects in OpsMgr based on Agents NetBIOS name.
Cannot open reporting pane on OpsMgr 2007 R2 remote console.
Cannot view schedule for scheduled report.
ManagementServerConfigTool with the option "promoterms" fails because it stops polling the SDK Service.
OpsMgr reports are failing on Windows 7 with the error: "Cannot initialize report."
ACS events have "n/a" as their category in the ACS database.
Watch agentless monitoring listener to detect failure to respond.
SCOM SDK memory leak on cryptography keys and cryptography contexts.
After you click Edit Schedule, a message box appears, and you cannot save the change value.
Audit events can be lost when the AdtServer process crashes.

Cumulative Update 6 for Operations Manager 2007 R2 resolves the following cross-platform issues:

The installation process for the IBM AIX 6.1 agent incorrectly checks for AIX 5.3 packages.
After a system restart, the OpsMgr agent for Solaris may start to run before local file systems are mounted.
On Red Hat Linux version 4 and SUSE Linux version 9, LVM disks are not discovered and cannot be monitored.
The OpsMgr agent for AIX does not report the arguments for monitored processes.
When Microsoft security update MS12-006 is installed on an OpsMgr management server, that management server can no longer communicate with the OpsMgr agent on any Linux or UNIX server.
On HP-UX, OpsMgr cannot discover and monitor a logical volume that is composed of more than 127 physical volumes.

Cumulative Update 6 for Operations Manager 2007 R2 adds the following cross-platform features:

Support for IBM AIX 7.1 (POWER).
Support for Oracle Solaris 11 (x86 and SPARC).

Updated Active Directory Management Pack Released - version 6.0.7822.0

2012-03-30T20:52:16Z

We have just released and updated version of the Active Directory Management Pack (version 6.0.7822.0) that corrects the problems addressed in my recent “Addendum” MP. Once this new ADMP is imported, the Addendum MP can be removed.

The new ADMP and guide can be downloaded from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=21357.

Here is a list of fixes, from the MP Guide:

Fix	Operating system monitoring rules impacted
Corrected some Publisher names (for example, changed from PublisherName=KDC to PublisherName=Microsoft-Windows-Kerberos-Key-Distribution-Center).	Windows Server 2008 and later
Updated several important rules to generate Alerts and not only go to the Event Viewer.	Windows Server 2008 and later
Removed unnecessary check for Event Source Name for all NTDS rules (for example, removed EventSourceName=”NTDS General”).	Windows Server 2003 and later
Corrected event parameter validation.	Windows Server 2003 and later
Updated some queries to search for correct event IDs.	Windows Server 2003 and later
Fixed spelling errors.	Windows Server 2003 and later
Added missing descriptions to several rules.	Windows Server 2003 and later
Fixed several problems with Health Monitoring scripts.	Windows Server 2003 and later
Removed user name checks from Userenv rules.	Windows Server 2003

Creating a Group of Windows Computer and Health Service Watcher Objects

2012-03-20T13:15:00Z

There are a lot of great blogs out there that discuss creating various types of groups (Kevin Holman and Jonathan Almquist have a number of them), and the stuff that I do in this post won’t be anything that hasn’t been written about before, but I wanted to have one spot to cover the questions that I get most often:

How do I create a group of Windows Computer objects for specific “types” of servers (Domain Controllers, SQL Servers, etc.)?
How do I include the associated Health Service Watcher objects in this group?

First, a little info about why you might need to do this. You can easily create a group of “Active Directory Domain Controller” objects in the OpsMgr Console….just add the class to your group criteria and add additional criteria for the computer name or other property:

But what if you want to use this group in the criteria for an Alert Subscription? If you create a subscription for any alert from an object in this group, the subscription will only pick up alerts that are targeted at the Domain Controller classes. This may be just want you want, but what if you want the subscription to pick up ALL alerts from the Domain Controllers? For that, you would need to have Windows Computer objects in the group. Also, what if you want to use the group as a target for overrides set on workflows targeted at non-Domain Controller classes (for example, disable a Rule in the Operating System Management Pack on all Domain Controllers)? Again, to do this, the group would need to contain Windows Computer objects.

Now, you can easily create a group of Windows Computers computer objects in the OpsMgr Console….just add the Windows Computer class to the group criteria and use the Windows Computer properties to determine which computer objects go into the group:

Ok, so now you have a group of Windows Computers based on Computer Name and Active Directory Site. You then set up a subscription to get all alerts for the servers in this group. Everything seems fine until one of the servers goes offline and you don’t get a notification about it. In the Console, you can see the “Health Service Heartbeat Failure” alert, but no notification was sent for it. This is because the “Health Service Watcher” object for each of the computers was not included in the group….this MUST be included if you want to include “Heartbeat Failure” alerts in the subscription.

So, now we can go back into the group criteria and add the Health Service Watcher object:

However, the Health Service Watcher class doesn’t include all of the Windows Computer properties, so we can basically only filter on the computer name….in the above example I cannot include the “Active Directory Site” criteria for HS Watcher like I did on Windows Computer, so this group will likely include more HS Watcher objects than you want. Another downside to this group is that the naming standard for your servers might now always indicate if they are a Domain Controller, SQL Server, etc….so you may not be able to use any Windows Computer properties to properly identify them.

So, we want to create a group that:

Contains computers of a specific class (Domain Controller, SQL, Exchange, etc)
Contains Windows Computer objects (so ALL alerts for the computers will be picked up by our Alert Subscription)
Contains Health Service Watcher objects for any computer object that is in the group (so Heartbeat Failure alerts will be picked up by our Alert Subscription)

In this example, the group will contain Domain Controllers that are in the NYC site.

First, we’ll create a new group in the OpsMgr Authoring Console (Jonathan covers this here):

Open the OpsMgr Authoring Console and either create a new Management Pack or open an existing one

Next, we’ll need to add a reference to the System Center Instance Group Library Management Pack (so we can use the relationship types that are defined in this MP). Go to File - Management Pack Properties - References - Add Reference and select the “Microsoft.SystemCenter.InstanceGroup.Library.mp” file (this should be in the installation directory on the RMS). Since we’ll be using the Domain Controller classes, we’ll also need a reference to the Active Directory Library MP, so let’s go ahead and add that one now too.

Now we’ll create a class for the group - in OpsMgr, a group is really just a class that is populated with various objects (the group members) - Kevin Holman covers this here. We’ll create a Discovery Rule for this next. In the Authoring Console, go to Service Model\Classes and create a new “Custom Class”. For the Base Class, select Microsoft.SystemCenter.InstanceGroup and select “Singleton” under Attributes. We don’t need to do anything on the other tabs….just click ok to save it.

Next we’ll need to create a Discovery Rule to populate the group . Go to Health Model - Discoveries and create a new “Custom Discovery”. On the General tab, give the Discovery a name, and under “Target” select the class that you created for the group:

On the Discovered Classes tab ,go to the “Discovered relationships and their attributes” section, select Add - Add Discovered Relationship and select “Microsoft.SystemCenter.InstanceGroupContainsEntities”.

On the Configuration tab, select “Browse for a type” and select the “Microsoft.SystemCenter.GroupPopulator type. In the “Module ID” section, type a name for the module

Click on Ok, and you will see the following error….just ignore it and click on Close

Next we see the Configuration tab with fields for RuleID, GroupInstanceID, and MembershipRules. For RuleID, enter $MPElement$. For GroupInstanceID, enter $MPElement[Name="NAME OF THE GROUP CLASS"]$…so in this case we would use $MPElement[Name="SAMPLE.OpsMgr.Groups.NYC.DCs"]$. We’ll do the rest by editing the XML directly, so just click on OK and save the Management Pack.

Save the MP, close the Authoring Console, and open the XML file in any XML or text editor (I like Notepad++). In the XML, fine the Monitoring\Discoveries section and find the discovery for your group….here’s what it should look like:

<Monitoring>
<Discoveries>
    <Discovery ID="SAMPLE.OpsMgr.Groups.NYCDCs.GroupPopulation" Enabled="true" Target="SAMPLE.OpsMgr.Groups.NYC.DCs" ConfirmDelivery="true" Remotable="true" Priority="Normal">
      <Category>Discovery</Category>
      <DiscoveryTypes>
        <DiscoveryRelationship TypeID="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities" />
      </DiscoveryTypes>
      <DataSource ID="GroupPopulator" TypeID="SC!Microsoft.SystemCenter.GroupPopulator">
        <RuleId>$MPElement$</RuleId>
        <GroupInstanceId>$MPElement[Name="SAMPLE.OpsMgr.Groups.NYC.DCs"]$</GroupInstanceId>
        <MembershipRules>
          <MembershipRule>
            <MonitoringClass>MonitoringClass</MonitoringClass>
            <RelationshipClass>RelationshipClass</RelationshipClass>
          </MembershipRule>
        </MembershipRules>
      </DataSource>
    </Discovery>
</Discoveries>
</Monitoring>

Next, we need to create membership rules for the Windows Computer and Health Service Watcher classes….the discovery already has one membership rule pre-created, so we’ll use this for the Windows Computer objects….here’s what the rule currently looks like:

<MembershipRules>
<MembershipRule>
<MonitoringClass>MonitoringClass</MonitoringClass>
<RelationshipClass>RelationshipClass</RelationshipClass>
</MembershipRule>

First, replace MonitoringClass with the Windows Computer Class, and RelationshipClass with InstanceGroupContainsEntities, so now it will look like this:

<MembershipRules>
<MembershipRule>
<MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
<RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
</MembershipRule>

This part of the membership rule just says that the group will contains Windows Computer objects…..next we’ll add criteria to include only Domain Controller computer objects. Here, we’re just adding an expression that says “Only include Windows Computer objects that contain the Domain Controller Role class”. The next XML for the membership rule will look like this:

            <MembershipRule>
              <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
              <RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
              <Expression>
                <Contains>
                  <MonitoringClass>$MPElement[Name="MicrosoftWindowsServerADLibrary!Microsoft.Windows.Server.AD.DomainControllerRole"]$</MonitoringClass>
                </Contains>
              </Expression>
            </MembershipRule>

Next, we’ll add a second membership rule for the Health Service Watcher objects. Here is the XML for this membership rule…it is basically saying “Include Health Service Watcher Objects for the computers that are already in this group”

<MembershipRule>
<MonitoringClass>$MPElement[Name="SystemCenter!Microsoft.SystemCenter.HealthServiceWatcher"]$</MonitoringClass>
<RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
<Expression>
    <Contains>
      <MonitoringClass>$MPElement[Name="SystemCenter!Microsoft.SystemCenter.HealthService"]$</MonitoringClass>
      <Expression>
        <Contained>
          <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
          <Expression>
            <Contained>
              <MonitoringClass>$Target/Id$</MonitoringClass>
            </Contained>
          </Expression>
        </Contained>
      </Expression>
    </Contains>
</Expression>
</MembershipRule>

So, here’s what the whole discovery looks like now:

      <Discovery ID="SAMPLE.OpsMgr.Groups.NYCDCs.GroupPopulation" Enabled="true" Target="SAMPLE.OpsMgr.Groups.NYC.DCs" ConfirmDelivery="true" Remotable="true" Priority="Normal">
        <Category>Discovery</Category>
        <DiscoveryTypes>
          <DiscoveryRelationship TypeID="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities" />
        </DiscoveryTypes>
        <DataSource ID="GroupPopulator" TypeID="SC!Microsoft.SystemCenter.GroupPopulator">
          <RuleId>$MPElement$</RuleId>
          <GroupInstanceId>$MPElement[Name="SAMPLE.OpsMgr.Groups.NYC.DCs"]$</GroupInstanceId>
          <MembershipRules>
            <MembershipRule>
              <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
              <RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
              <Expression>
                <Contains>
                  <MonitoringClass>$MPElement[Name="MicrosoftWindowsServerADLibrary!Microsoft.Windows.Server.AD.DomainControllerRole"]$</MonitoringClass>
                </Contains>
              </Expression>
            </MembershipRule>
            <MembershipRule>
              <MonitoringClass>$MPElement[Name="SC!Microsoft.SystemCenter.HealthServiceWatcher"]$</MonitoringClass>
              <RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
              <Expression>
                <Contains>
                  <MonitoringClass>$MPElement[Name="SC!Microsoft.SystemCenter.HealthService"]$</MonitoringClass>
                  <Expression>
                    <Contained>
                      <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
                      <Expression>
                        <Contained>
                          <MonitoringClass>$Target/Id$</MonitoringClass>
                        </Contained>
                      </Expression>
                    </Contained>
                  </Expression>
                </Contains>
              </Expression>
            </MembershipRule>
          </MembershipRules>
        </DataSource>
      </Discovery>

This group will include all Domain Controllers and their Health Service Watcher objects. Next, we’ll add criteria to only include Domain Controllers that are in the NYC Site. For this I’ll go back to the first membership rule and add an expression for the AD Site name and enclose this and the original expression in an AND expression. Here’s what the new xml for the discovery will look like (changes highlighted):

      <Discovery ID="SAMPLE.OpsMgr.Groups.NYCDCs.GroupPopulation" Enabled="true" Target="SAMPLE.OpsMgr.Groups.NYC.DCs" ConfirmDelivery="true" Remotable="true" Priority="Normal">
        <Category>Discovery</Category>
        <DiscoveryTypes>
          <DiscoveryRelationship TypeID="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities" />
        </DiscoveryTypes>
        <DataSource ID="GroupPopulator" TypeID="SC!Microsoft.SystemCenter.GroupPopulator">
          <RuleId>$MPElement$</RuleId>
          <GroupInstanceId>$MPElement[Name="SAMPLE.OpsMgr.Groups.NYC.DCs"]$</GroupInstanceId>
          <MembershipRules>
            <MembershipRule>
              <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
              <RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
              <Expression>
               <And>
              <Expression>
                <Contains>
                  <MonitoringClass>$MPElement[Name="MicrosoftWindowsServerADLibrary!Microsoft.Windows.Server.AD.DomainControllerRole"]$</MonitoringClass>
                </Contains>
              </Expression>
                  <Expression>
                    <SimpleExpression>
                      <ValueExpression>
                        <Property>$MPElement[Name="Windows!Microsoft.Windows.Computer"]/ActiveDirectorySite$</Property>
                      </ValueExpression>
                      <Operator>Equal</Operator>
                      <ValueExpression>
                        <Value>NYC</Value>
                      </ValueExpression>
                    </SimpleExpression>
                  </Expression>
                </And>
              </Expression>
            </MembershipRule>
            <MembershipRule>
              <MonitoringClass>$MPElement[Name="SC!Microsoft.SystemCenter.HealthServiceWatcher"]$</MonitoringClass>
              <RelationshipClass>$MPElement[Name="MicrosoftSystemCenterInstanceGroupLibrary!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
              <Expression>
                <Contains>
                  <MonitoringClass>$MPElement[Name="SC!Microsoft.SystemCenter.HealthService"]$</MonitoringClass>
                  <Expression>
                    <Contained>
                      <MonitoringClass>$MPElement[Name="Windows!Microsoft.Windows.Computer"]$</MonitoringClass>
                      <Expression>
                        <Contained>
                          <MonitoringClass>$Target/Id$</MonitoringClass>
                        </Contained>
                      </Expression>
                    </Contained>
                  </Expression>
                </Contains>
              </Expression>
            </MembershipRule>
          </MembershipRules>
        </DataSource>
      </Discovery>

Now I just save the MP and import into my Management Group and verify the group membership is correct

The MP that I created is attached, as well as MPs that created Computer/HSW groups for Domain Controllers, SQL servers and Exchange 2010 servers.

OpsMgr 2007 R2 Agent Installation Error with Microsoft.VC90.CRT

2012-02-16T23:44:00Z

I’ve seen this issue a couple of times now…an OpsMgr 2007 R2 Agent installation fails with the following error:

Product: System Center Operations Manager 2007 R2 Agent -- Error 1935.An error occured during the installation of assembly component {9718BF1D-6A9F-3406-A992-630F7EF2F164}. HRESULT: 0x800736FD. assembly interface: IAssemblyCacheItem, function: Commit, assembly name: Microsoft.VC90.CRT,version="9.0.30729.1",publicKeyToken="1fc8b3b9a1e18e3b",processorArchitecture="amd64",type="win32"

Error 1935.An error occured during the installation of assembly component {9718BF1D-6A9F-3406-A992-630F7EF2F164}. HRESULT: 0x800736FD. assembly interface: IAssemblyCacheItem, function: Commit, assembly name: Microsoft.VC90.CRT,version="9.0.30729.1",publicKeyToken="1fc8b3b9a1e18e3b",processorArchitecture="amd64",type="win32"

I’ve seen this problem resolved a few different ways…here are the steps I would recommend:

Reboot the server and try the installation again. If you still get the error, go to step 2
Reinstall the Visual C++ 2008 SP1 redistributable Package…this can be downloaded here. If it still fails, or if you get the same error when installing the Visual C++ redistributable, go to step 3.
Made the following registry change, then reboot and try the install again:

Set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control - RegistrySizeLimit to 4294967295 (decimal)

Of course, you could probably save some time by just reinstalling the VC++ package, setting the registry key, and rebooting.

Please comment if you come across any other resolutions to this error.

ADMP – Workaround for Some Broken Win2k8 Rules

2012-01-16T20:29:00Z

In a previous blog post, I provided an “Addendum” MP to fix some Rules that do not work correctly in the Windows Server 2008 version of the Active Directory Management Pack. All of these Rules have been fixed in the latest version of the ADMP (version 6.0.7670.0)…so if you’ve installed the latest version you should go ahead and remove that Addendum MP if you haven’t already.

However, we’ve recently found that we have some other Rules which are not working correctly, and these have not been corrected in the 7670 release of the MP. These include some Rules that alert on events in the System and Application Event Log on Windows Server 2008 Domain Controllers, and some Rules that alert on events in the Application log on Windows Server 2003 Domain Controllers.

I’ve created a new “Addendum” MP to correct these Rules….note that these Rules are broken in previous versions of the ADMP as well, so if you are not currently using the 7670 version of the MP, you will still need this + the previous addendum.

The ADMP developers are aware of this problem and it will be corrected in the next release of the AD Management Pack.

Unsealed and sealed versions of the new Addendum MP are attached.

Logical Disk Fragmentation Report

2011-08-25T19:47:00Z

Here’s a simple custom report that shows the fragmentation of each logical disk based on the Logical Disk Fragmentation Monitor in the Windows Server Operating System Management Pack. The report has a “Fragmentation Percentage Threshold” parameter, and will list any disk whose fragmentation level is higher than the threshold. The report will also show the last time the fragmentation monitor changed state.

NOTE: This report uses the Operations Database instead of the Data Warehouse….in order for it to work, you must create an SRS Data Source for the OpsDB…Kevin Holman has instructions for this here.

Workflow Failure Queries

2011-07-30T02:02:54Z

I regularly see customers getting a lot of alerts indicating that workflows have failed to run a script of WMI query…the alert name is usually one of the following:

Workflow Runtime: Failed to run a process or script
Workflow Runtime: Failed to run a WMI query
Workflow Runtime: Failed to run a WMI query for WMI events
Workflow Initialization: Failed to start a workflow that runs a process or script
Workflow Initialization: Failed to start a process or script

The descriptions on these alerts will give details about the workflow that failed and the script or WMI error. It’s important to review these alerts to determine if there is:

A specific workflow that is consistently failing
Specific Agents where workflows are consistently failing
Specific Operating Systems where workflows are consistently failing

The following queries can be run against the OpsMgr Data Warehouse database to get more information about these alerts:

--Get all workflow failure alerts within the last 30 days
select
a.RaisedDatetime
,me.DisplayName as Agent
,a.AlertName
,a.AlertDescription
,RepeatCount
from Alert.vAlert a with (NOLOCK)
join vManagedEntity me on me.ManagedEntityRowId=a.ManagedEntityRowId
where (AlertName like '%workflow runtime%' or AlertName like '%workflow initialization%')
and a.RaisedDateTime > DateAdd (dd,-30,GetUTCDate())
order by Displayname,RaisedDateTime

--Get server name, Alert name, and count of workflow failure alerts within the last 30 days
select
me.DisplayName as Agent
,a.AlertName
,Count(*) as Count
,Sum(RepeatCount) as TotalRepeatCount
from Alert.vAlert a with (NOLOCK)
join vManagedEntity me on me.ManagedEntityRowId=a.ManagedEntityRowId
where (AlertName like '%workflow runtime%' or AlertName like '%workflow initialization%')
and a.RaisedDateTime > DateAdd (dd,-30,GetUTCDate())
group by me.DisplayName,a.AlertName
order by Count(*) desc,Displayname,AlertName

--Get Operating System name, Alert name, and count of workflow failure alerts within the last 30 days
select
me2.DisplayName as "Operating System"
,a.AlertName
,Count(*) as Count
,Sum(RepeatCount) as TotalRepeatCount
from Alert.vAlert a with (NOLOCK)
join vManagedEntity me on me.ManagedEntityRowId=a.ManagedEntityRowId
join vManagedEntity me2 on (me2.TopLevelHostManagedEntityRowId=me.TopLevelHostManagedEntityRowId and me2.ManagedEntityTypeRowId in (select ManagedEntityTypeRowId from vManagedEntityType where ManagedEntityTypeSystemName = 'Microsoft.Windows.OperatingSystem'))
where (AlertName like '%workflow runtime%' or AlertName like '%workflow initialization%')
and a.RaisedDateTime > DateAdd (dd,-30,GetUTCDate())
group by me2.DisplayName,a.AlertName
order by Count(*) desc,me2.Displayname,AlertName

Maintenance Mode Scheduling Tool

2011-07-14T20:30:20Z

The Operations Manager Product Team recently released an Administration Resource Kit for System Center Operations Manager 2007 R2. One of the tools in the kit is a Maintenance Mode Scheduler….I finally got around to trying this out today…below are my notes.

Setup

Download the Resource Kit and run the installation (System Center Operations Manager 2007 R2 Admin Reskit.msi)
Navigate to the directory where the MM tool is installed (in my case, C:\Program Files (x86)\System Center\SystemCenter Operations Manager 2007 R2 Admin Reskit\ScheduleMaintenanceMode)…we should see four folders:
In the “Guide” folder is a doc that explains how to setup, configure and use the tool…BE SURE TO READ THIS FIRST.
Here’s a quick summary of the setup instructions:
1. Copy the EnableMmodeWriteActionModule.dll and MOMSCRIPTAPILib.dll to the installation directory on the RMS (usually C:\Program Files\System Center Operations Manager 2007). These are in the x64 and x86 subfolders of the folder mentioned above….use whichever architecture applies to your installation.
2. Import the ScheduleMaintenanceMode.Library.MP Management Pack using the OpsMgr console. This is in the MP subfolder mentioned above, but we have an updated version of this MP available here.
3. Restart the Health Service on the RMS
4. Copy the ScheduleMaintenanceMode.exe file to the computer where you will be running the tool…again, this is in the x64 and x86 directories…use whichever one matches the architecture of your computer. Also note that you must have the OpsMgr 2007 R2 Console and .NET Framework 3.5 installed on the computer where you will be running the tool

Configuration

On the computer that you will do the scheduling from, run "ScheduleMaintenanceMode.exe”. The first time, you will get a pop-up:

Just click on OK to get to the “Connect” UI

Enter the name of the Management Group and RMS, then click Test to verify we can connect:

Success!

Click on “Save” to save your configuration (note that the “Connect” button won’t light up until you do this), then click on “Connect”…this should bring up the Jobs/Schedule UI:

If you get any errors running the tool, make sure:

You have copied EnableMmodeWriteActionModule.dll and MOMSCRIPTAPILib.dll to the installation directory on the RMS
You have restarted the Health Service on the RMS after copying the above DLL files
Your workstation has the OpsMgr 2007 R2 Console and .NET Framework 3.5 installed

Usage

Using the tool is pretty simple….you create Jobs (which objects to put into MM) and Schedules (when to run the Jobs).

On the Jobs tab, click on “Create” to create a new Maintenance Mode job. Here, you will specify which objects will be put into MM. Select the object type, then select the specific objects and click “Add”. You’ll also need to give the job a Name, Category, and Comment. Here, I will put my SQL1 and SQL2 servers into MM. Since I want to take the servers offline, I will need to put all objects into MM, so I select “Windows Computer” as the Object Type and select “Selected objects and all their contained objects”.

Next I’ll need to create a schedule to determine when the servers will go into MM and for how long. Click on the “New” button to create a new schedule (if you’ve already created other schedules, they will be seen in the drop down list.

Give the schedule a name, and select “Run Once” or “Recurrence”. For this job, I’ll just select “Run Once”, we’ll look at the Recurrence options later. NOTE: The scheduler will use the local time on the RMS.

A look in the console verifies that the servers are in Maintenance Mode:

What if I don’t want to put all objects for a Computer into MM, and just want it for a specific object, like a SQL Database? Simple, create a new job, select the object type and the object:

What if I want to put a group into MM? This one isn’t quite as obvious. If you select the group from the drop down list, the object list will show everything in the group and you’ll have to add each individually.

If you have over 20 objects in the group, this is not a good idea, and you’ll want to put the group object into MM. To do this, select the “Group | System.Group” item in the Object Type drop-down, then select the group from the Object list:

Canceling Maintenance Mode

If you need to cancel a job before its end time, just open the scheduler tool, find the job, and click on “Cancel Job”

Recurrence options

For scheduling the MM jobs, we have options for “Run Once” (you just give the start and end date/time) and “Recurrence” (daily, weekly, or monthly). For each of these, you supply the start date/time and duration (how long to be in MM)

Daily

For the “Daily” recurrence, you just specify number of days between jobs

Weekly

For the “Weekly” option, you specify how many weeks between jobs and which days of the weeks to run it

Monthly

For the “Monthly” option, you select which months to run the job in, and which day(s) of the month….you can select day 1-31, “Last day”, or the First/Second/Third/Fourth/Fifth Sun-Sat

Management Pack

The Management Pack for the Maintenance Mode Scheduler contains classes for the Jobs, Schedules, and “Schedule Maintenance Mode Service” (this is just the RMS). The MP contains three Rules:

Collect Scheduled Maintenance Mode Events
Target: Job component
This Rule collects Event ID 700, which is logged on the RMS when a MM job runs

Execute Maintenance Mode Workflow
Target: Job component
This rule runs every 60 seconds and manages the jobs and schedules (starts/stops MM)…here is some more info on this workflow (copied/pasted directly from the guide):

Every minute, the Execute Maintenance Mode Workflow runs against each scheduled job created.
If the Job Status is Scheduled, the workflow will read the schedule start time.
1. If the start time is equal to the current running time, the workflow will place every object defined in the job into maintenance mode and update the Job Status to Running.
2. If the start time is not equal to the current running time, the workflow will end.
If the Job Status is Running, the workflow will read the schedule start and end time.
1. If the current time is not past the end time, the workflow will check and make sure everything is still in maintenance mode. If something is not in maintenance mode and should be, the workflow will place it back in and drop an event to the event log. A rule targeting the job will read this event and generate an alert to notify the Operator that the object was placed back into maintenance mode.
2. If the Job is past its end time the workflow will update the Job Status to say “Completed”
If the Job Status is Completed, the workflow will check to see if this is run once or is a recurrence
1. If recurrence, the workflow recalculates the next run time of the schedule, writes the completed job to history, and updates job status to Scheduled
2. If Run once, the job status is updated to Run Once Finished and a history event dropped.
If the Job is marked Canceled, the workflow removes each object from maintenance mode and marks the Job status as Completed – User canceled.
If the Job is marked Finished (in the case of a run once job), the job is deleted

You can set an override on this rule to enable logging for troubleshooting purposes….set the “TraceLevel” parameter to 4 for the job that you need to troubleshoot, then review the log file on the RMS…details on this from the guide:

If HealthService is running under local system it is found under “C:\Windows\System32\config\systemprofile\AppData\Local”

If Health Service is running under some other account
“\Users\<AccountName>\AppData\Local”

The name of the files begins with “Enablemmode” and ends with the “ME.ID .log”

For example:
Enablemmode_2372ef81-5c64-75e3-5fe0-c297884004b2.log

Place back into Maintenance Mode
Target: Schedule Maintenance Mode Service
This Rule watches for event 623 on the RMS and generates an Alert. Event 623 indicates that an object was taken out of Maintenance Mode before its end time (according to the defined jobs/schedules). When this happens, the object will be placed back into MM (by the Execute Maintenance Mode Workflow Rule). The purpose of this Rule is to generate an alert when this happens.

The Management Pack also contains views to show the Jobs, Schedules and Events. In the Monitoring section of the Console, navigate to the “Schedule Maintenance Mode” folder:

The Jobs view shows all of the Jobs that have been created, and their current status. Note that jobs that are scheduled to run only once are deleted after they have finished.

The Schedules view shows all of the schedules that have been created

The “Job History Events” view shows all of the Event ID 700 events that have been collected….these events tell use when the jobs start and finish

Here are some best practices and known issues (copied directly from Rob Kuehfus’ blog here:

Best Practices:

Do not go beyond 20 Jobs scheduled. Anything over this will start to place too much load on your Root Management Server
Do not schedule more then 20 items in one Job. If you need to go over this please create a group and target this. Note: The MP has a bug that only limits you to select up to six objects. I have attached an updated MP that corrects this issue below.
When scheduling a group make sure to select system.group then select the group. If you select the group itself the tool lists the individual group membership.

Known Issues:

A common issue with our international customers is a date time casting exceptions or the schedule is not landing on the correct day. This tool was designed and tested on EN-US only. I cannot make any promises but I will look into extending this to support other locals in the future. The work around is to install the Windows EN-US language pack.

Unhealthy Monitors Report

2011-05-06T20:11:00Z

I recently put together an “Unhealthy Monitors” report for an OpsMgr Administrator to see which Monitors are causing the most state degradation in the environment. This report lists all monitors that have objects in an unhealthy state, so we can see which monitors have the most unhealthy objects. Each monitor can be expanded to see which objects are unhealthy.

This does not show us which Monitors are generating the most State Changes, just which ones have the most objects which are currently unhealthy. The Monitors at the top of the list would generally need to be tuned (reconfigure thresholds, etc.) or disabled (Administrator is not concerned with the issue), or a systemic problem in the environment needs to be fixed.

The report doesn’t take any parameters…just import into SRS and run it.

The output will show the Monitor name, Target class, Management Pack name, and the number of objects that are not in a healthy state:

Each Monitor in the list is expandable to show a list of objects that are in an unhealthy state…the list shows the Path, Display Name, Health State, and Last Modified time (when it last changed to an unhealthy state):

The table format of the report makes it very easy to export to Excel, where you can do filtering and sorting: