With System Center Operations Manager 2012, the 32-bit agent cannot be installed in a 64-bit server. Although - in general - 32-bit applications can still be monitored by the 64-bit agent, you won’t be able to collect 32-bit custom performance counters by using the standard, performance-based, rules and monitors. By standard, I mean the rules and monitors you can create via the console by selecting “Windows Performance Counters” in the Create a New Rule/Monitor wizards.

Nonetheless, with some dose of backstage maneuvering, this limitation can be circumvented. In this post, I would like to show you how you can do that. The short answer is: use the log-based performance counter data source!

It turns out that, buried in the Performance Library Management Pack (System.Performance.Library), we can find the building blocks we need to achieve our goal. This MP includes all of the rules and monitors used in the standard performance counter collection, as well as those needed for log-based counter collection. These rules and monitors are able to collect performance counter data by parsing Windows Performance Monitor (perfmon) log files. Hence, they are called “PerformanceLogBased” and all rely on the PerformanceLogDataProvider data source module to do the parsing.  Unfortunately, this feature is not exposed through the SCOM console, so that’s where the “backstage” maneuvering comes into place.

What is needed?

1. A simple custom MP which takes advantage of the rules/monitors defined in the Performace Library.
2. Windows Performance Monitor logging (perfmon). Logging must be enabled for each counter which needs to be collected.

A Simple Example

Goal: Collect and monitor the number of remote desktop sessions in agents and change state of the monitor to “Critical” when this number exceeds 4 over 2 consecutive samples. We also want to store the counter values as performance data in the Operational Database.

Performance Counter: Terminal Services/Total Sessions

Unit Monitor Definition

 <UnitMonitor ID="PerfLogConsecutiveSamplesMonitor" Accessibility="Public" Enabled="true" Target="SystemCenter!Microsoft.SystemCenter.Agent" ParentMonitorID="Health!System.Health.AvailabilityState" Remotable="true" Priority="Normal" TypeID="Performance!System.Performance.PerformanceLogBased.ConsecutiveSamplesThreshold" ConfirmDelivery="false">

  <Category>AvailabilityHealth</Category>

  <OperationalStates>

    <OperationalState ID="Green" MonitorTypeStateID="ConditionFalse" HealthState="Success" />

    <OperationalState ID="Red" MonitorTypeStateID="ConditionTrue" HealthState="Error" />

  </OperationalStates>

  <Configuration>

    <LogFileDirectory>%SystemDrive%\LogFiles\Perflogs</LogFileDirectory>

    <LogFilePattern>TerminalServices*.csv</LogFilePattern>

    <LogIsUTF8>false</LogIsUTF8>

    <Threshold>4</Threshold>

    <Direction>greater</Direction>

    <NumSamples>2</NumSamples>

  </Configuration>

</UnitMonitor>

As it can be seen, this monitor uses the System.Performance.PerformanceLogBased.ConsecutiveSamplesThreshold monitor type, which does exactly what we want: It changes state whenever the value of the performance counter violates the configured threshold over a consecutive number of samples.

Configuration

LogFilePattern: the name of the file(s) containing the performance data. In this case, I've used the wild card character (*) to indicate that it should look for all files which start with “TerminalServices” and have a .csv extension. Later, we need to tell perfmon to log to a file with the same name format.

LogFileDirectory: where the monitor will look for the files matching the pattern.

LogIsUTF8: indicates whether the parsing algorithm should decode based on UTF-8 or not.

Threshold: the value above or under which this monitor will change state.

NumSamples: the number of consecutive samples that need to violate the threshold before the monitor changes state.

Collection Rule Definition (Storing counter data in the Ops DB)

The following rule collects the performance counters logged to files in the TerminalServices*.csv format (e.g. TerminalServices01.csv, TerminalServices02.csv), residing in the %SystemDrive%\LogFiles\Perflogs directory. It will then store the collected values in the Operational database as performance data. This data can then be viewed as usual, through performance views and dashboards. 

 <Rule ID="PerfLogBasedCollectionRule" Enabled="true" Target="SystemCenter!Microsoft.SystemCenter.Agent" ConfirmDelivery="false">

     <Category>Maintenance</Category>

     <DataSources>

       <DataSource ID="PerfLogRuleDS" TypeID="Performance!System.Performance.PerformanceLogDataProvider">

         <LogFileDirectory>%SystemDrive%\LogFiles\Perflogs</LogFileDirectory>

         <LogFilePattern>TerminalServices*.csv</LogFilePattern>

         <LogIsUTF8>false</LogIsUTF8>

         </DataSource>

       </DataSources>

       <WriteActions>

          <WriteAction ID="PerfLogRuleWA" TypeID="SystemCenter!Microsoft.SystemCenter.CollectPerformanceData"/>

       </WriteActions>

   </Rule>

Some Observations:

1. Only 1 performance counter (combination of performance object + counter name + instance) per file. The PerformanceLogDataProvider data source module does not support multiple counters. For example, logging PrivateBytes for all instances of the w3wp.exe process will not work. Just one instance per file.
2. The file should be in CSV format (by default, perfmon logs are in a binary format – that needs to be changed, as shown below)
3. Cyclic logging should not be used. I recommend starting a new file in a sequential manner, based on a size limit or time schedule. For instance, in this case we would have TerminalServices1.csv, TerminalServices2.csv, etc. All logged to the same directory.
4. If the health service is stopped and then started at a later time, parsing should resume from where it stopped at the time the service shut down.  When a computer comes back from maintenance mode however, collection will restart as new entries are logged (data produced while the agent was under maintenance is not collected).
5. You can define Alerts for the monitor via the UI console by going to this monitor's properties and setting the Alert configuration.
6. Although these workflows are targetting the Agent computers, you could define your own type (e.g. a certain app) and specify that as the target. Of course, you would have to create discovery rules to discover instances of the type.

Import the MP with the rules and monitors

Assemble an MP file with your final rules and monitors. One option here is to create a new MP by using the Create a Rule wizard, exporting that MP and then editing it to include your custom rules and monitors. Once ready, re-import your final MP.

Enabling PerfMon Logging

The last piece of work that needs to be done before data starts flowing is actually logging the counters to where Operations Manager is supposed to find them and in the format it is expecting. 

The steps involved are: 

1. Enable a new User Defined Data Collector Set in Windows Performance Monitor (perfmon). Choose the Create manually (Advanced) option and then check the Performance counter checkbox.

 
2. Add the desired counter to the set (only 1 counter should be added) and choose the Sample Interval.
  
  

3. Choose the root directory to where the file will be written, save and close the wizard, without starting the collector. This should be the same directory specified in the rule/monitor.

           
  
  
  4. Right-Click on the newly created Data Collector Set and select Properties.
  
  
  
  
  
  
  5. On the Directory tab, uncheck the Prefix subdirectory with computer name checkbox and make sure the Subdirectory name format textbox is empty. Apply those settings. On the Schedule and Stop Condition tabs in this window, you can optionally set a new file to be generated after the current one reaches a limit or based on a schedule.

 6. Open the Properties for the DataCollector file. In the Log format drop box, select Comma Separated.
 
        

            

7. Select the File tab and make sure the Log file name matches the pattern expected by Operations Manager (as specified in the configuration for LogFilePattern). Apply and close the properties window.

8. Start the data collector.

Other Monitor Types Available

There's pretty much a 1-1 matching between what can be done via standard performance data collection and the log-based one. Just like in the standard case, there are log-based monitors for:

• Self Tuning Thresholds
• 2-State Above
• 2-State Baselining
• 2-State Below
• 3-State Baselining
• Single Thresholds
• Average Threshold
• Consecutive Samples over Threshold
• Delta Threshold
• Simple Threshold
• Double Thresholds
• Double Threshold
• Delta Double Threshold
• Average Double Threshold

Refer to the Performance Library MP (System.Performance.Library) to look at their configuration.

Happy Log-Based Performance Counter Collection!