Matt Goedtel on Operations Management

Here is a question that I see come up from time to time that I thought deserved some focus.  Let's say you are looking to scale out your management group and install an additional management server, install a new gateway server, or maybe install the console locally (an exception of this scenario is the OpsMgr Reporting Server & Data Warehouse database).  You could be in a pinch becuase you don't have any new hardware to dedicate for this requirement, dealing with a DR scenario, etc.  While running through setup you notice that in the UI you cannot select a different installation location, the browse button is nowhere to be found, and it is likely defaulting to C:\Program Files\System Center Operations Manager 2007.  But you want to install it to a different volume, like E: or whatever. 

So then you scratch your head and think, "well I understand MSI packages and their command-line settings.  Let me pass the INSTALLDIR property and see if that works."  So you try and it still fails.  Why is this you wonder?

Most often times than not, it is due to the fact that the OpsMgr agent is already installed and has been for some time, in order to proactively monitor the system (following recommended practices for any server that is considered "live" or "production").  All the components of Operations Manager such as the console, management server and so forth have a dependency on the common components for the Agent Health Service (as defined in the MSI database).  In order to work around this, you would need to first:

1. Uninstall the agent from the system
2. Install the Operations Manager component
3. Reinstall the agent if not a management server. 

So just consider this when going through this exercise and ensure you document this in if you intend on taking an existing server monitored by OpsMgr to fill a role. 

I would like to provide a couple of templates I mentioned during my presentation yesterday that many of you may find of value.  In the .ZIP file are two tempaltes:

• Override Change Log spreadsheet
• Test Plan

I had a project plan template somewhere that helps in managing the testing of a management pack or the development of a new MP and the testing of that MP.  Sadly I cannot find it right now so I'll have to publish it tomorrow after I find it.

Remember, the templaets I am providing are to help you in developing your own standards that you and/or your team should be following as part of the Release Management process.  Hopefully you find them of value.

Tomorrow I'll continue my discussion regarding MP Lifecycle Management as a follow on to my post from last night.  Alas I am in need of a break this evening :)

Today I had the pleasure of presenting about System Center Operations Manager 2007 MP Lifecycle Management via TechNet Webcast.  After I completed the presentation I shortly realized I failed to mention some important facts/recommendations that I had notated yet overlooked due to time and a bit of nervousness.  I digress.  Anyway, here are a few things I would like to elaborate on or just highlight.

1. The tools to augment your "toolkit" with respect to managing overrides, reviewing overrides in your management group, etc. can be downloaded off of Boris Yanushpolsky's Blog - http://blogs.msdn.com/boris_yanushpolsky/default.aspx.  Those tools I mentioned were Override Explorer and Override Creator.
2. Another tool to augment your "toolkit" with respect to viewing the configuration of a management pack can be downloaded off of Boris Yanushpolsky's Blog - http://blogs.msdn.com/boris_yanushpolsky/default.aspx.  This tool is called MP Viewer.
3. Silect MP Studio provides many features with respect to viewing the configuration of a management pack, performing overrides, test and tune the MP by running the feature "Resultant Set of Alerts", opening a management pack and printing out the default rules and monitors that are enabled so the SME/SO (Subject Matter Expert/Service Owner) can understand the scope of what is monitored and what default thresholds may be defined.  The Resultant Set of Alerts feature is valuable when verifying the management pack as part of your test phase.  Also their MPStuidoLite tool should be considered if you don't decide to purchase the MPStuido product.  I should also note that the folks at Silect are also working towards ensuring that the tool compliments the ITIL processes with respect to Change, Release, and Config.  You can find out more about it here - http://www.silect.com/solutions/opsmgr_Sol/opsmgr_Sol_studio.html.
4. When conducting the pilot of your management pack, leverage the built-in reports Operations Manager - Microsoft Generic Report Library\Most Common Alerts and Microsoft ODR Report Library\Most Common Alerts.  These reports will help you in identifying the top offenders that may require additional tuning, adjusting the configuration of rule/monitor. or a logic error in a rule script.
5. For the pilot phase I recommended multi-homing the agent managed system in production against the QA management group.  This allows you to narrow the scope of your pilot without having to deal with overrides for controlling the discovery rules that would identify systems that match the criteria required for the MP to begin monitoring the application or service.  Multi-homing allows you to distribute the monitoring between one or more management groups and the processing of rules is managed independently.  Meaning if I have my production management group monitoring the server/service/application using x version of a management pack and in the QA management group I have version y of the same management pack, the processing is independent.  It is like having two agents running on the same system (even though with MOM/OpsMgr 2007, it is one agent and the configuration information in the Registry dictates it reports to two separate management groups).  The multi-homed agent processes the workflows from the different configuration groups independently and there is no conflict of rules.
6. The goal of the alert tuning process is to minimize the alert "noise" or the barrage of false alerts that have been generated and overshadow the relevant alerts that are actionable.  Sometimes the source of alert noise can be attributed to false positives, false negatives, and multiple alerts with the same root cause.  This is an on-going process and may not be resolved in a short period of time.  For further guidance, please review the MOM 2005 Alert Tuning Solution Accelerator found here - http://www.microsoft.com/technet/solutionaccelerators/cits/mo/smc/sts05.mspx.  The prescriptive guidance provided in this SA is still very relevant and helpful, so please take a moment and check it out.
7. When developing a custom management pack to monitor an in-house developed application or service, one recommendation I can make is consider disabling the performance collection monitors or rules by default.  This approach allows you to enable them via override one at a time to verify their configuration and avoid impacting the agent, management server, and the SQL Servers hosting the OperationsManager or the OperationsManagerDW databases (if you are leveraging the performance data for reports). 

I think this is a good start and I'll continue this topic tomorrow.  So stay tuned and for those of you who attended today's session, I hope you found it of value and I thank you for attending.  For those of you who were unable to, it will be available for on-demand viewing shortly, so you can access it here - https://www106.livemeeting.com/cc/mseventsbmo/view?id=1032379751&role=attend&pw=4611BEDB.  If there is a particular topic you would like me to go into greater detail on, please let me know and I'll will be happy to expand and post it here on my blog.

Thanks!

So yesterday a question was presented internally between my colleagues and I with respect to a situation a colleague was facing.  Suppose by chance you are installing the first management server, which happens to be the Root Management Server, in the management group and you install it to the %SystemDrive% volume.  Later you realize that as time goes on, the free space on the system volume decreases and you 1) wonder why and 2) discover it is due to Operations Manager.

After doing some further diagnosis, you identify the Health Service ESE database as the culprit and therefore, decide you want to move it to a different volume with ample free space to support it for the long haul.  It is installed by default in %ProgramFiles%$\System Center Operations Manager 2007\Health Service State.  This procedure is not currently documented in the Operations Manager Operations Guide, On-line Help, Knowledge Base, etc.  However, this solution I am about to provide you with was presented to the product group and they acknowledged it is approved and supported.  In order to move the Health Service State folder structure to a different volume, perform the following steps:

1. Stop the following services - OpsMgr Health Service, OpsMgr Config Service, OpsMgr SDK Service.
2. Create on the destination volume the directory you want to move the Health Service State directory to.  I recommend maintaining consistency and naming it - "Program Files\System Center Operations Manager 2007\Health Service State."
3. Move the source "Program Files\System Center Operations Manager 2007\Health Service State" directory to the destination folder you created in Step 2.
4. Verify the ACL permissions are at a minimum, set to Administrators (Full Control) and System (Full Control) on the directory and sub-directories.
5. Warning:  Editing the Registry incorrectly can result in your operating system or applications failing to function normally.  Be sure you backup the system before you proceed with this step.  Open the Registry Editor and under HKLM\System\CurrentControlSet\Services\HealthService\Parameters key modify the following value "State Directory" with the destination path you have moved the source to. 
6. Close the Registry Editor.
7. Restart the following services - OpsMgr SDK Service, OpsMgr Config Service, and OpsMgr Health Service.
8. Open the Event Log and in the Operations Manager Event View, verify there are no errors from the source Health Service, Health Service Modules, and Health Service ESE Store.

These steps are also applicable for the Agent, management server, and Gateway Server.

On June 11th at 1 PM, I'll be hosting a TechNet Webcast on Operations Manager 2007 Management Pack Lifecycle Management. 

Join this webcast to learn how people, process, and technology can help you manage the life cycle of developing or tuning management packs in support of Microsoft System Center Operations Manager 2007. In this webcast, we discuss the consideration of roles and responsibilities, the approach to managing overrides, developing custom management packs, version control, and much more. We also discuss how Information Technology Infrastructure Library (ITIL) processes play an important part in ensuring a consistent, repeatable approach to managing the life cycle of management packs. 

Here is the link to register - http://www.microsoft.com/events/EventDetails.aspx?CMTYSvcSource=MSCOMMedia&Params=%7eCMTYDataSvcParams%5e%7earg+Name%3d%22ID%22+Value%3d%221032379751%22%2f%5e%7earg+Name%3d%22ProviderID%22+Value%3d%22A6B43178-497C-4225-BA42-DF595171F04C%22%2f%5e%7earg+Name%3d%22lang%22+Value%3d%22en%22%2f%5e%7earg+Name%3d%22cr%22+Value%3d%22US%22%2f%5e%7esParams%5e%7e%2fsParams%5e%7e%2fCMTYDataSvcParams%5e

 I look forward to presenting on this topic and hope that you can attend.

When you launch the Operations Console on your workstation or even on the management server (or RMS for that matter), the data from the Operations Manager database is cached locally in the file "momcache.mdb." By default, the console will poll the SDK Service on the RMS every 15 seconds to update the local cache.  Depending on the number of agent managed systems that are in the management group, potential size of the database will be (as indicated by the Product Group): 

• 100 server deployment = 300 MB
• 1000 server deployment = 600 MB
• 5000 server deployment = 1 GB

The database file is located in %USERPROFILE%\Local Settings\MIcrosoft\Microsoft.Mom.UI.Console. 

For my small lab environment with only 12 servers being actively monitored by Operations Manager with many core and custom management packs, the file size is 34 MB (as an example).

So if you are intending or are already hosting the console on a Terminal Server or Citrix Server, you will need to factor this in to ensure you don't impact the volume supporting other shared applications. 

So I'll be heading out to Las Vegas tomorrow to attend MMS 2008 and assist in staffing the event.  It will be great to get out there and help provide my experience and guidance to those attending and I look forward to meeting new people. If you are looking to place a face with the name, I'll be proctoring the Hands-on Labs for 4/28 between 1:30 and 4:30 and then on 4/29 between 10:15 and 1:00.

Hope to see you there and you leave feeling well served with solid information!

In continuation of my prior blog post a couple of weeks ago (see - http://blogs.technet.com/mgoedtel/archive/2008/03/18/verifying-mp-workflow-on-agent.aspx), I  realized this evening there is another method that is much easier and more straight forward.  While Walter Chomak recommended checking the Operations Manager Event Log on the agent managed system, by looking for Event ID 1201 specific to the Management Pack itself (See here to read Walter's Blog Post - http://wchomak.spaces.live.com/blog/cns!F56EFE25599555EC!896.entry) most customers that I engage with are looking for a more centralized approach. 

My initial thought was that the Event Collection Rule - Collect Health Service Configuration Updated Events from the System Center Core Monitoring MP would have captured this particular event, but after inspecting it I was wrong.  It only looks for Event ID 1210 from the Health Service and not any of the ones we are more interested in.  I uncovered this particular Event Collection rule by reviewing the Dashboard View  - Operations Manager\Health Service Configuration\Config Update Events and reviewing the "Received and Activated New Config." 

Since this Event Collection Rule does not look for Event ID 1201, I figured I would create a custom rule and view to be able to monitor this from the Operations Console.  So in order to do this, perform the following:

1. Create an Event Collection Rule and call it "Collect Health Service MP Update Events" with a Rule Category of "EventCollection" and the Rule target being "Health Service."
2. The Log name is "Operations Manager" and the Expression is Event Source "HealthService" with EventID 1201.
3. Save this in a custom Management Pack and not the Default Management Pack.
4. Under the Monitoring view, create a Event View and call it "Config Update Events (Agent) - Management Packs" or something logical that meets your needs.
5. The Criteria for the view is the following:  "with a specific event number 1201" and from a specific source "HealthService."  The target should be "Agent" under the Show data related to drop down list.

Once you carry out those steps, from the console you can view when an agent receives a new Management Pack.

The Exchange Server 2003 Management Pack for Operations Manager 2007 has been updated (version 6.0.6278.5) and is now available for download - http://www.microsoft.com/downloads/details.aspx?FamilyId=9FF454F4-6D34-4FB9-9E0B-F5B68C6EDC4F&displaylang=en&displaylang=en

 The minor update includes:

• Fixed an issue with OWA monitoring using custom URLs so that the Management Pack now looks for the CustomUrls registry key instead of the CustomOwaUrls registry key (consistent with previous versions of the Management Pack)
• Fixed an issue with data collection rules for mailbox or public folder size in MB. On monitored servers running Exchange 2003 and using regional settings that are not English (United States) an issue can occur where the size of the mailbox or public folder is reported incorrectly.  This will also affect the associated reports.

Today we released the Exchange 2007 SP1 management pack for MOM 2005 to MS downloads:

http://www.microsoft.com/downloads/details.aspx?FamilyID=30EEBC7C-A35A-41AE-9CD1-2047847FDE85&displaylang=en

In this MP, you’ll find:

·         Renamed some event rules to better reflect their purpose

·         Improved some report descriptions

·         Updated for performance counter name changes

·         Fixed PowerShell issue when one or more necessary parameters are not passed to a cmdlet

·         Updated many reports to use a 90%-100% scale rather than dynamic scale to gain visual fidelity in that range. Affected reports are as follows:

o   ActiveSync Internal Service Availability

o   Mailbox Service Availability

o   Mailflow Local Service Availability

o   Mailflow Remote Service Availability

o   Outlook Web Access External Service Availability

o   Outlook Web Access Internal Service Availability

o   Unified Messaging Local Fax Service Availability

o   Unified Messaging Local Voice Service Availability

o   Unified Messaging Remote Voice Service Availability

·         Added/Modified Event Rules:

o   Added: Transport message rejection because of lack of disk space

o   Added: Transport message rejection because of memory consumption exceeding the configured threshold

o   Added: 10 error level event rules for Active Directory directory server Rights Management Server (RMS) in the Bridgehead Agents Messaging Policies health manifest

o   Added: 3 error level event rules for the Transport receive connector in the Bridgehead Transport common health manifest

o   Modified: Escalated “The STARTTLS certificate will expire soon” from warning to error alert

·         Removed many miscellaneous outdated event rules

·         Enabled collection of performance data for reporting in Test-WebServiceConnectivity ScriptResponseRule

·         Added support for Test-ReplicationHealth cmdlet; removed numerous event rules made redundant by this addition

·         Added support for Test-POPConnectivity cmdlet execution through the management pack and monitoring service

·         Added support for Test-IMAPConnectivity cmdlet execution through the management pack and monitoring service

·         Changed all outdated references from “Bridgehead” to “Hub Transport”

·         Added ‘From’ and ‘MediaSecured’ property support for Test-UMConnectivity for the monitoring service

·         Added ‘LightMode’ property support for Test-ActiveSyncConnectivity, Test-WebServicesConnectivity and Test-OWAConnectivity

·         Added ‘LightMode’ property support for Test-POPConnectivity and Test-IMAPConnectivity

·         Updated all reports to use provided client language for localized dates

Just posted is an updated version of the Windows 2003 Cluster Management Pack. This update resolves several customer initiated issues. This update also includes some changes in order to prepare for the Windows 2008 Cluster MP.

Here is a link to the download: http://www.microsoft.com/downloads/details.aspx?FamilyId=AC7F42F5-33E9-453D-A923-171C8E1E8E55&displaylang=en&displaylang=en

Due to the nature of the changes you will need to first uninstall the previous version of the cluster management pack. The steps are documented in the MP Guide which is part of the download package. The MP guide also has a complete list of changes.

So I came across this a week or so ago but forgot to install and play with it until today.  PowerGUI is a freeware product from Quest that provides an IDE for working with PowerShell on your systems, and has integration with Operations Manager 2007, in addition to Active Directory and Exchange Server 2007.  I just started to poke around with it on my Management Server, and it is actually pretty cool because it can help you in learning how to leverage the power of PowerShell integration in managing your Operations Manager environment.  Coding in PowerShell is one area I have not been able to devote enough time to, but I see how this tool can somewhat reduce that learning curve.

It is a bit basic, but it's a start and the developer is looking to provide additional features in the near future. 

You can learn more about it and download it from here - http://www.powergui.org/index.jspa

So I have finally found some free time to devote to updating my blog this evening.  Today as I was giving a presentation at my customer regarding Management Pack Lifecycle Management, they approached me with some good questions on the topic.  Specifically around "how do I verify 1) that the MP imported successfully, and 2) that once the Management Pack is imported, the agent actually received the rules and is running them?" 

The first question is pretty straight foward, the import wizard will inform you of success or failure immediately.  If that is not enough to satisfy the most anal individuals, then if it imported successfully go into the Operations Console and under the Administration view, select the Management Pack node and verify the MP is listed.  And if that is not enough, then run the pre-canned report "Microsoft ODR Report Library\Management Packs."   That will triple verify the MP is installed in the MG.  Now onto the second part, validation of agent recieving rules which are:

1. Run the Effective Configuration Viewer Reskit tool for Operations Manager 2007, which can be downloaded from here - http://www.microsoft.com/downloads/details.aspx?FamilyId=A9DB4DCA-6716-478D-89B9-42F27EBC76A8&displaylang=en.  Once you install it on your MS, RMS, or desktop with the Operations Console, you can execute it and select an object, such as a computer, and then view a list of monitors and rules.  In my testing of this tool thus far, you are not going to see the same results as you would if you ran the task I highlight in option 2.  I am still playing with this to understand why this is, so bear with me because I just started to sink my teeth into this Reskit tool. 
2. From the Operations Console, select the Monitoring view and target the Agent Health Service (this can be accomplished a couple of different ways, but the default OOBE is expand Operations Manager\Agent node in the left pane, and select the Active Alerts dashboard view).  Highlight an agent and select the Task - Show Running Rules and Monitors for this Health Service.  When you execute it and it displays the results, take note in the Task Output the tag "Count" as this will inform you as to how many are actually running against the agent.  This is a bit crude, but effective as a "quick and dirty" approach.
3. From the Command Shell, run a script provided by my colleague Boris Yanushpolsky that retrieves rules and monitors for a specific target, found here - http://blogs.msdn.com/boris_yanushpolsky/archive/2007/11/21/retrieving-rules-and-monitors-targeted-to-a-particular-class-target.aspx.

I am actually thinking of seeing if I can create a Report that would provide you with this information also.  If  can find the time in the very near future, I just may see if I can knock it out.

The Operations Manager 2007 Reporting Authoring Guide is now available for download - http://download.microsoft.com/download/7/4/d/74deff5e-449f-4a6b-91dd-ffbc117869a2/OpsMgr2007_RprtGuide.doc.

This document provides an in-depth understanding of the reporting feature, such as:

• Working with published and linked reports
• Developing custom reports
• Schema of the Microsoft.SystemCenter.DataWarehouse.Report.Library.xml (Reporting Management Pack)
• Scheduling reports (example documented is for e-mail) but now with SP1 we support publishing to Microsoft Office SharePoint Server.

and much more.

Here is a link to TechNet that outlines the key improvements and features included in Operations Manager 2007 Service Pack 1:  http://technet.microsoft.com/en-us/library/bb821996.aspx