Kevin Holman's OpsMgr Blog : exchange

Updated Exchange 2003 MP released version 6.0.6387.0

kevinhol — Fri, 22 Aug 2008 08:12:33 GMT

http://www.microsoft.com/technet/prodtechnol/scp/opsmgr07.aspx?SCPProdID=3

This is a pretty major update - and it looks to address the biggest issues with the Exchange 2003 MP.... most notably reducing Self Tuning Threshold noise, SSL enabled alerts, Report fixes and added report descriptions to make them easier to run, fixes to Mail flow, etc...

Note: You will need to take a look at the overrides you have put in place for the existing MP. If you have already addressed some of these issues fixed below - you may want to consider deleting your overrides that deal with these updates, and re-tune if necessary.

The following list includes all of the changes contained in this update of the Exchange Server 2003 Management Pack.

· Removed Exchange front end service state monitors that would cause duplicate alerts to be generated.

· Updated the KnowledgeBase for Performance Collection Rules for Mail flow receiver, Exchange Mail Flow Monitor (Sender Part) and Exchange Mail Flow Monitor (Receiver part) to clarify how these should be configured.

· Updated the alert description for ExBPA alerts to direct the administrator towards the ExBPA events view since more than one issue may have been found, while only one alert is generated.

· Changed the scripts for the Exchange Mail Flow Monitor (Receiver Part) so that setting the number of Max missed runs override now works, enabling users to set a threshold for when mailflow alerting should occur.

· Documented supported overrides for Mail Flow monitoring in the Exchange Server 2003 Management Pack guide. For more information, see the “How to Configure Mail Flow Monitoring” section of this guide.

· Changed the top level display name of the folder in the Operations Manager 2007 console to be “Exchange Server 2003”. Note that all Operations Manager 2007 console users will need to clear the console cache for this to display correctly.

· Changed the Collect_Message_Tracking_statistics script so that it generates an alert if it can’t access the message tracking logs.

· Added logic to collect event 9950 from the Check_Mailbox_Store_Status.Monitor, enabling users to find which store was dismounted.

· Removed the override from the “SSL Should be Enabled” monitor specifying the list of servers to be excluded from the Secure Sockets Layer (SSL) check. Added documentation in the Exchange Server 2003 Management Pack Guide for how to handle this scenario. For more information, see the “How to Configure SSL Monitoring” section of this guide.

· The sensitivity of all Self Tuning Threshold rules and monitors in the management pack was lowered, increasing the threshold at which the monitors alert.

For this management pack version, the following Self Tuning Threshold monitors have been disabled and replaced with static “consecutive samples over threshold” monitors:

· MSExchangeIS\RPC Averaged Latency

· MSExchangeIS Mailbox\Send Queue Size

· SMTP Server\Remote Retry Queue Length

· SMTP Server\Local Queue Length

· SMTP NTFS Store Driver\Messages in the queue directory

· MSExchangeIS Transport Driver\TempTable Current

· SMTP Server\Remote Queue Length

Note: The Self Tuning Threshold monitors above are still in the management pack, but disabled by default.

For more information, see the “How to Configure Self Tuning Threshold Monitors” section of this guide.

· Updated several display strings and alert messages in the management pack.

· Removed the reference to the Backwards Compatibility Management Pack. Removed the Exchange Server Configuration Report, as this was dependent on collecting Backwards Compatibility Management Pack properties.

· Updated the Active Client Logons by day report to display the correct data.

· Updated the Exchange Information Store Usage report to display the correct data

· Updated the Exchange Message Transfer Agent (MTA) Usage report to display the correct data.

· Updated the MTA Work Queue Length by day report to display the correct data.

· Updated the Top 100 public folders by message count report to display the correct data.

· Updated a number of scripts that collect data mainly for reporting purposes to correctly collect data if the managed server running Exchange uses locales other than US English.

· Updated the monitor checking the Extensible Storage Engine (ESE) Log CheckPoint Depth so that it collects the correct counter and can alert appropriately.

· Updated the KnowledgeBase for the Outlook Web Access Logon Failure performance collection rule to clarify its behavior and available overrides.

· Updated the Disk Space Monitor to generate warning or error alerts depending on the condition, also changed it to a 3-state monitor. For documentation of this monitor, see the “How to Configure Disk Space Monitoring” section of this guide.

· Updated the Exchange Topology Discovery script to generate less data. Removed discovery of the AdminGroup and RoutingGroup instances, which were not used in the topology diagram. Added an override to the Topology Discovery script to control whether Routing Group connectors should be discovered or not. The default value for this override is “true”.

· Updated the Exchange Server 2003 Management Pack Guide with instructions for monitoring the MTA service on clustered servers running Exchange. For more information, see the “How to Configure Exchange Server Clusters to be Monitored” section of this guide.

· Updated report descriptions in the management pack.

· Updated the management pack guide to new format.

The Exchange Server Configuration report doesn't return any data

kevinhol — Tue, 15 Jul 2008 21:34:00 GMT

I figured out why - and instead of just posting about the workaround... I thought I'd take a moment and post on the process I used to determine the how and the why:

When you run the "Exchange Configuration Report" per the guide, you are supposed to add objects of type "Exchange 2003 Role". Well, if you do this - the reports will never return any data.

So - to try and look at the SQL code... I went to the RDL file in the reporting website.... http://reportingservername/reports

I browsed over to the report - which I found at: Home > Microsoft.Exchange.Server.2003.Monitoring > Report.Exchange.ExchangeServerConfiguration. Select the "Properties" link.... and normally there would be an "Edit" button here... where I could download the RDL (report definition) file.... but instead - I see this is a linked report:

Aha! It is linked to: /Microsoft.SystemCenter.DataWarehouse.Report.Library/Microsoft.SystemCenter.DataWarehouse.Report.CustomConfiguration

Microsoft.SystemCenter.DataWarehouse.Report.Library is the Generic Report library.... so off I go looking for a generic report.

Found it:

So - I simply run this report.... and immediately I can see something different... I am now offered a new parameter area... called Report Fields. So, I add the Exchange 2003 Roles instance to the report as an object.... and now I can pick and choose from a whole list of discovered attribute data.... and customize the order I want it to show up in:

Here I am running the report for the same object... and now I get to pick the cool things I want to see:

Now.... if I want to run this report in the future... and add more servers, etc... I simply save it to my favorites. Then - on future runs of the report, I can drop the parameter header back down and make any changes I want.

Exchange 2007 Management pack updated 6-22-2008 version 6.0.6278.12

kevinhol — Mon, 23 Jun 2008 20:11:36 GMT

The updated MP is released!

Now supports Exchange 2007 SP1 and a bunch of other fixes.

Get it from the catalog:

http://www.microsoft.com/technet/prodtechnol/scp/opsmgr07.aspx

There are three hotfixes you should apply as a pre-requisite....

Install the agent update specified in Knowledge Base article 950853 on all Exchange-based servers managed by Operations Manager before importing the Exchange Server 2007 Management Pack. This update addresses an agent memory leak issue.

· Install the update specified in Knowledge Base article 951979. This update contains an updated agent restart script and fixes issues with cluster discovery.

· If you are monitoring Exchange Server 2007 clusters, ensure that you have installed the agent update specified in Knowledge Base article 951380 on all Exchange Server 2007 cluster nodes managed by Operations Manager. This update addresses an issue with cluster discovery.

The May 2008 update to the October 2007 version of the Exchange Server 2007 Management Pack, version 6.0.6278.12, includes the following changes:

· All updates included in the 08.01.0240.001 version of the Exchange Server 2007 SP1 Management Pack for Microsoft Operations Manager 2005, except updates relating to reports.

· Overrides were documented in the Management Pack Guide for the LDAP Search Time and Failure DSNs Total rules and monitors.

· The management pack was updated to support the renamed performance counters in Exchange Server 2007 Service Pack 1. Performance counters for the Database object were renamed to MSExchange Database.

· The management pack was updated to support non-default names of the Reporting data warehouse.

· The OWA Connectivity performance view was updated to show performance data.

· Fixed an issue where cluster virtual servers where discovered as type Microsoft Exchange 2007 Mailbox Servers Installation.

· The Management Pack was updated so that alerts are correctly generated for events logged by physical cluster nodes in an Exchange Server 2007 cluster.

· Fixed an issue where the Exchange cluster virtual servers were discovered as type Microsoft Exchange 2007 Mailbox Servers – Physical Computers Installation.

· The Microsoft_Exchange_Server_Exchange_2007_Mailbox_Replication_Health_Test_ReplicationHealth_Events view was updated to target the Microsoft.Exchange.2007.Microsoft_Exchange_2007_Mailbox_Servers___Physical_Computers_ComputerGroup.

· The reports were updated to support non-US locales on the Reporting Server.

· The date/time picker was added to the reports, allowing for more flexibility in report scheduling.

· The Failure and Delay DSNs Total monitors were updated to look for deltas. Previously, the monitors measured the averages for the sampling interval.

· Fixed an issue where the cluster virtual servers were discovered as type Microsoft Exchange 2007 All Servers Installation.

Exchange 2007 Management Pack - the good, the bad, and the ugly

kevinhol — Sat, 22 Mar 2008 00:09:00 GMT

UPDATE 6-23-08

The updated MP has been released, along with three hotfixes that addresses everything in this blog post. Please install the hotfixes (listed below) and download the latest version of the Exchange 2007 MP.

The current Exchange 2007 Management pack (6.0.5000.0) is designed to support Exchange 2007 RTM. It is a conversion MP, from the MOM 2005 MP.

There is an updated MP coming out soon (possibly May timeframe) which will support Exchange 2007 SP1.

This is good... however, it does present some new challenges. With Exchange 2007 SP1 - the service pack renames some perfmon counters.... which will break the MP capability if you dont upgrade it to the newer version that supports SP1. If you are already running Exchange 2007 SP1 - then you are already affected.

Likewise.... if your environment is running Exchange 2007 RTM, and you upgrade to the newer MP version.... you will have the same mismatch. Here are some details:

In the graphic below - we can see the perfmon counter.... MsExchange Database. This is the renamed counter seen in SP1 of Exchange 2007.

The issue becomes - that the management pack is looking for the Exchange 2007 RTM named object - which is "Database":

If you need to workaround this.... and feel these counters are critical - then simply disable the rules/monitors in your default Exchange MP via override, and then create new rules/monitors to grab the correct counters.

The following rules target the "Microsoft Exchange 2007 Mailbox Servers Installation" class

Collect Database: I/O Database Reads Average Latency (Report Collection)
Collect Database: I/O Database Reads/sec (Report Collection)
Collect Database: I/O Database Writes Average Latency (Report Collection)
Collect Database: I/O Database Writes/sec (Report Collection)
Collect Database: I/O Log Writes Average Latency (Report Collection)
Collect Database: Version buckets allocated.
Information Store: Version buckets allocated - Red(>2000)
Information Store: Version buckets allocated - Yellow(>1800)

The following Monitors target the "Ex. Mailbox" class

Information Store: Version buckets allocated - Red(>2000)
Information Store: Version buckets allocated - Yellow(>1800)

Now.... for part II:

The Exchange 2007 MP may expose a memory leak of Monitoringhost.exe private bytes. How you will know if you are affected.... is you might find that your Exchange 2007 servers Healthservice is restarting on a regular basis. If you watch the Monitoringhost.exe process private bytes counter, it will continually grow. There is a rule which watches this counter, and restarts the Healthservice when it hits a threshold.

Here is an example of the event you will see in the OpsMgr event log on the agent:

-----------------------------

Event Type: Warning

Event Source: Health Service Script

Event Category: None

Event ID: 6026

Date: 3/21/2008

Time: 11:30:25 PM

User: N/A

Computer: EX07A

Description:

RestartHealthService.js : Restarting Health Service. Monitoring Host exceeded Process\Private Bytes threshhold.

-------------------------------

The Healthservice might restart fine and keep going - or the restart might fail.

If you create a rule to collect the perf counter for Process\Private bytes for Monitoringhost.exe.... you might see something like this in a performance view:

What you can see if that the process consumes about 60MB at startup.... leaks up to 150MB, and then the restart kicks in. The gap toward the end is where the automated script failed to restart the service correctly.... and I received a heartbeat failure from this agent. I had to start it back up manually.

So - from this - we see three issues:

1. The process is leaking memory, rapidly, and hitting an internal threshold.

2. The restart script RestartHealthService.js is not reliable.... on the Exchange 2007 agents, there are a huge number of rules/monitors loaded, and this is likely the cause.

3. The threshold to restart is incorrect. If you look at the rule for this - the threshold is set to 100MB, and we dont restart the service until 150MB. No matter what threshold we set, it appears to be a factor of 1.5x.

Also - we recommend enabling the automated restart recovery to the heartbeat failure alert.... if the service restart script fails to start the service, this recovery will start the service within minutes and get the agent back up and running on E2K07 servers.

UPDATE 06-23-2008

The Memory leak hotfix is available: You should apply this to all SP1 deployments of Opsmgr. I have seen this leak rapidly exposed on Exchange 2007 servers, and SCCM servers.

http://support.microsoft.com/kb/950853/en-us

The service restart hotfix is available: You should apply this to all SP1 deployments of OpsMgr, as it is simply updated core MP's which also contain a large list of other fixes:

http://support.microsoft.com/kb/951979/en-us

The cluster discovery hotfix is available: You should apply this to all SP1 environments that monitor Microsoft clusters.

http://support.microsoft.com/kb/951380/en-us

The updated Exchange 2007 management pack (6.0.6278.12) is available which also includes fixes for the database counters. Please download it from the catalog:

http://www.microsoft.com/technet/prodtechnol/scp/opsmgr07.aspx