Kevin Holman's OpsMgr Blog

Exchange 2007 Management Pack - the good, the bad, and the ugly

UPDATE 6-23-08

The updated MP has been released, along with three hotfixes that addresses everything in this blog post.  Please install the hotfixes (listed below) and download the latest version of the Exchange 2007 MP.

 

 

The current Exchange 2007 Management pack (6.0.5000.0) is designed to support Exchange 2007 RTM.  It is a conversion MP, from the MOM 2005 MP.

There is an updated MP coming out soon (possibly May timeframe) which will support Exchange 2007 SP1.

This is good... however, it does present some new challenges.  With Exchange 2007 SP1 - the service pack renames some perfmon counters.... which will break the MP capability if you dont upgrade it to the newer version that supports SP1.  If you are already running Exchange 2007 SP1 - then you are already affected.

Likewise.... if your environment is running Exchange 2007 RTM, and you upgrade to the newer MP version.... you will have the same mismatch.  Here are some details:

In the graphic below - we can see the perfmon counter.... MsExchange Database.  This is the renamed counter seen in SP1 of Exchange 2007.

The issue becomes - that the management pack is looking for the Exchange 2007 RTM named object - which is "Database":

If you need to workaround this.... and feel these counters are critical - then simply disable the rules/monitors in your default Exchange MP via override, and then create new rules/monitors to grab the correct counters.

The following rules target the "Microsoft Exchange 2007 Mailbox Servers Installation" class

Collect Database: I/O Database Reads Average Latency (Report Collection)
Collect Database: I/O Database Reads/sec (Report Collection)
Collect Database: I/O Database Writes Average Latency (Report Collection)
Collect Database: I/O Database Writes/sec (Report Collection)
Collect Database: I/O Log Writes Average Latency (Report Collection)
Collect Database: Version buckets allocated.
Information Store: Version buckets allocated - Red(>2000)
Information Store: Version buckets allocated - Yellow(>1800)

The following Monitors target the "Ex. Mailbox" class

Information Store: Version buckets allocated - Red(>2000)
Information Store: Version buckets allocated - Yellow(>1800)

Now.... for part II:

The Exchange 2007 MP may expose a memory leak of Monitoringhost.exe private bytes.  How you will know if you are affected.... is you might find that your Exchange 2007 servers Healthservice is restarting on a regular basis.  If you watch the Monitoringhost.exe process private bytes counter, it will continually grow.  There is a rule which watches this counter, and restarts the Healthservice when it hits a threshold.

Here is an example of the event you will see in the OpsMgr event log on the agent:

-----------------------------

Event Type:      Warning

Event Source:   Health Service Script

Event Category:            None

Event ID:          6026

Date:                3/21/2008

Time:                11:30:25 PM

User:                N/A

Computer:         EX07A

Description:

RestartHealthService.js : Restarting Health Service. Monitoring Host exceeded Process\Private Bytes threshhold.

-------------------------------

The Healthservice might restart fine and keep going - or the restart might fail.

If you create a rule to collect the perf counter for Process\Private bytes for Monitoringhost.exe.... you might see something like this in a performance view:

What you can see if that the process consumes about 60MB at startup.... leaks up to 150MB, and then the restart kicks in.  The gap toward the end is where the automated script failed to restart the service correctly.... and I received a heartbeat failure from this agent.  I had to start it back up manually.

So - from this - we see three issues:

1.  The process is leaking memory, rapidly, and hitting an internal threshold.

2.  The restart script RestartHealthService.js is not reliable.... on the Exchange 2007 agents, there are a huge number of rules/monitors loaded, and this is likely the cause.

3.  The threshold to restart is incorrect.  If you look at the rule for this - the threshold is set to 100MB, and we dont restart the service until 150MB.  No matter what threshold we set, it appears to be a factor of 1.5x.

 

Also - we recommend enabling the automated restart recovery to the heartbeat failure alert.... if the service restart script fails to start the service, this recovery will start the service within minutes and get the agent back up and running on E2K07 servers.

 

 

 

 

UPDATE 06-23-2008

The Memory leak hotfix is available:  You should apply this to all SP1 deployments of Opsmgr.  I have seen this leak rapidly exposed on Exchange 2007 servers, and SCCM servers.

http://support.microsoft.com/kb/950853/en-us

 

The service restart hotfix is available:  You should apply this to all SP1 deployments of OpsMgr, as it is simply updated core MP's which also contain a large list of other fixes:

http://support.microsoft.com/kb/951979/en-us

 

The cluster discovery hotfix is available:  You should apply this to all SP1 environments that monitor Microsoft clusters.

http://support.microsoft.com/kb/951380/en-us

 

The updated Exchange 2007 management pack (6.0.6278.12) is available which also includes fixes for the database counters.  Please download it from the catalog:

http://www.microsoft.com/technet/prodtechnol/scp/opsmgr07.aspx