In efforts to help streamline performance data collection on Exchange 2007/Exchange 2010 servers, I have created a powershell script that will automate the data collection for you. One of the nice features of the script is that you do not have to tell the script what roles are installed as it will automatically detect what is installed locally on the server, thus adding the appropriate counters for you. Previously, you had to manually select an XML file from here for Exchange 2007 servers and here for Exchange 2010 servers and then import it in to the performance console.

I’ve seen a lot of cases that use the previous Perfwiz utility, but unfortunately, this was originally designed to collect data for Exchange 2003 servers and was never updated to support the later versions of Exchange. This older version of Perfwiz should never be used to troubleshoot performance issues for versions later than Exchange 2003 as the pertinent counters are not being collected to accurately troubleshoot a performance issue.

During the development phase of this script, it was found that starting with Windows 2003 x64 that the log roll mechanism no longer worked properly and stopped once the maximum log file size was hit. Even though this worked previously in on Windows 2003 x86 versions, something changed on the 64-bit platform which prevented this from working. This problem is also inherent in the Windows 2008 operating system, but eventually was resolved in Windows 2008 R2. The script works around all of these issues to help you collect the right data at the right time by doing the following:

• If Windows 2003 x64 and –circular switch not specified, then roll log to next log file once maxsize is reached or duration time is hit, whichever one is first.
• If Windows 2008 RTM/SP1/SP2 and –circular switch not specified, then roll log every 4 hours. If Interval is set to less than 30 seconds, then roll log every hour.

IMPORTANT: To help save on the disk space consumed to write these log files out, the *default duration* is set to 8 hours. This time duration should be enough to capture most performance cases during the day, but if longer durations are needed, then refer to the switches listed in the table below to help set the desired configuration for your needs.

Listed below are the switches that can be used with this script at the time of this posting. New switches will be added as time goes on. These switches should help allow you to collect the right data at the right time and also allows the flexibility to set the appropriate settings.

Parameter	Description
-help or -?	Provides help regarding the overall usage of the script
-circular	Turns on circular logging to save on disk space. Negates default duration of 8 hours
-delete	Deletes the currently running Perfwiz data collection
-duration	Specifies the overall duration of the data collection. If omitted, the default value is (08:00:00) or 8 hours
-EseExtendedOn	Enables Extended ESE performance counters
-EseExtendedOff	Disables Extended ESE performance counters
-filepath	Sets the directory location of where the blg file will be stored
-full	Defines a counter set that includes all Counters/instances
-interval	Specifies the interval time between data samples. If omitted, the default value is (00:00:30) or 30 seconds
-maxsize	Specifies the maximum size of blg file in MB. If omitted, the default value is 512
-query	Queries configuration information of previously created Exchange_Perfwiz Data Collector
-start	Starts Exchange_Perfwiz data collection
-stop	Stops the currently running Perfwiz data collection
-StoreExtendedOn	Enables Extended Store performance counters
-StoreExtendedOff	Disables Extended Store performance counters
-threads	Specifies whether threads will be added to the data collection. If omitted, threads counters will not be added to the collection
-webhelp	Launches web help for script

For additional information, you can check out the website that includes the latest 1.3 version download at http://code.msdn.microsoft.com/ExPerfwiz.

If you have an issue with this script or have a feature suggestion, use the Discussions/Issue Tracker tabs on the Experfwiz page listed above. There are also additional examples of how to run the script with additional switches on the site.

Enjoy!!

Mike

NOTE: This version of Perfwiz has been replaced by a newly written script that is talked about in http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange-2007-2010-performance-data-collection-script.aspx

1. Download the appropriate version of Perfwiz for your server
   
   How to download
   To download these XML files to your computer, right click the file of your choice, select Save Target As... , and then save it to a directory location of your choice on your Exchange Server
   
   Role Based
   Use these as a high level look in to how the server is performing and if you need to branch out with more counters, use the Full Counter/Instance set below.
   
   Exchange_2010_Perfwiz-MBX.xml
   Other roles coming soon....
   
   All Counters/All Instances
   Use this counter set at your own discretion as this could potentially cause performance degradation on your server trying to log this amount of counters.
   
   Exchange_2010_Perfwiz-Full.xml
2. Open Performance Monitor
3. Expand Reliability and Performance and then expand Data Collector Sets
4. Right click User Defined, Select New, and then Data Collector Set
5. Enter a unique name for this Data Collector set (ie. ExPerfwiz), select Create from a template (Recommended) and then click Next
6. Select the Browse button, navigate to the XML file that was saved in Step 1, select Open
7. Select Next on the next screen
8. Enter in a root Directory of where you would like to store the performance log files. Click Next
9. If you need to run this performance log under different credentials, enter it on this page. Click Finish

For a list of current recommendations to help alleviate these issue, click here

Recently, there has been a rash of performance issues on Exchange 2007 Mailbox servers where they become unresponsive due to excessive paging. Previously this was tracked down to .NET garbage collection not occurring properly which caused managed services to consume excessive amounts of memory. Applying http://support.microsoft.com/kb/942027 was the fix for this. If you have installed SP1 for Exchange 2007, we currently recommend to apply .NET 2.0 SP1 to the server which contains this fix. This is not a hard block, but a warning during the pre-req checks during setup to apply this.

Even if you have this hotfix installed, excessive paging was still occurring. After looking in to this further, we noticed that the working set for store.exe was getting trimmed causing us to page that trimmed memory to the paging file. Here is what you will see if you experience this in performance monitor. The red line is Memory\Pages/sec when the trim operations occur.

Here is a screenshot of all processes working sets.

If you look at all of the processes, they are all getting trimmed at the same time. With Exchange 2007, we have lifted the ceiling on store cache and can use upwards of 80-90% of the memory on a server. So if you have 32GB of RAM on the server, the store process itself will take longer to warm-up the cache than it will with only 16GB of RAM. During this time, clients may experience slower than normal responses while the cache is being populated. Looking at the above data, store.exe is using over 20GB of RAM and when we reach a certain peak, the working set gets trimmed causing this cache to also get trimmed. This is where performance dies when this working set is being paged to disk. Once the paging has occurred, we then have to reload that data back in to the working set as that is needed to perform a current operation. If the roller coaster ride begins, the server is going to fall over as you can see at midnight that even performance monitor couldn't even connect to the server. Once paging storm died down 30 minutes later, we were back working again, albeit slow.

Memory Planning considerations can be found in the help file at http://technet.microsoft.com/en-us/library/bb738124.aspx 

So you may ask, what is causing this? Well, in Windows 2003, if a driver makes a call to allocate a considerable amount of memory, specifically MMAllocateContiguousMemory, the working sets of processes need to get trimmed to give this call the memory that it needs. This can actually be any driver on the server that has the need to request these memory allocations. Detecting what driver it is much harder than you would think as you have to perform some type of kernel debugging to get to the bottom of it.

Currently in Windows 2003, we will trim about 1/4 of the working sets for each process to satisfy these requests and as you can see, it appears to be much more than 1/4 of the working set being trimmed, more like 3/4. Luckily, the below hotfix helps with this situation greatly when these trimming operations occurs as with the hotfix applied, we only trim 8,192 pages of each working set instead of a percentage.

A Windows Server 2003-based computer becomes unresponsive because of a memory manager trimming operation that is caused by an indeterminate module that requests lots of memory
http://support.microsoft.com/kb/938486 

As you can see below after the hotfix is applied, the store working set remained steady throughout it's lifetime which is a huge improvement over the previous screenshots.

Shown below are the working sets from all processes. When these trim operations occur, we trim memory in a step stair pattern, thus preventing your server from all of this excessive paging. This more or less prevents faulty drivers from hanging/crashing your server.

ExBPA over time will be updated to detect drivers that may aggravate this problem, but at least there is some protection built in to the Operating system now to help this situation with this hotfix applied.

Currently, you can request this hotfix and the link is in the aforementioned article to get it. 

Mike

I commonly get calls on the inability to see performance counters in Performance Monitor (perfmon) and the inability to query them through WMI. I thought I would take some time to write about how to look for any problems with Exchange Performance Counters and then provide some high level insight on how to possibly fix them. Most of this information applies to Windows 2003 servers.

If the counters are not being shown at all, the first place to check is the registry to see if the counters are not disabled. Here is a snippet of what one of the registry keys would look like

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance]
"Close"="ClosePerformanceData"
"Collect"="CollectPerformanceData"
"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"
"Open"="OpenPerformanceData"
"PerfIniFile"="eseperf.ini"

If you also see a value of Disable Performance Counters in addition to the above default entries and is set to a nonzero value, the counter at one point had a problem loading and the Operating System disabled them for whatever reason. Set the value to 0 and then close and open Perfmon again to see if you can see the counters again. More information on this Disable Performance Counters setting can be found here . If this works for you, then whew, that was an easy one….

If the Performance key is missing for a particular service, then we have bigger problems. I am not sure what causes this key to get removed, but if the key is not there, Perfmon or WMI does not know how to load the counters. There are a couple of key required parts that you need to understand before we can load any Performance counter, not just Exchange. The key pieces that are needed to reload any Performance counter is the following:

• Performance key must be created under the specified service
• Library path must be specified to the appropriate DLL for the service
• A PerfIniFile must be specified which is the name of the ini file that will reload a specific services performance counters
• Lastly, we need to have the Close, Collect, and Open values which specify what method is used to retrieve the Performance Counter Data. Note: This is unique to each service, so they will not always have the same information

If we have these key pieces of information in the registry, we have the ability to reload said services performance counters. If we take the example above for ESE, if we opened a command prompt and navigated to the C:\Program Files\Microsoft\Exchange Server\bin\perf\AMD64 directory and then typed lodctr eseperf.ini, this will reload the counters for ESE. If the counters were loaded successfully, we should now see that we have also added the First Counter, First Help, Last Counter, Last Help values as shown below.. These values correspond to specific data that was loaded in to the Perflib library.

If everything went well and you reopen Perfmon, You should now hopefully see the counters loaded. If they have not loaded, refresh the registry to see if the Disable Performance Counters key shows back up, If not, check the application log for Perflib errors which should provide additional information regarding why these counters did not load successfully.

If you don’t know already, on Windows 2003 servers, you can actually pull up performance counters using the command Perfmon /WMI. If you do not see the newly added counters, then they have not been synchronized with the WMI repository yet. To help force this along, you could run wmiadap /f to force the reload of all counters in to the WMI repository.

If this was successful, you will now see some additional Wbem entries as shown in the below pictorial.

Pulling up Perfmon /WMI again should hopefully show the counters that you are looking for. In some cases, monitoring software can still not pick up the newly added counters until the WMI service (Windows Management Instrumentation) has been restarted.

If you ever wanted to unload Performance counters, one might think that you could simply unload the counters by running unlodctr eseperf.ini. Unfortunately, this will not work because the unlodctr utility requires that a service name be passed in instead of the ini file. To find the actual name of the service, you could simply open up eseperf.ini and at the top of the file, you should notice an entry similar to the following

[info]
drivername=ESE

Ahh, there is the service name. Now if I run unlodctr ESE, this will now be successful. Doing this will remove the First Counter, First Help, Last Counter, Last Help values from the registry.

Hopefully you are still with me at this time. Now what happens if the performance registry keys for all of your services went missing, now what do you do? Reinstall, flatten the box and reinstall to get them back? Well, unfortunately, there is not a direct way of recreating these registry keys as they are created during the installation of Exchange.

The majority of the folks just export the data from another server, clean out any of the data that references performance counter data from the old server and then import them on the affected server. This does in fact work and is what I am going to talk about next on how to recover from a complete Performance key meltdown.

Attached to this post is a zip file that contains all of the Performance keys across various different role combinations such as MBX, CAS, HUB, HUB/CAS, HUB/CAS/MBX. I’ve done all of the dirty work for you, so all you have to do is to perform some very simple modification steps to the files and then you are in business.

CAUTION!!!: DO NOT IMPORT these registry keys if the Performance registry keys already exist as it will overwrite the data that currently exists in the registry and could potentially break your Performance counters that are currently working. If you only need to reload the Performance key for a single service, then pull out the data for that specific service, save it to a reg file and then import only that data. Basically use it as a reference point to help get you back running again.

If you feel the need to use these reg import files due to all of the performance keys missing for all services, simply open the file that pertains to the role that you have installed and verify that the paths are correct to the correct library files. By Default, we install Exchange in to the to c:\program files\microsoft\Exchange Server directory, so if Exchange was installed outside of the default directory, you will need to update the file manually. Let’s take the ESE performance key below:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance]
"Close"="ClosePerformanceData"
"Collect"="CollectPerformanceData"
"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"
"Open"="OpenPerformanceData"
"PerfIniFile"="eseperf.ini"

Here you will see that library has the following value:

"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"

What you will need to do is to replace the path with the correct path in which you have installed Exchange. If you installed Exchange on D: in the following directory (D:\\Program Files\\Microsoft\\Exchange Server\\bin), you would simply need to modify the first part of the path to show D:\\ instead of C:\\. A quick find and replace should work to hit all Performance keys. If you have installed it in to another Directory outside of the default paths, then you have a little more work to do to replace the path information. Just remember that for each backslash (\), you have to include double-backslashes (\\) to allow for proper importing of the reg files.

There are only a handful of entries you have to manually modify, so this really shouldn’t take too long. Once you have the paths changed, save the appropriate file as a .reg file and import it by double-clicking on the file. Verify the Performance reg keys are good and valid by opening the Registry Editor to verify.

Once the keys have been verified in the registry and look good, you can then run the powershell script to reload all of the Exchange performance counters. Simply copy the ReinstallAllPerCounters.pst.txt file to the Exchange server and then remove the .txt extension on the file. Open the Exchange Management Shell and then run the script. The screenshot below shows each ini file attempting to be loaded. Of course, on my server, I already had all of the performance keys, so we simply reported that the counters were already installed.

Note: If you would like to transfer this data to WMI, simply type Y when asked.

Once this has completed, be sure to check the application event log for details on any counters that failed to load. If everything went well, voila, you should have most if not all of your Exchange Performance Counters back once again.

If the counters are still not showing up for whatever reason in WMI, you can run the following 2 commands to clear the WMI Adap cache and then re-sync the counters again to hopefully kick start things once again.

See http://msdn.microsoft.com/en-us/library/aa394525(VS.85).aspx for more information on some of the additional commands included with the winmgmt command.

Hopefully this will help you out trying to get your Exchange performance counters going once again.

NOTE: This version of Perfwiz had been replaced by a newly written script that is talked about in http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange-2007-2010-performance-data-collection-script.aspx

Here is a replacement for general Exchange 2007 Performance data gathering. Use this in place of Perfwiz.exe as this utility does not capture all of the counters needed to properly troubleshoot Exchange 2007 performance issues.

Follow the steps below to enable these performance counters on any given Exchange 2007 server

Windows 2003 version of Perfwiz

1. Extract the contents of Exchange_2007_Perfwiz.zip to your Exchange server.
2. Open Performance Monitor
3. Expand Performance Logs and Alerts and select Counter Logs.
4. Right click Counter logs and select "New Log Settings From". Select the htm file that was extracted in Step 1. Click OK
5. Select the Log Files tab and click the Configure button
6. For the Log location, change this to a location of your choice. Click Ok
7. Click OK to save the Performance Counter log.
8. To start the Perfmon log, right click "Exchange 2007 Perfwiz" and then select Start.
9. Let this run during the problem period where performance is affected
10. Stop the perfmon log by right-clicking "Exchange 2007 Perfwiz" and selecting Stop
11. Make arrangements with a CSS representative to get the files analyzed.

Windows 2008 version of Perfwiz

1. Download the appropriate version of Perfwiz for your server. Currently I have 2 versions.
   
   
   • Exchange 2007 Perfwiz (MBX-CAS-HUB specific) - This file has targeted counters instead of including all (*) counters for obvious performance reasons and easier parsing.
     
   • Exchange 2007 Perfwiz (All Counters). Use this at your own discretion as this will collect a *lot* of data which may decrease performance on an Exchange Server.
     
   • Other Role based XMLs are self-explanatory
     
     How to download 
     To download these XML files to your computer, right click the file of your choice, select Save Target As... , and then save it to a directory location of your choice on your Exchange Server
     
     Role Based
     Use these as a high level look in to how the server is performing and if  you need to branch out with more counters, use the Full Counter/Instance set below.
     
     Exchange_2007_Perfwiz-AIO.xml (HUB/CAS/MBX All-In-One)
     Exchange_2007_Perfwiz-CAS.xml
     Exchange_2007_Perfwiz-MBX.xml
     Exchange_2007_Perfwiz-HUB.xml
     Exchange_2007_Perfwiz-HUB-CAS.xml
     Exchange_2007_Perfwiz-UM.xml
     
     All Counters/All Instances
     Use this counter set at your own discretion as this could potentially cause performance degradation on your server trying to log this amount of counters.
     
     Exchange_2007_Perfwiz-Full.xml
     
     
2. Open Performance Monitor
3. Expand Reliability and Performance and then expand Data Collector Sets
4. Right click User Defined, Select New, and then Data Collector Set
5. Enter a unique name for this Data Collector set (ie. ExPerfwiz), select Create from a template (Recommended) and then click Next
6. Select the Browse button, navigate to the XML file that was saved in Step 1, select Open
7. Select Next on the next screen
8. Enter in a root Directory of where you would like to store the performance log files. Click Next
9. If you need to run this performance log under different credentials, enter it on this page. Click Finish

Recognition

I'd like to thank Ben Winzenz for creating the Windows 2008 *full* version of Perfwiz and John Rodriguez for getting me the counter sets for the All-In-One XML which I doctored up in a custom XML file. 

Updates
2/13/2009 - Updated all XML files to include counters from Monitoring Without System Center Operations Manager

9/28/2009 - Added TCPv4 and TCPv6 counters to all Perfwiz counter sets

10/15/2009 – Large update to the XML files. See below what has been added

Role: HUB, CAS, HUB-CAS, UM, MB, AIO

\Memory\*
\Netlogon(*)\*
\Process(*)\*
\Processor(*)\*
\Redirector\*
\Server\*
\System\*
\MSExchange ADAccess Domain Controllers(*)\LDAP Searches timed out per minute
\MSExchange ADAccess Domain Controllers(*)\Long running LDAP operations/Min
\MSExchange ADAccess Domain Controllers(*)\Number of outstanding requests
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Read calls/Sec
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Read Time
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Search calls/Sec
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Search Time
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Searches timed out per minute
\MSExchange ADAccess Local Site Domain Controllers(*)\Long running LDAP operations/Min
\MSExchange ADAccess Local Site Domain Controllers(*)\Number of outstanding requests

Role: HUB, CAS, HUB-CAS, UM, AIO
\MSExchange Store Interface(*)\ConnectionCache num caches
\MSExchange Store Interface(*)\ConnectionCache out of limit creations
\MSExchange Store Interface(*)\ConnectionCache total capacity
\MSExchange Store Interface(*)\ExRPCConnection creation events
\MSExchange Store Interface(*)\ExRPCConnection disposal events
\MSExchange Store Interface(*)\ExRPCConnection outstanding

Role: CAS, HUB-CAS, AIO
\MSExchange OWA\AS Queries Failure %
\MSExchange OWA\Average Search Time
\MSExchange OWA\Failed Requests/sec
\MSExchange OWA\Logons/sec
\MSExchange OWA\Proxy Response Time Average
\MSExchange OWA\Proxy User Requests/sec
\MSExchange OWA\Store Logon Failure %

Full
Netlogon(*)\*
\MSExchange ADAccess Local Site Domain Controllers(*)\*

Windows 2003 version of Perfwiz for Exchange 2007
Netlogon(*)\*

10/21/2009 – Updated Database Instance Counters to include all instances. Affects AIO and MBX Role XML
11/4/2009 – Added the following Physical Disk Counters to all XML files.

\PhysicalDisk(*)\Avg. Disk Queue Length
\PhysicalDisk(*)\Avg. Disk sec/Read
\PhysicalDisk(*)\Avg. Disk sec/Write
\PhysicalDisk(*)\Disk Reads/sec
\PhysicalDisk(*)\Disk Transfers/sec
\PhysicalDisk(*)\Disk Writes/sec

01/11/2010 -  Added ASP.NET\Requests queued to the CAS,HUB-CAS, and AIO XMLs. Added MSExchange Database(Information Store)\Database Cache % Hit to the AIO and the MBX XMLs.

01/14/2010 - Added all MSExchangeIS Client counters to MBX and AIO XMLs