If you have had Exchange performance issues in the past, you must know by now that there are a lot of variables that affect Exchange overall performance and sometimes took a long time to find root cause of those problems as they can get quite complex.
If you are looking for some more detailed reports with some added charting capabilities, there is the Performance Analyzer Tool or PAL that you can grab from http://www.codeplex.com/PAL .
PAL is an extremely useful and powerful tool for automatically generating HTML reports that were collected from a performance monitor counter log. This tool uses a subset of dependencies such as Log Parser and the Office Web Components to create these reports.
PAL uses XML configuration files that parse the most important counters for Exchange performance issues and throws alerts when thresholds are exceeded for those counters. All that is needed is a blg file that was previously collected. This is not a replacement for standard performance analysis, but helps with automating some of the common tasks that are performed when troubleshooting performance issues for Exchange. With the amount of perf analysis that I do, I need a way to automate a lot of this to help give me the 30,000 foot view of how an Exchange server is performing.
With the release of the Exchange 2007 Performance Counters and their associated thresholds at http://go.microsoft.com/fwlink/?LinkId=122254, PAL consumes all of those counters and creates some nice reports for you. PAL also uses the counters in the Exchange 2003 Performance Troubleshooting white paper at http://technet.microsoft.com/en-us/library/aa997270.aspx, so no matter what type server you have, you should have reporting capabilities for each.
Let’s take a look at what kind of reports this creates.
After PAL has analyzed a blg file, an Internet Explorer Window will open with the analysis. As you can see below, there is a listing of counters that were parsed and checked for any over specific thresholds as set in the Exchange XML file. At the end of each counter, there is an Alerts part that tells you that if any of the samplings were over a specified threshold. If there was one, that value is incremented by one.
So if we look further down the list, we can see that the RPC Averaged Latency threshold was hit 6 times as shown circled in red below.
Clicking on that counter brings you to the chart for this counter. This shows a rather large spike in overall RPC Averaged Latency for that specific time.
Scrolling downward in this report now shows the thresholds that were hit for a specific time and their associated values.
As you can see, we hit a critical threshold for RPC latency that could be affecting overall client performance. This is merely a symptom of the underlying problem which could be disk latencies, LDAP Latencies, High CPU, etc., but at least the data that you would normally look at would be at your fingertips. Clicking the “Back to the top” link under each counter section will get you back to the Counter list at the beginning of the report. This allows for very quick counter analysis to see which counters could be affecting overall server performance.
The great thing about the PAL tool is that you can customize the XML files to your liking, so if you need to add/remove a specific counter and its associated threshold, you can do this very easily. For the threshold creation, if you know vbscript enough to create If/Then/Else statements, then you can customize this tool very easily to generate any report and threshold.
If you would like to see a complete sample report for a general system overview, go to http://www.codeplex.com/PAL/Release/ProjectReleases.aspx?ReleaseId=11105 and then click on the PAL Sample Report.mht link. If you need to download and install PAL, always select the most recent version for download as the above link presents an older build.
1. Ensure that the PAL tool and dependency components have been installed from http://www.codeplex.com/PAL .
2. Click Start, Run, and then PAL. This will launch the PAL wizard interface.
Follow these steps to create a counter log .htm file once the PAL wizard has been launched. Note: This contains very specific counters instead of the full counter set that perfwiz uses, so you can choose how granular you would like to get.
This is a resource intensive application while these perfmon files are being parsed, so I would recommend using your fastest machine to run these reports on. Once PAL has completed processing the queued items, an IE window will open for each report in the queue.
I hope you have found this information useful and if you should have any questions regarding the tools usage, any possible problems that you may run in to or just suggestions to improve the tool, you can email paltool@microsoft.com
Happy reporting!!!
Mike
For the latest XML updates that are not included with the current released version of the PAL tool, see http://blogs.technet.com/mikelag/archive/2008/08/20/xml-updates-for-the-pal-tool.aspx.
Switching to or running Outlook in Online mode against an Exchange 2003 server may exhibit slower than normal performance when clicking on particular folders in your mailbox that contain high item counts.
This is a known issue when the “Show in Groups” option is turned on in Outlook when many items exist in a particular folder. The fix is actually two parts. One for the server side and one for the client side. Note: The server side fix is not necessary for Exchange 2007.
What you might see in Performance Monitor is similar to the load that Client side search engines add. This counter in particular is “Slow Findrow Rate” located under MSExchangeIS Mailbox. One might think that this counter is only used for detecting client search engines, but this counter increases whenever an application is crawling through a users mailbox.
In this case, Outlook creates a search restriction (“Show in Groups” with a descending date sort order) initially using a Fast FindRow call, but the server immediately switches to a Slow FindRow call which causes a large delay in returning the view for that newly created restriction while items in that particular folder are being queried. This could lead up to the dreaded RPC dialog box or the balloon stating the Outlook is requesting data from the server. While we are creating this restriction on the Exchange Server, this may also cause an added CPU performance hit. Since these restrictions are not cached, these problems may occur each time the user clicks on these folders that may have high item counts.
To better optimize this search restriction behavior, an optimized search was added server side so that these restrictions created by Outlook will use the Fast FindRow call method instead of switching to the Slow Findrow method. This provides a huge performance gain and mitigates the Outlook delay problems.
To be able to handle these optimized searches efficiently, your server performance has to be within our recommended performance guidelines, so even though you have applied these fixes, there is still a possibility that these problems could occur due to underlying performance problems server side.
Here are the links to the articles and fixes for each.
Exchange 2003 Server Side fix Error message when you enable the "Show in Groups" option for a folder that contains thousands of e-mail messages in Outlook: "Outlook is retrieving data from the Microsoft Exchange Server <ServerName>" http://support.microsoft.com/kb/948828
Outlook fixes Description of the Outlook 2003 hotfix package: February 21, 2008 http://support.microsoft.com/kb/949289
Description of the Outlook 2007 hotfix package: February 27, 2008 http://support.microsoft.com/kb/949401
For troubleshooting Exchange user performance related issues or to help plan your design for Exchange 2007, Exchange User Monitor (Exmon) is a great utility to have in your tool bag and contains a wealth of knowledge on your current user activities. Exmon has been previously talked about at http://msexchangeteam.com/archive/2005/04/06/403409.aspx and some FAQs regarding the tool can be seen at http://msexchangeteam.com/archive/2005/06/13/406276.aspx. There is also another link which is much harder to find at http://technet.microsoft.com/en-us/library/bb508855(EXCHG.65).aspx which explains how to use the tool in much greater detail.
Exmon tracing uses the ETW (Event Tracing for Windows) facility of Windows to send internal application event data to .etl files for later analysis. Detailed information regarding this tracing can be found in the Windows DDK at http://msdn2.microsoft.com/en-us/library/aa468736.aspx and also in http://msdn2.microsoft.com/en-us/library/aa363668.aspx
ETW tracing uses event providers that were specifically written for an application or driver and can be referenced either by its name or by its GUID. To view a list of current providers, run logman query providers from a command prompt which will give you a list similar to the below screenshot.
Notice the Exchange Information Store provider with its GUID of 2EACCEDF-8648-453e-9250-27F0069F71D2. This is the provider that ExMon uses for its tracing and is also the information that is needed later on this article to enable ETW tracing from a command line. Please note that this article will work on Exchange 2003 or Exchange 2007 servers as the provider GUID has not changed.
Before enabling ExMon tracing, the following registry keys must be added to the registry to allow Exmon to collect data in the ETL file.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem] "RpcEtwTracing"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Trace] "UsePerformanceClock"=dword:00000001
To enable tracing on any given server, there is essentially 2 methods that can be used to create and start/stop Exmon tracing.
1. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en
2. Create a directory called Tracing (ex. C:\tracing)
3. Copy tracelog.exe to this directory from the default install location of c:\program files\Resource Kit.
4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
tracelog.exe -start Exmon_Trace -f c:\Tracing\Exmon_trace.etl -seq 3500 -guid control.guid
5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
tracelog.exe -stop Exmon_Trace
6. Create a control.guid file and then add the appropriate GUID for Exmon tracing. Note: This GUID should be the only piece of information in this file. Save this to the location created in Step 2.
2EACCEDF-8648-453e-9250-27F0069F71D2
7. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog -l and look for an entry call Exmon_Trace. If this is found in the list, then the tracing has been enabled as shown below.
8. Run stop_tracing.cmd to stop the tracing.
---------------
1. Create a directory called Tracing (ex. C:\tracing)
2. Create the trace log using the "Exchange Information Store" provider that Exmon tracing uses. The GUID or the name of the provider can be used as shown below. logman create trace Exmon_Trace -p {2EACCEDF-8648-453e-9250-27F0069F71D2} -nb 3 25 -bs 3 -o c:\Tracing\
logman create trace Exmon_Trace -p "Exchange Information Store" -nb 3 25 -bs 3 -o c:\Tracing\
3. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.
Logman start Exmon_Trace
4. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2. logman stop Exmon_Trace
To create a schedule to start/stop these batch files at a particular time that you would like to specify, you could use the schtasks.exe command to do this. For more information on schtasks.exe syntax, follow the steps in http://support.microsoft.com/?id=814596 .
I am sure you are all are intimately familiar with the problems with the Scalable Networking Pack (SNP) including it's use of the TCP Chimney feature that I blogged about at http://msexchangeteam.com/archive/2007/07/18/446400.aspx.
The problems that surfaced due to these features being enabled by default in the Service Pack 2 release of Windows 2003 brought out the worst in some network card drivers causing all types of connectivity related problems and crippled some organizations.
After some time, a hotfix (http://support.microsoft.com/kb/948496) was created to disable these features by adding the appropriate registry keys to a server which then required a reboot to get the SNP features disabled.
As of 8/27/2008, a new hotfix (http://support.microsoft.com/kb/950224) has been released to help address some of the commonly reported problems with relationship to Chimney and RSS for Windows 2003 servers. As more companies deploy Windows 2008 and Vista, it is crucial, or in my opinion, critical that this hotfix be applied to all Windows 2003 servers that may communicate with these operating systems. One of the main reasons is a new feature called TCP auto-tuning which makes use of RSS to expand and shrink the sizes of your TCP window to increase/decrease throughput based on current network load. This feature greatly increases throughput on your network, but if there is an underlying problem with the network card driver or any of these features between disparate systems, you may experience slower than normal network performance. The good news is that the Chimney feature is disabled by default in Vista/Windows 2008.
Here you will find the latest XML updates to the PAL tool for Exchange 2003 and Exchange 2007. See below for more information
Exchange 2003 XML Updates - Download Interim update here (Last updated 8-15-2008) 5-25-2008 Updates - Added Slow QP threads, Slow Search Threads and Search Task Rate counters to detect deep folder traversal searches 8-15-2008 Updates - Updated Available Memory Trend calculations - Updated Processor and Processor Queue Length calculations - Added Network\Outbound Packet Errors 2007 XML Updates
- Download Interim update here (Last Updated 12-14-2009)
5-2-2008 Updates - Updated Mailbox Role for SP1 counters due to Database counter renames 8-4-2008 Updates - Updated All Counters sets to fix .NET memory issue, minor changes/updates and updated Roles with all counters from our published counters listed at http://technet.microsoft.com/en-us/library/bb201720(EXCHG.80).aspx 8-15-2008 Updates - Updated Available Memory Trend calculations - Updated Processor and Processor Queue Length calculations - Added Network\Outbound Packet Errors - Added new combined role XML that includes all HUB/CAS/MAILBOX role counters 8-25-2008 - Windows 2008 counters for Exchange 2007 are in the works. Stay tuned!!!! 8-30-2008 - Removed Single Role XML files and replaced it with one combined Exchange 2007 XML (Let me know if you still want individual role XML files) 9-16-2008 - Huge update to the XML files. Broke out the XML files again to work around the "Log row too long" error. Information overload for the logparser tool :) Also added a HUB/CAS combined XML file. 12-14-2009 - Overhauled the XML files for HUB, CAS, HUB/CAS, and MBX roles to provide better analysis. See http://blogs.technet.com/mikelag/archive/2009/12/14/improved-pal-analysis-for-exchange-2007.aspx for all of the updates.
One of the problems we see in support from time to time is High CPU or server not responding scenarios causing 623 events on Exchange servers. This has been talked about in great detail in the following links:
Troubleshooting Version Store issues - JET_errVersionStoreOutOfMemory http://msexchangeteam.com/archive/2006/04/19/425722.aspx
Version Store issues revisited - Event ID 623 at Information Store service startup http://msexchangeteam.com/archive/2008/05/23/448923.aspx
Dave did a good job of explaining what they are on what happens on the Exchange side when search folders are created in the following blog
Microsoft Exchange and Search Folders http://blogs.msdn.com/dgoldman/archive/2008/07/01/microsoft-exchange-and-search-folders.aspx
For a simple look at how to create a visible Search Folder in outlook, see http://office.microsoft.com/en-us/outlook/HA100389111033.aspx
Most of the times, this problem occurs because of hidden search folders that Dave mentions.
To break this down in to simple terms, you can think of Version Store of a predetermined amount of memory to build a queue for transactions in the store which can be tracked via the Version Buckets allocated counter in Performance monitor. What happens in this case, is that the amount of Version Buckets goes up due to a queued transaction taking a long time to finish. Since this queue is serialized, any transaction queued that takes a long time to complete will back up any other request behind it. During Store startup, there are a lot of transactions that are occurring, so you essentially get a snowball effect causing the store to go unresponsive.
This could also happen during runtime as well, but the bad part is that if you have hidden search folders causing this problem, this queued transaction will survive a restart of the information store and that is when things get really bad.
To resolve this problem, you will normally have to dump the store.exe process and have someone from CSS look at the dump to confirm what the underlying problem is and who is causing it.
After looking at this from the performance side as this fits in to the performance category, you can actually see this by adding just a few counters in perfmon to detect this type situation.
As I mentioned before, The Version buckets allocated counter is what you use to track transactions in the store and is also how we trigger getting the store dump for analysis.
Here is what this looks like in Performance monitor. This blg file was taken at 5 second intervals.
As you can see about in the middle of the perfmon, is when the Store process was restarted denoted by the dip in version buckets. As soon as the store started, Version buckets shot all the way up to a maximum of 4,000. That is extremely high and bad for Store health.
One of the characteristics of this phenomenon, is during the time of these queued transactions, you will see RPC requests flat line as shown below. What this means is that during the time that the search folder is being updated, we are performing a deep search traversal. Remember that I mentioned earlier that this was serialized? Well, while we are processing that transaction, that is the only thing we are doing, so we are not creating or removing any RPC threads currently running, thus the flat line for this counter.
OK, so now you have this pattern, but another counter to add is Slow QP threads. This counter is essentially incremented when an un-optimized search for a string in the text context or subject occurred. This is what a Search folder really is basing it on some text or some property of a message. This confirms what is happening here. During normal operations, this counter should remain below 10 at all times, ideally under 5 on average.
One of the side effects of these searches is a high rate of Disk Read activity due to the amount of data we are requesting.
The unfortunate problem with finding who the culprit is that you have to take that mailbox store down and run an isinteg -dump against that database. Once you have this file, you can use Dave's method of searching through the file to determine who has a high amount of Search folders in their mailbox using his SearchFolderFinder utility at http://blogs.msdn.com/dgoldman/archive/2008/08/26/searchfolderfinder-has-now-been-posted.aspx
Maybe this will give you some fighting power in tackling these problems if not now, but in the future as well.
cya!!
During your normal flow of troubleshooting performance for Exchange, you may have the need to see what functions might be consuming higher than normal resources on your Exchange server. These functions might be causing spikes in CPU usage or sustained CPU usage over time.
Luckily, the Exchange Troubleshooting Assistant could help you gather the necessary data to understand what is going on inside store without having the need to dump the store process and then calling Support for analysis of this file.
Mind you, this doesn't cover every scenario, but will help to bring out some of the more common issues you may see with latent calls. Here is a short list of what we could report on:
With that said, let's get on to running the tool. It is important to note that this tool should be run during the time of the problem, not after as whatever it was causing the issue, is now gone.
To start the tool, you can do this either of two ways.
Once the Performance Troubleshooter has been selected, Enter an identifying name for this run. Click Next
In the drop down list, select "The number of outstanding RPC requests is high". Click Next
Enter the appropriate Exchange server name and Global Catalog Server name. Note: If you are logging on from an account that does not have permissions to the Exchange server, select the "Show advanced login options" to enter a different set of credentials. Click Next when done.
Select a location to store the Performance logs
Select "Show advanced data collection options"
Make sure that "Collect Function log (FCL) data is selected. Click Next
The trace will the start and collect data for 3 minutes and then perform some statistical analysis on the data collected.
You will then be presented with the results of the analysis and then tell you if any function calls are consuming an exorbitant amount of time. If anything if found, it will be reported here. In this case, there was not a problem on the server, but if there was, it would be reported right here.
This may give you some immediate clues in less than 5 minutes on what might be occurring inside the store process. Happy Troubleshooting!!
This may give you some immediate clues in less than 5 minutes on what might be occurring inside the store process.
Happy Troubleshooting!!
Tracing Store startup problems upon server reboot is not an easy task as ExTRA tracing does not survive the server being rebooted whereas in Exchange 2003, the “Enable Tracing” key did survive a reboot.
After enabling an ExTRA session for PFD store startup tracing and looking at the contents of what was enabled when doing this, you will see the following from a command prompt.
Notice the “Microsoft Exchange Server 2007” provider with its GUID of 79bb49e6-2a2c-46e4-9167-fa122525d540. This is the provider that Exchange 2007 uses for most of its tracing
So now we need a way to trace this out. First, we need to create a method to enable logging and another way to enable the logging upon reboot of the server
Logman (Built in to the OS)
2. Create the trace log using the “Microsoft Exchange Server 2007” provider that Exchange 2007 tracing uses. The GUID or the name of the provider can be used as shown below. logman create trace ExchangeDebugTraces -p {79bb49e6-2a2c-46e4-9167-fa122525d540} -nb 3 25 -bs 3 -o c:\Tracing\
logman create trace ExchangeDebugTraces -p " Microsoft Exchange Server 2007" -nb 3 25 -bs 3 -o c:\Tracing\
After performing this, you can see this using a logman query command.
3. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config
TraceLevels:Pfd Store:tagPFDServerStart FilteredTracing:No InMemoryTracing:No
Note: You could also create this same file by using the ExTRA interface to enable this tracing as well, thus populating the same information as listed above.
4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.
Logman start ExchangeDebugTraces Running logman query shows this trace running.
5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2. logman stop ExchangeDebugTraces
6. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en
7. Create a directory called Tracing (ex. C:\tracing)
8. Copy tracelog.exe to this directory from the default install location of c:\program files(x86)\Resource Kit.
9. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
tracelog.exe -start ExchangeDebugTraces -f c:\Tracing\ExchangeDebugTraces.etl -seq 3500 -guid control.guid
10. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
tracelog.exe -stop ExchangeDebugTraces
11. Create a control.guid file and then add the appropriate GUID for Microsoft Exchange Server 2007 tracing. Note: This GUID should be the only piece of information in this file. Save this to the c:\tracing directory.
79bb49e6-2a2c-46e4-9167-fa122525d540
12. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config
13. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog –q ExchangeDebugTraces to check if we are now tracing
This looks good. We are now happily tracing using PFD store startup tracing.
14. Run stop_tracing.cmd to stop the tracing.
On Windows 2003 and Windows 2008 servers, we can enable this tracing when the server is rebooted by using the built-in utility called Schtasks. More information regarding this tool can be found in the following locations
Windows 2008 schtasks help Windows 2003 schtasks help
To create a new scheduled task, open a command prompt and type the following: schtasks /create /tn "Start Store PFD Tracing Upon Reboot" /tr "C:\tracing\start_tracing.cmd" /sc onstart /ru "" Note: Specifying "" for the /RU switch, tells the task to run as SYSTEM. To verify that the scheduled task has been added properly, type schtasks from a command prompt. This should return the 'TaskName" and the "Next Run Time" of "At system start up"
You could also view this in the Control Panel under Scheduled Tasks
Reboot the server and the tracing will begin before the Information Store service starts.
Once the tracing has been collected, it is important to stop tracing and then remove this schtask to prevent this tracing from occurring in the future upon reboots.
To stop tracing, simply launch the stop_tracing.cmd file in the c:\tracing directory.
To remove the scheduled event, type the following command:
schtasks /delete /TN "Start Store PFD Tracing Upon Reboot"
Unfortunately, the trace file cannot be read unless you are working with Microsoft Support representative, but at least this will give you some ways that will help catch problems as they occur and when they occur. These methods can be applied for any ExTRA tracing in general, so this is not just limited to tracing Store Startup issues.
Need help troubleshooting Outlook 2007 performance issues? Well, this link will help you do just that to rule out the specific things that might be causing significant performance degradation.
Performance tips for deploying Outlook 2007 http://technet.microsoft.com/en-us/library/cc179071.aspx
Here is a hidden little gem (Messaging guide for Office 2007) that has all kinds of useful information in it, but unfortunately you cannot find it since it is posted as a document and not as a nicely formatted HTML page so our Indexing services can consume it.
Downloadable book: Messaging guide for the 2007 Office release http://technet.microsoft.com/en-us/library/cc179109.aspx
In the next couple of months, I will be talking about how I tackle performance related issues in Exchange 2007 and some of the methodology used. Exchange 2007 is a beast and you need to rule things out rather quickly to find possible root cause.
I hope you find this information useful
If during your troubleshooting, you run in to some counters (Disk Read/Writes) that just don't seem correct based on the way the server is performing, then you might be running in to a drift calculation bug with certain AMD processors. You may also see negative PING response times as well.
Here is what it looks like in Performance Monitor as a steady rise in disk response times.
Avg. Disk sec/Read Counter
Avg. Disk sec/Write Counter
Over time, this may look like a problem, but it really isn't, it is just that it is being reported incorrectly.
Clint Huffman, which is the creator of the PAL tool has a nice write-up on this problem at http://blogs.technet.com/clint_huffman/archive/2008/03/21/warning-perf-counter-data-might-be-inaccurate-on-some-amd-processor-computers.aspx
OEM References
HP - http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=428936&prodTypeId=15351&objectID=c01075682
Citrix - http://support.citrix.com/article/CTX114782
IBM - http://www-01.ibm.com/support/docview.wss?rs=0&q1=opteron&q2=usepmtimer&uid=swg21267793&loc=en_US&cs=utf-8&cc=us&lang=en and ftp://ftp.software.ibm.com/systems/support/system_x/41y7884.rtf
Hope this helps in your travels.