Mike Lagase : Tools

New Windows Dynamic Cache Service for 64-bit Windows 2003 servers

mikelag — Mon, 09 Feb 2009 23:04:49 GMT

If you've ever had an issue where low memory conditions were causing working set trimming issues due to excessive use of the System File Cache, then we have just released a new service that can be used to help alleviate this issue called Microsoft Windows Dynamic Cache Service.

More information regarding this new service can be found here and a direct link to download this new service can be found here

With Exchange 2007 servers also running in to these issues which I blogged about here, this service could potentially allow other 3rd party services to play nice with Exchange 2007 which may be consuming more than it's fair share of the System File Cache.

So if you find that Exchange performance is suffering because of some other service taking up overall memory in the System File Cache, then this service may be just for you.

Hope this helps with some of your performance related issues.

Mike

Updated Exchange 2003 Perfwiz

mikelag — Mon, 02 Feb 2009 23:18:31 GMT

For all of you folks out there still hanging on to Exchange 2003 and dealing with performance issues, I have taken the liberty to update the Perfwiz data counter collection to the latest/greatest counters as the old Perfwiz tool on our download site is severely outdated.

Exchange 2003 Update Perfwiz

Extract the contents of E2k3_Perfwiz.zip to your Exchange server.
Open Performance Monitor
Expand Performance Logs and Alerts and select Counter Logs.
Right click Counter logs and select "New Log Settings From". Select the htm file that was extracted in Step 1. Click OK
Select the Log Files tab and click the Configure button
For the Log location, change this to a location of your choice. Click Ok
Click OK to save the Performance Counter log.
To start the Perfmon log, right click "Exchange_2003_Perfwiz" and then select Start.
Let this run during the problem period where performance is affected
Stop the perfmon log by right-clicking "Exchange_2003_Perfwiz" and selecting Stop
Make arrangements with a CSS representative to get the files analyzed.

Counter Collection List

\Database ==> Instances(*)\*
\Database(*)\*
\Epoxy(*)\*
\Exchange Server HTTP Extensions(*)\*
\LogicalDisk(*)\*
\Memory\*
\Microsoft Exchange ActiveSync\*
\MSExchange Intelligent Message Filter(*)\*
\MSExchange Oledb Events(*)\*
\MSExchange Oledb Resource(*)\*
\MSExchange Sender ID(*)\*
\MSExchange Web Mail(*)\*
\MSExchangeActiveSyncNotify OmaPush\*
\MSExchangeAL(*)\*
\MSExchangeDSAccess Caches(*)\*
\MSExchangeDSAccess Domain Controllers(*)\*
\MSExchangeDSAccess Processes(*)\*
\MSExchangeIMAP4(*)\*
\MSExchangeIS Mailbox(*)\*
\MSExchangeIS Public(*)\*
\MSExchangeIS\*
\MSExchangeMTA Connections(*)\*
\MSExchangeMTA\*
\MSExchangeOMA\*
\MSExchangeSA - NSPI Proxy\*
\MSExchangeSRS\*
\MSExchangeTransport Filter Sink(*)\*
\MSExchangeTransport Store Driver(*)\*
\Network Interface(*)\*
\Paging File(*)\*
\PhysicalDisk(*)\*
\Process(*)\*
\Processor(*)\*
\Server\*
\SMTP NTFS Store Driver(*)\*
\SMTP Routing(*)\*
\SMTP Server(*)\*
\System\*
\Web Service(*)\*

Enjoy!!

Getting started with Exchange 2007 Performance monitoring and Windows 2008

mikelag — Fri, 23 Jan 2009 13:41:24 GMT

With all the moving bells and whistles of Exchange 2007 and the amount of dependencies that Exchange relies on, it could be a bit overwhelming on where to get started. For some, they may have their own monitoring tools such as MOM/SCOM to pull this data, but when you are in a real-time crisis or something that affecting critical business functions over time, finding out where to start can lead one in to frustration.

Well, never fear, I'll help guide you in to looking at the server from the 30,000 ft view and then delve deeper in to the plethora of pertinent counters to look at. So where do I get started?

We could start off with a once upon the time story or 10 ways to lick an Exchange performance problem in the bud, but lets start with the simple basics. First, you will need to start some monitoring on your server to start collecting the data. I've made this quite easy for you as I have created templates for you that are very easily importable in to the Reliability and Performance monitor in Windows 2008. I'll be concentrating on Windows 2008 analysis in this blog as that is the wave of the future. Go to http://blogs.technet.com/mikelag/archive/2008/05/02/perfwiz-replacement-for-exchange-2007.aspx to find the templates for your particular role installation. Once you have imported your favorite XML file, you will get a screen similar to the following. Simply right-click on your imported counter set and click start. Let this run through the problem time in which you are trying to troubleshoot. Note: I have imported all of the XML files here to get started.

Once you have collected enough data, you can stop the Performance log. From there, you can then expand reports in the menu and then click on the report associated with the log that you just ran. You will get a screen similar to the following. The really cool thing about Windows 2008 is that if you should double-click on a performance file (.blg) or open this report view, you will get all counters added automatically with all Totals listed for each counter. Trying to load all counters/all instances would take a substantial amount of time to load, so I feel this is the best place to begin your troubleshooting. This is what I call the 30,000 ft view. Loading this type of view is normally very fast to load and provides you a wealth of information right at your fingertips.

What you will want to do is to compare each of the counters with the threshold values that are listed in the Technet article Monitoring Without System Center Operations Manager. If any counters are deemed suspect, I would put them on your list of things to delve deeper in to. Once you go through all of these counters, you could see if there is anything that stands out which will give you something to concentrate on. Counters that you no longer want to see can be easily hidden as shown below or by simply un-checking the box in the Show column of each counter

Now, since we are only showing totals at this point, you may need to jump off the diving board and dive right in to specific instances of a counter in which a threshold is above normal. Simply click the Plus sign on the menu and then add the counters of your choice.

That is basically all there is too it in getting started, but there are other ways of getting to root cause of a complex issue. Read on...

If you have collected a .blg file already, you can parse all of the data automatically via the Performance Analyzer Log tool. See my previous blog on how to use this most helpful tool here

Another way to troubleshoot Exchange Performance can be had in the Exchange Management Console Toolbox as shown below via the Performance Troubleshooter. This can also be very helpful in troubleshooting these issues as this tool has some built in tracing mechanisms that allow you to peer in to some of the Exchange processes in which you cannot get to by any normal logging or tracing means without PSS involved.

I'll be posting most of these type posts coming in the future, so keep a watch on my blog.

Mike

Showing Process ID Information in Performance Monitor

mikelag — Mon, 13 Oct 2008 16:00:51 GMT

In some situations while monitoring performance of a server, you may find a time when a particular process is restarting or crashing over time. By default, in Performance Monitor, you cannot view Process ID (PID) values to determine if a process was restarted for whatever reason. This is apparent for IIS Application pools as these processes could frequently crash/restart throughout the day. With each new restart of a process, a different PID is generated, so it would be nice if you could see this happening in Perfmon.

Luckily, there is a way to do just that to append the PID number for each process, so that if the PID should change, you can easily detect this in Performance Monitor.

To show ProcessID's along with process names, add the following registry entry on the server

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance]

"ProcessNameFormat"=dword:00000002

This needs to be implemented on the machine in which the perfmon is being taken from. For more information regarding this registry setting, see 281884

Performance Troubleshooting using the PAL tool

mikelag — Thu, 21 Aug 2008 03:06:18 GMT

If you have had Exchange performance issues in the past, you must know by now that there are a lot of variables that affect Exchange overall performance and sometimes took a long time to find root cause of those problems as they can get quite complex.

If you are looking for some more detailed reports with some added charting capabilities, there is the Performance Analyzer Tool or PAL that you can grab from http://www.codeplex.com/PAL .

PAL is an extremely useful and powerful tool for automatically generating HTML reports that were collected from a performance monitor counter log. This tool uses a subset of dependencies such as Log Parser and the Office Web Components to create these reports.

PAL uses XML configuration files that parse the most important counters for Exchange performance issues and throws alerts when thresholds are exceeded for those counters. All that is needed is a blg file that was previously collected. This is not a replacement for standard performance analysis, but helps with automating some of the common tasks that are performed when troubleshooting performance issues for Exchange. With the amount of perf analysis that I do, I need a way to automate a lot of this to help give me the 30,000 foot view of how an Exchange server is performing.

With the release of the Exchange 2007 Performance Counters and their associated thresholds at http://go.microsoft.com/fwlink/?LinkId=122254, PAL consumes all of those counters and creates some nice reports for you. PAL also uses the counters in the Exchange 2003 Performance Troubleshooting white paper at http://technet.microsoft.com/en-us/library/aa997270.aspx, so no matter what type server you have, you should have reporting capabilities for each.

Let’s take a look at what kind of reports this creates.

After PAL has analyzed a blg file, an Internet Explorer Window will open with the analysis. As you can see below, there is a listing of counters that were parsed and checked for any over specific thresholds as set in the Exchange XML file. At the end of each counter, there is an Alerts part that tells you that if any of the samplings were over a specified threshold. If there was one, that value is incremented by one.

PAL Sample report

So if we look further down the list, we can see that the RPC Averaged Latency threshold was hit 6 times as shown circled in red below.

Clicking on that counter brings you to the chart for this counter. This shows a rather large spike in overall RPC Averaged Latency for that specific time.

Scrolling downward in this report now shows the thresholds that were hit for a specific time and their associated values.

As you can see, we hit a critical threshold for RPC latency that could be affecting overall client performance. This is merely a symptom of the underlying problem which could be disk latencies, LDAP Latencies, High CPU, etc., but at least the data that you would normally look at would be at your fingertips. Clicking the “Back to the top” link under each counter section will get you back to the Counter list at the beginning of the report. This allows for very quick counter analysis to see which counters could be affecting overall server performance.

The great thing about the PAL tool is that you can customize the XML files to your liking, so if you need to add/remove a specific counter and its associated threshold, you can do this very easily. For the threshold creation, if you know vbscript enough to create If/Then/Else statements, then you can customize this tool very easily to generate any report and threshold.

If you would like to see a complete sample report for a general system overview, go to http://www.codeplex.com/PAL/Release/ProjectReleases.aspx?ReleaseId=11105 and then click on the PAL Sample Report.mht link. If you need to download and install PAL, always select the most recent version for download as the above link presents an older build.

How to launch the PAL tool

1. Ensure that the PAL tool and dependency components have been installed from http://www.codeplex.com/PAL .

2. Click Start, Run, and then PAL. This will launch the PAL wizard interface.

How to create a counter log file using PAL

Follow these steps to create a counter log .htm file once the PAL wizard has been launched. Note: This contains very specific counters instead of the full counter set that perfwiz uses, so you can choose how granular you would like to get.

Launch PAL.
Click the Threshold File Tab.
In the Threshold File Title drop down box, select the Threshold File Title version of your choice
Click the button.
Save the settings to a .htm file. Follow the steps in the Exchange 2007 Perfwiz replacement steps at http://blogs.technet.com/mikelag/archive/2008/05/02/perfwiz-replacement-for-exchange-2007.aspx starting at step 4 to import this .htm file in to Performance monitor.

Note: This export feature only works on Windows 2003 servers since the ability to import htm files in Windows 2008 has changed. I will post an update on how to do this on Windows 2008 servers at a later time.

How to run the PAL wizard

Launch PAL. This will bring you to the Welcome Screen. Click Next
On the Counter Log tab, select a blg file of your choice. Click Next
Select the appropriate threshold file
Answer any questions that are listed on that page. The answers to these questions are required so that during the processing of each performance file, we consume this information and pass this in to the tool for proper calculations. Click Next once finished.
On the Analysis Interval tab, select the interval that you would like to use. Note: The default of AUTO is recommended as that is the best performance option for running the tool. Any changes to this setting could cause the report generation process to be that much slower, but will allow you to get a little more granular if needed.

Click Next.
On the Output Options tab, you can select an Output Directory to store the PAL reports and what format you would like to use. Click Next once you have made your selections.
On the Queue tab, you will notice the parameters that will be passed in to the PAL tool for processing. Click Next if this satisfies your needs.
On the Execute tab, you can execute what you have just added to the queue or you could add more items to the queue for processing.
Click the button to execute the queued items.

This is a resource intensive application while these perfmon files are being parsed, so I would recommend using your fastest machine to run these reports on. Once PAL has completed processing the queued items, an IE window will open for each report in the queue.

I hope you have found this information useful and if you should have any questions regarding the tools usage, any possible problems that you may run in to or just suggestions to improve the tool, you can email paltool@microsoft.com

Happy reporting!!!

Mike

Additional Resources

For the latest XML updates that are not included with the current released version of the PAL tool, see http://blogs.technet.com/mikelag/archive/2008/08/20/xml-updates-for-the-pal-tool.aspx.

XML updates for the PAL tool

mikelag — Thu, 21 Aug 2008 02:53:51 GMT

Here you will find the latest XML updates to the PAL tool for Exchange 2003 and Exchange 2007. See below for more information

Exchange 2003 XML Updates

- Download Interim update here (Last updated 8-15-2008)

5-25-2008 Updates
- Added Slow QP threads, Slow Search Threads and Search Task Rate counters to detect deep folder traversal searches
8-15-2008 Updates
- Updated Available Memory Trend calculations
- Updated Processor and Processor Queue Length calculations
- Added Network\Outbound Packet Errors

2007 XML Updates

- Download Interim update here (Last Updated 9-16-2008)

5-2-2008 Updates
- Updated Mailbox Role for SP1 counters due to Database counter renames
8-4-2008 Updates
- Updated All Counters sets to fix .NET memory issue, minor changes/updates and updated Roles with all counters from our published counters listed at http://technet.microsoft.com/en-us/library/bb201720(EXCHG.80).aspx
8-15-2008 Updates
- Updated Available Memory Trend calculations
- Updated Processor and Processor Queue Length calculations
- Added Network\Outbound Packet Errors
- Added new combined role XML that includes all HUB/CAS/MAILBOX role counters
8-25-2008
- Windows 2008 counters for Exchange 2007 are in the works. Stay tuned!!!!
8-30-2008
- Removed Single Role XML files and replaced it with one combined Exchange 2007 XML (Let me know if you still want individual role XML files)
9-16-2008
- Huge update to the XML files. Broke out the XML files again to work around the "Log row too long" error. Information overload for the logparser tool :) Also added a HUB/CAS combined XML file.

Mike

How to enable Exchange User Monitor tracing via the command line

mikelag — Wed, 20 Aug 2008 15:09:28 GMT

For troubleshooting Exchange user performance related issues or to help plan your design for Exchange 2007, Exchange User Monitor (Exmon) is a great utility to have in your tool bag and contains a wealth of knowledge on your current user activities. Exmon has been previously talked about at http://msexchangeteam.com/archive/2005/04/06/403409.aspx and some FAQs regarding the tool can be seen at http://msexchangeteam.com/archive/2005/06/13/406276.aspx. There is also another link which is much harder to find at http://technet.microsoft.com/en-us/library/bb508855(EXCHG.65).aspx which explains how to use the tool in much greater detail.

Exmon tracing uses the ETW (Event Tracing for Windows) facility of Windows to send internal application event data to .etl files for later analysis. Detailed information regarding this tracing can be found in the Windows DDK at http://msdn2.microsoft.com/en-us/library/aa468736.aspx and also in http://msdn2.microsoft.com/en-us/library/aa363668.aspx

ETW tracing uses event providers that were specifically written for an application or driver and can be referenced either by its name or by its GUID. To view a list of current providers, run logman query providers from a command prompt which will give you a list similar to the below screenshot.

Notice the Exchange Information Store provider with its GUID of 2EACCEDF-8648-453e-9250-27F0069F71D2. This is the provider that ExMon uses for its tracing and is also the information that is needed later on this article to enable ETW tracing from a command line. Please note that this article will work on Exchange 2003 or Exchange 2007 servers as the provider GUID has not changed.

Prerequisite

Before enabling ExMon tracing, the following registry keys must be added to the registry to allow Exmon to collect data in the ETL file.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem]
"RpcEtwTracing"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Trace]
"UsePerformanceClock"=dword:00000001

To enable tracing on any given server, there is essentially 2 methods that can be used to create and start/stop Exmon tracing.

Method 1

Tracelog

1. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en

2. Create a directory called Tracing (ex. C:\tracing)

3. Copy tracelog.exe to this directory from the default install location of c:\program files\Resource Kit.

4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -start Exmon_Trace -f c:\Tracing\Exmon_trace.etl -seq 3500 -guid control.guid

5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -stop Exmon_Trace

6. Create a control.guid file and then add the appropriate GUID for Exmon tracing. Note: This GUID should be the only piece of information in this file. Save this to the location created in Step 2.

2EACCEDF-8648-453e-9250-27F0069F71D2

7. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog -l and look for an entry call Exmon_Trace. If this is found in the list, then the tracing has been enabled as shown below.

8. Run stop_tracing.cmd to stop the tracing.

Method 2

Logman (Built in to the OS)

---------------

1. Create a directory called Tracing (ex. C:\tracing)

2. Create the trace log using the "Exchange Information Store" provider that Exmon tracing uses. The GUID or the name of the provider can be used as shown below.
logman create trace Exmon_Trace -p {2EACCEDF-8648-453e-9250-27F0069F71D2} -nb 3 25 -bs 3 -o c:\Tracing\

logman create trace Exmon_Trace -p "Exchange Information Store" -nb 3 25 -bs 3 -o c:\Tracing\

3. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.

Logman start Exmon_Trace

4. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
logman stop Exmon_Trace

To create a schedule to start/stop these batch files at a particular time that you would like to specify, you could use the schtasks.exe command to do this. For more information on schtasks.exe syntax, follow the steps in http://support.microsoft.com/?id=814596 .

Mike

Perfwiz replacement for Exchange 2007

mikelag — Fri, 02 May 2008 13:19:00 GMT

Here is a replacement for general Exchange 2007 Performance data gathering. Use this in place of Perfwiz.exe as this utility does not capture all of the counters needed to properly troubleshoot Exchange 2007 performance issues.

Follow the steps below to enable these performance counters on any given Exchange 2007 server

Windows 2003 version of Perfwiz

Extract the contents of Exchange_2007_Perfwiz.zip to your Exchange server.
Open Performance Monitor
Expand Performance Logs and Alerts and select Counter Logs.
Right click Counter logs and select "New Log Settings From". Select the htm file that was extracted in Step 1. Click OK
Select the Log Files tab and click the Configure button
For the Log location, change this to a location of your choice. Click Ok
Click OK to save the Performance Counter log.
To start the Perfmon log, right click "Exchange 2007 Perfwiz" and then select Start.
Let this run during the problem period where performance is affected
Stop the perfmon log by right-clicking "Exchange 2007 Perfwiz" and selecting Stop
Make arrangements with a CSS representative to get the files analyzed.

Windows 2008 version of Perfwiz

Download the appropriate version of Perfwiz for your server. Currently I have 2 versions.
- Exchange 2007 Perfwiz (MBX-CAS-HUB specific) - This file has targeted counters instead of including all (*) counters for obvious performance reasons and easier parsing.
- Exchange 2007 Perfwiz (All Counters). Use this at your own discretion as this will collect a *lot* of data which may decrease performance on an Exchange Server.
- Other Role based XMLs are self-explanatory
  
  How to download
  To download these XML files to your computer, right click the file of your choice, select Save Target As... , and then save it to a directory location of your choice on your Exchange Server
  
  Role Based
  Use these as a high level look in to how the server is performing and if you need to branch out with more counters, use the Full Counter/Instance set below.
  
  Exchange_2007_Perfwiz-AIO.xml (HUB/CAS/MBX All-In-One)
  Exchange_2007_Perfwiz-CAS.xml
  Exchange_2007_Perfwiz-MBX.xml
  Exchange_2007_Perfwiz-HUB.xml
  Exchange_2007_Perfwiz-HUB-CAS.xml
  Exchange_2007_Perfwiz-UM.xml
  
  All Counters/All Instances
  Use this counter set at your own discretion as this could potentially cause performance degradation on your server trying to log this amount of counters.
  
  Exchange_2007_Perfwiz-Full.xml
Open Performance Monitor
Expand Reliability and Performance and then expand Data Collector Sets
Right click User Defined, Select New, and then Data Collector Set
Enter a unique name for this Data Collector set (ie. ExPerfwiz), select Create from a template (Recommended) and then click Next
Select the Browse button, navigate to the XML file that was saved in Step 1, select Open
Select Next on the next screen
Enter in a root Directory of where you would like to store the performance log files. Click Next
If you need to run this performance log under different credentials, enter it on this page. Click Finish

Recognition

I'd like to thank Ben Winzenz for creating the Windows 2008 *full* version of Perfwiz and John Rodriguez for getting me the counter sets for the All-In-One XML which I doctored up in a custom XML file.

Updates
2/13/2009 - Updated all XML files to include counters from Monitoring Without System Center Operations Manager

9/28/2009 - Added TCPv4 and TCPv6 counters to all Perfwiz counter sets

10/15/2009 – Large update to the XML files. See below what has been added

Role: HUB, CAS, HUB-CAS, UM, MB, AIO

\Memory\*
\Netlogon(*)\*
\Process(*)\*
\Processor(*)\*
\Redirector\*
\Server\*
\System\*
\MSExchange ADAccess Domain Controllers(*)\LDAP Searches timed out per minute
\MSExchange ADAccess Domain Controllers(*)\Long running LDAP operations/Min
\MSExchange ADAccess Domain Controllers(*)\Number of outstanding requests
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Read calls/Sec
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Read Time
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Search calls/Sec
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Search Time
\MSExchange ADAccess Local Site Domain Controllers(*)\LDAP Searches timed out per minute
\MSExchange ADAccess Local Site Domain Controllers(*)\Long running LDAP operations/Min
\MSExchange ADAccess Local Site Domain Controllers(*)\Number of outstanding requests

Role: HUB, CAS, HUB-CAS, UM, AIO
\MSExchange Store Interface(*)\ConnectionCache num caches
\MSExchange Store Interface(*)\ConnectionCache out of limit creations
\MSExchange Store Interface(*)\ConnectionCache total capacity
\MSExchange Store Interface(*)\ExRPCConnection creation events
\MSExchange Store Interface(*)\ExRPCConnection disposal events
\MSExchange Store Interface(*)\ExRPCConnection outstanding

Role: CAS, HUB-CAS, AIO
\MSExchange OWA\AS Queries Failure %
\MSExchange OWA\Average Search Time
\MSExchange OWA\Failed Requests/sec
\MSExchange OWA\Logons/sec
\MSExchange OWA\Proxy Response Time Average
\MSExchange OWA\Proxy User Requests/sec
\MSExchange OWA\Store Logon Failure %

Full
Netlogon(*)\*
\MSExchange ADAccess Local Site Domain Controllers(*)\*

Windows 2003 version of Perfwiz for Exchange 2007
Netlogon(*)\*

10/21/2009 – Updated Database Instance Counters to include all instances. Affects AIO and MBX Role XML
11/4/2009 – Added the following Physical Disk Counters to all XML files.

\PhysicalDisk(*)\Avg. Disk Queue Length
\PhysicalDisk(*)\Avg. Disk sec/Read
\PhysicalDisk(*)\Avg. Disk sec/Write
\PhysicalDisk(*)\Disk Reads/sec
\PhysicalDisk(*)\Disk Transfers/sec
\PhysicalDisk(*)\Disk Writes/sec