Mike Lagase : Tracing

How to enable Store PFD tracing upon Reboot

mikelag — Fri, 22 Aug 2008 16:59:37 GMT

Tracing Store startup problems upon server reboot is not an easy task as ExTRA tracing does not survive the server being rebooted whereas in Exchange 2003, the “Enable Tracing” key did survive a reboot.

After enabling an ExTRA session for PFD store startup tracing and looking at the contents of what was enabled when doing this, you will see the following from a command prompt.

Notice the “Microsoft Exchange Server 2007” provider with its GUID of 79bb49e6-2a2c-46e4-9167-fa122525d540. This is the provider that Exchange 2007 uses for most of its tracing

So now we need a way to trace this out. First, we need to create a method to enable logging and another way to enable the logging upon reboot of the server

Enable Tracing

Method 1

Logman (Built in to the OS)

---------------

1. Create a directory called Tracing (ex. C:\tracing)

2. Create the trace log using the “Microsoft Exchange Server 2007” provider that Exchange 2007 tracing uses. The GUID or the name of the provider can be used as shown below.

logman create trace ExchangeDebugTraces -p {79bb49e6-2a2c-46e4-9167-fa122525d540} -nb 3 25 -bs 3 -o c:\Tracing\

logman create trace ExchangeDebugTraces -p " Microsoft Exchange Server 2007" -nb 3 25 -bs 3 -o c:\Tracing\

After performing this, you can see this using a logman query command.

3. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config

TraceLevels:Pfd
Store:tagPFDServerStart
FilteredTracing:No
InMemoryTracing:No

Note: You could also create this same file by using the ExTRA interface to enable this tracing as well, thus populating the same information as listed above.

4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.

Logman start ExchangeDebugTraces
Running logman query shows this trace running.

5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

logman stop ExchangeDebugTraces

Method 2

Tracelog

6. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en

7. Create a directory called Tracing (ex. C:\tracing)

8. Copy tracelog.exe to this directory from the default install location of c:\program files(x86)\Resource Kit.

9. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -start ExchangeDebugTraces -f c:\Tracing\ExchangeDebugTraces.etl -seq 3500 -guid control.guid

10. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -stop ExchangeDebugTraces

11. Create a control.guid file and then add the appropriate GUID for Microsoft Exchange Server 2007 tracing. Note: This GUID should be the only piece of information in this file. Save this to the c:\tracing directory.

79bb49e6-2a2c-46e4-9167-fa122525d540

12. Create an EnabledTraces.Config file and add the following information to it and then save this file to the root of your C: drive. C:\EnabledTraces.Config

TraceLevels:Pfd
Store:tagPFDServerStart
FilteredTracing:No
InMemoryTracing:No

13. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog –q ExchangeDebugTraces to check if we are now tracing

This looks good. We are now happily tracing using PFD store startup tracing.

14. Run stop_tracing.cmd to stop the tracing.

Enable Logging upon bootup

On Windows 2003 and Windows 2008 servers, we can enable this tracing when the server is rebooted by using the built-in utility called Schtasks. More information regarding this tool can be found in the following locations

Windows 2008 schtasks help
Windows 2003 schtasks help

To create a new scheduled task, open a command prompt and type the following:

schtasks /create /tn "Start Store PFD Tracing Upon Reboot" /tr "C:\tracing\start_tracing.cmd" /sc onstart /ru ""

Note: Specifying "" for the /RU switch, tells the task to run as SYSTEM.

To verify that the scheduled task has been added properly, type schtasks from a command prompt. This should return the 'TaskName" and the "Next Run Time" of "At system start up"

You could also view this in the Control Panel under Scheduled Tasks

Reboot the server and the tracing will begin before the Information Store service starts.

Once the tracing has been collected, it is important to stop tracing and then remove this schtask to prevent this tracing from occurring in the future upon reboots.

To stop tracing, simply launch the stop_tracing.cmd file in the c:\tracing directory.

To remove the scheduled event, type the following command:

schtasks /delete /TN "Start Store PFD Tracing Upon Reboot"

Unfortunately, the trace file cannot be read unless you are working with Microsoft Support representative, but at least this will give you some ways that will help catch problems as they occur and when they occur. These methods can be applied for any ExTRA tracing in general, so this is not just limited to tracing Store Startup issues.

Happy Troubleshooting!!

Mike

Using ExTRA to find long running transactions inside Store

mikelag — Thu, 21 Aug 2008 17:40:00 GMT

During your normal flow of troubleshooting performance for Exchange, you may have the need to see what functions might be consuming higher than normal resources on your Exchange server. These functions might be causing spikes in CPU usage or sustained CPU usage over time.

Luckily, the Exchange Troubleshooting Assistant could help you gather the necessary data to understand what is going on inside store without having the need to dump the store process and then calling Support for analysis of this file.

Mind you, this doesn't cover every scenario, but will help to bring out some of the more common issues you may see with latent calls. Here is a short list of what we could report on:

Slow LSASS calls
Slow Directory lookups
Long running Virus Scanner calls
IMAIL conversion
Jet Snapshot calls taking a long time
Slow calls (read/write) to the database

With that said, let's get on to running the tool. It is important to note that this tool should be run during the time of the problem, not after as whatever it was causing the issue, is now gone.

Start ExTRA

To start the tool, you can do this either of two ways.

Using the Performance Troubleshooter tool in the Exchange Management Console (EMC) under Toolbox
Directly launching ExTRA.Exe from the \program files\Microsoft\Exchange Server\bin directory

Click on the "Select a Task" option in the menu on the left, Then select the "Performance Troubleshooter" in the list of tasks.

Configuring ExTRA for a new session

Once the Performance Troubleshooter has been selected, Enter an identifying name for this run. Click Next

In the drop down list, select "The number of outstanding RPC requests is high". Click Next

Enter the appropriate Exchange server name and Global Catalog Server name. Note: If you are logging on from an account that does not have permissions to the Exchange server, select the "Show advanced login options" to enter a different set of credentials. Click Next when done.

Select a location to store the Performance logs

Select "Show advanced data collection options"

Make sure that "Collect Function log (FCL) data is selected. Click Next

The trace will the start and collect data for 3 minutes and then perform some statistical analysis on the data collected.

You will then be presented with the results of the analysis and then tell you if any function calls are consuming an exorbitant amount of time. If anything if found, it will be reported here. In this case, there was not a problem on the server, but if there was, it would be reported right here.

This may give you some immediate clues in less than 5 minutes on what might be occurring inside the store process.

Happy Troubleshooting!!

How to enable Exchange User Monitor tracing via the command line

mikelag — Wed, 20 Aug 2008 15:09:28 GMT

For troubleshooting Exchange user performance related issues or to help plan your design for Exchange 2007, Exchange User Monitor (Exmon) is a great utility to have in your tool bag and contains a wealth of knowledge on your current user activities. Exmon has been previously talked about at http://msexchangeteam.com/archive/2005/04/06/403409.aspx and some FAQs regarding the tool can be seen at http://msexchangeteam.com/archive/2005/06/13/406276.aspx. There is also another link which is much harder to find at http://technet.microsoft.com/en-us/library/bb508855(EXCHG.65).aspx which explains how to use the tool in much greater detail.

Exmon tracing uses the ETW (Event Tracing for Windows) facility of Windows to send internal application event data to .etl files for later analysis. Detailed information regarding this tracing can be found in the Windows DDK at http://msdn2.microsoft.com/en-us/library/aa468736.aspx and also in http://msdn2.microsoft.com/en-us/library/aa363668.aspx

ETW tracing uses event providers that were specifically written for an application or driver and can be referenced either by its name or by its GUID. To view a list of current providers, run logman query providers from a command prompt which will give you a list similar to the below screenshot.

Notice the Exchange Information Store provider with its GUID of 2EACCEDF-8648-453e-9250-27F0069F71D2. This is the provider that ExMon uses for its tracing and is also the information that is needed later on this article to enable ETW tracing from a command line. Please note that this article will work on Exchange 2003 or Exchange 2007 servers as the provider GUID has not changed.

Prerequisite

Before enabling ExMon tracing, the following registry keys must be added to the registry to allow Exmon to collect data in the ETL file.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem]
"RpcEtwTracing"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Trace]
"UsePerformanceClock"=dword:00000001

To enable tracing on any given server, there is essentially 2 methods that can be used to create and start/stop Exmon tracing.

Method 1

Tracelog

1. Download and install tracelog from http://www.microsoft.com/downloads/details.aspx?FamilyID=55E51B3B-6C26-4CA0-ABF1-0E51D92B8298&displaylang=en

2. Create a directory called Tracing (ex. C:\tracing)

3. Copy tracelog.exe to this directory from the default install location of c:\program files\Resource Kit.

4. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -start Exmon_Trace -f c:\Tracing\Exmon_trace.etl -seq 3500 -guid control.guid

5. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.

tracelog.exe -stop Exmon_Trace

6. Create a control.guid file and then add the appropriate GUID for Exmon tracing. Note: This GUID should be the only piece of information in this file. Save this to the location created in Step 2.

2EACCEDF-8648-453e-9250-27F0069F71D2

7. To verify if these batch files work successfully, run start_tracing.cmd and then run tracelog -l and look for an entry call Exmon_Trace. If this is found in the list, then the tracing has been enabled as shown below.

8. Run stop_tracing.cmd to stop the tracing.

Method 2

Logman (Built in to the OS)

---------------

1. Create a directory called Tracing (ex. C:\tracing)

2. Create the trace log using the "Exchange Information Store" provider that Exmon tracing uses. The GUID or the name of the provider can be used as shown below.
logman create trace Exmon_Trace -p {2EACCEDF-8648-453e-9250-27F0069F71D2} -nb 3 25 -bs 3 -o c:\Tracing\

logman create trace Exmon_Trace -p "Exchange Information Store" -nb 3 25 -bs 3 -o c:\Tracing\

3. Create a start_tracing.cmd file and add the following information to the file. Save this to the location created in Step 1.

Logman start Exmon_Trace

4. Create a stop_tracing.cmd file and add the following information to the file. Save this to the location created in Step 2.
logman stop Exmon_Trace

To create a schedule to start/stop these batch files at a particular time that you would like to specify, you could use the schtasks.exe command to do this. For more information on schtasks.exe syntax, follow the steps in http://support.microsoft.com/?id=814596 .

Mike