Hi there, this is Mark with the Windows Performance Team.  This post is a quick introduction to a new performance tracing tool called XPerf.  Moving forward with Windows Vista and Windows Server 2008, this powerful tool will hopefully be utilized more often.  XPerf is part of the Windows Performance Toolkit (WPT) which can be downloaded from the Microsoft web site.

XPerf is designed for Windows Vista and Windows Server 2008, but will run on Windows XP and Windows Server 2003 by copying Xperf.exe and Perfctrl.dll to the target machine.  XPerf uses ETW (Event Tracing for Windows) and is very low overhead, requiring only about 1500-2000 clock cycles per log.  As an example, 20,000 calls per second is less than 2% CPU on a 2.0 GHz processor.  While the data collection is running, the XPerf tools are not even loaded - the kernel itself is collecting the data.  All analysis is done in post processing. Since Xperf is ETW based, sample profiling can be started and stopped at any time, without stopping or restating even a single process.  You can profile anything at any time on any system.  The traces are logged in a circular fashion, based on the buffer size that is set.  The basic process is this:

1. ETW tracing is enabled by using XPerf.
2. Operations are performed.
3. ETW tracing is disabled by using XPerf, and the data is saved to an ETL trace file.
4. Trace files can then be further processed by using XPerf or viewed by using Performance Analyzer (XPerfView).

Once a trace is taken, you can copy it to a Windows Vista or Windows Server 2008 machine for trace decode and viewing.  The traces are also cross-platform, so you can view them on Windows Vista or Windows Server 2008 (either x86 or x64) regardless of what type of machine they were taken on.

The following is a quick introduction and few examples. The image below shows the basic workflow:

 Some usage of this tool:

1. If DPC isn't running in Idle, then we can see what driver is responsible for that.
2. Stackwalk reports stack report and hierarchy module and function calls.
3. XPerf can start a new process from a given command line and use its process ID for the duration of the trace (-pidnewprocess)
4. Disk resource and activity including per process disk activities
5. Looking at Hard Faults we can see when I/O's are blocking a process.
6. Marks can be added during the trace for information. They can be viewed in the trace during the analysis.
7. Outstanding I/O's for different processes can be observed
8. Dynamic grouping capability when looking at the graphs
9. Can be used to check all registry access during the trace
10. All file activities can be traced
11. Heap allocation tracing
12. Using –i switch, specific items in trace can be parsed and saved into different formats including CSV, XML and plain text for further analysis. This switch also could be used to automate analyzing the XPerf trace files.
13. XPerfview.exe can be used to view graphs of the .etl trace log. 
14. Kernel and User providers can be used to take multiple traces. Using the merge switch, they can be merged into one .etl file.

In order to look at the modules and function calls from a trace that was taken with Stackwalk enabled, Symbols must be set in the environment variables. In order to configure symbols, you can use the following command:

   C:\>Set.exe  C:\WebSYMBOLS;srv*C:\WebSYMBOLS*HTTP://MSDL.MICROSOFT.COM/DOWNLOAD/SYMBOLS

To view all providers;

    Xperf.exe -providers

To view only Kernel Providers (including Kernel Groups and Kernel Flags);

   Xperf.exe –providers K

Here’s the list of all Kernel Flags

There are a ton of stack walking flags, so instead of listing them all, please refer to the following MSDN link for more info:

http://msdn.microsoft.com/en-us/library/windows/desktop/ff191012(v=vs.85).aspx

 Please note: the stackwalk feature currently doesn’t work on Windows Server 2003 and Windows XP since their older kernel doesn't support it.

Here are some examples:

Example 1 - Start a trace with BASE Kernel Group and cswitch Kernel Flag and then stop the trace and save it in mytrace.etl file;

                xperf -on base+cswitch

                 xperf -d mytrace.etl

    Then use this command to view the information about the trace itself (-detail at the end will give more detailed information about the trace)

                 xperf -i mytrace.etl -a tracestats -detail

    This will report the processes information(Start Time, End Time, Process, DataPtr, Process Name (PID), ParentPID, SessionID, UniqueKey) in the trace;

                 xperf -i mytrace.etl -a process

    Finally, redirect the output to a CSV file;

                xperf -i mytrace.etl -o C:\xperf-process.csv -a process

Example 2 - To start user mode logging and kernel logging at the same time:

               xperfinfo –on base+Network –f kernel.etl -start UserTrace -on Microsoft-Windows-MeetingSpace+Microsoft-Windows-Firewall -f user.etl

Explanation: A kernel trace is started. The kernel session does not need a session name to be specified because its name is unique. The groups Base and Network are enabled on the kernel provider. This trace will be collected in a file called Kernel.etl

At the same time, a user trace named UserTrace is started and the provider’s Microsoft-Windows-MeetingSpace and Microsoft-Windows-Firewall are enabled to it. This trace will be collected in a file called User.etl

Using the following command, the UserTrace session is stopped. The two user-mode providers no longer produce events to this session (they might still be generating events to other independent sessions).

                xperfinfo –stop UserTrace

   Stop the kernel session:

                xperf -stop

    Merge the user and kernel traces into a single trace called System.etl:

                xperf -merge user.etl kernel.etl system.etl

   Post-process the binary trace file System.etl into an ANSI text file called System.txt:

                xperf -i system.etl -o system.txt -a dumper 

Additional Resources:

• Windows Performance Tools Kit, v.4.1.1 (QFE)
• Windows Performance Toolkit (WPT)
• Using Xperf to take a Trace
• Xperf support for XP

We’ll be writing more about this tool and how to troubleshoot certain scenarios in future posts. There are several new powerful tools available for troubleshooting and diagnostics – XPerf is only one of them.  See you next time!

- Mark Ghazai

Share this post :