Thoughts from the EPS Windows Server Performance Team
Hello there! My name is Mike and I am an Escalation Engineer with the Platforms team. I’ve been reading the AskPerf blog for a while and wanted to share some information about a very useful troubleshooting tool. A quick word of warning though – to really get the most out of this tool, you need to be comfortable with some pieces of debugging – in particular understanding what a stack is (we’ll go over a little bit of this in the next post). I know that we’ve talked a little bit about applications leaking memory and heap allocations in previous posts. Most administrators are fairly comfortable using tools such as Performance Monitor to do some relatively straightforward identification of memory leaks. However, where many administrators start running into problems is when they suspect a memory leak – but the culprit is an application that is developed in-house. Enter UMDH.EXE – a tool that is included with the Debugging Tools for Windows. Before we dig into using the tool though, let’s quickly go over a couple of quick caveats about Memory Leaks.
A perceived memory leak may not actually be a memory leak. No, we’re not trying to confuse you, or start an existential debate! The fact of the matter is that there are applications that may be perceived to be leaking that are actually consuming memory by design. Assuming that you actually do have a leak, capturing Performance Monitor Data may help you identify what is leaking, but generally does not provide more information beyond that. The key piece of data in definitively identifying the memory leak within an application is the thread stack. The thread stack is a list of functions. Trying to capture the exact thread stack that is leaking is difficult because threads execute so fast that knowing exactly when to dump the process is not an easy task.
This is where UMDH.EXE comes into play. UMDH stands for User Mode Dump Heap. UMDH is a lightweight standalone executable that does not require a separate installation routine beyond the installation of the Debugging Tools for Windows. Using UMDH you can find the exact set of function calls responsible for the memory consumption. UMDH make use of the Global Flags (GFLAGS) tool settings. Below are the steps for using UMDH:
The first thing you need to do is enable a setting in GFLAGS called “Create user mode stack trace database” for the process that you are investigating. This setting creates a registry value which is read when the process starts up that allows the system to keep track of the functions allocating memory inside the process. You can enable this value either via the GUI or via the command-line. When enabling this value through the GUI, remember to switch to the Image File tab (as shown below) – in this case, I am selecting NOTEPAD.EXE as the process to watch:
The syntax to set this value from the command line is: gflags.exe -i notepad.exe +ust. If the process is already running, you will need to restart the process for this change to take effect.
In addition to setting the Global Flags setting for the process, you will also need to configure the symbol path. You can configure the Symbol Path by setting an environment variable within the properties of “My Computer”. The environment variable name is _NT_SYMBOL_PATH. To set up a symbol path to point to the Microsoft public symbol server, set the variable as follows:srv*c:\symbols*http://msdl.microsoft.com/download/symbols. This creates a local symbol store at C:\Symbols that is populated from the public symbol server. Having this symbol path set allows UMDH to resolve binary information to friendly function names.
Once we have our configuration settings in place, we can take our UMDH snapshots. Taking a snapshot is a straightforward process – the one thing you will need is the Process ID (which you can find by using Task Manager). The syntax is as follows: umdh –p:<PID> –f:<filename>.txt. The –p switch specifies the Process ID and the –f switch specifies the file name for the snapshot file. After you have allowed some time to pass (and the process has leaked some memory), you can take your second snapshot using the same syntax as above. When taking snapshots, it is important to remember that each output file is a discrete entity – data is not appended to the end of each file. So if you set up a batch file to run every five minutes, ensure that the output file name is different for each snapshot.
Once you have at least two snapshots you can use UMDH to compare the two files and create a more useful output file using the following command: umdh -v<filename1>.txt <filename2>.txt > comparefiles.txt.
Assuming that you have set up all the configuration pieces correctly, the output in the output file that you used to capture the comparison data should have text along the lines of what is below in it. In my example I used a Windows Server 2008 x64 machine and captured two quick snapshots of Notepad.exe:
// Debug library initialized ...
DBGHELP: NOTEPAD - public symbols
DBGHELP: ntdll - public symbols
DBGHELP: kernel32 - public symbols
DBGHELP: ADVAPI32 - public symbols
DBGHELP: RPCRT4 - public symbols
DBGHELP: GDI32 - public symbols
DBGHELP: USER32 - public symbols
DBGHELP: msvcrt - public symbols
DBGHELP: COMDLG32 - public symbols
DBGHELP: SHLWAPI - public symbols
DBGHELP: COMCTL32 - public symbols
DBGHELP: SHELL32 - public symbols
DBGHELP: WINSPOOL - public symbols
DBGHELP: ole32 - public symbols
DBGHELP: OLEAUT32 - public symbols
DBGHELP: IMM32 - public symbols
DBGHELP: MSCTF - public symbols
DBGHELP: LPK - public symbols
DBGHELP: USP10 - public symbols
DBGHELP: uxtheme - public symbols
DBGHELP: RTSUltraMonHook - export symbols
DBGHELP: RTSUltraMonHookRes - no symbols loaded
DBGHELP: MSIMG32 - public symbols
DBGHELP: CLBCatQ - public symbols
// Each log entry has the following syntax:
// + BYTES_DELTA (NEW_BYTES - OLD_BYTES) NEW_COUNT allocs BackTrace TRACEID
// + COUNT_DELTA (NEW_COUNT - OLD_COUNT) BackTrace TRACEID allocations
// ... stack trace ...
// BYTES_DELTA - increase in bytes between before and after log
// NEW_BYTES - bytes in after log
// OLD_BYTES - bytes in before log
// COUNT_DELTA - increase in allocations between before and after log
// NEW_COUNT - number of allocations in after log
// OLD_COUNT - number of allocations in before log
// TRACEID - decimal index of the stack trace in the trace database
// (can be used to search for allocation instances in the original
// UMDH logs).
As you can see from the output, any Windows binaries for which public symbols (SHELL32.DLL, NTDLL.DLL etc) are available has a path to the symbols listed. For any binaries for which symbols are not available from my source symbol server, there is no path listed.
With that, we’re going to bring this post to a close. In a future post, we'll go over troubleshooting a memory leak using UMDH.EXE.
- Mike Morales
Good post about UMDH, Mike! Wonder if this may also lead into a discussion about DebugDiag and the Memory Pressure Analyzers? (Granted, we're awaiting the x64 version of DebugDiag, while apparently UMDH works with 64-bit processes...)
Its a great information Mike thanks for sharing it. It would be great if you can demonstrate and give the explaination about the memory leak using Performance Monitor.
Its really a great piece of Information. It would be really great if you can give a screenshoted demonstration for Identifieng memory Leak's using
I get kern1l gflags: error: 5 when trying to set UST
What about "In a future post, we'll go over troubleshooting a memory leak using UMDH.EXE." ?
The future is now.
I'm using umdh tool to find memory leaks.
when I take diff. of 2 log(dump) files, then in the diff. file, full path of the reference file is truncated. So getting exact location is difficult from it.
Is there any settings in umdh, where I can put maximum allowable path length?
Is there any limitation in using the 64bit UMDH for monitoring mixed mode processes?
Even after enabling the stack trace, UMDH still reports following
Warning: (Restarted the process after enabling the trace)
Warning: UMDH didn't find any allocations that have stacks collected.
Warning: Use gflags to enable allocation stack collection.
Warning: Restart the application for the setting to be in effect.
Warning: A 64bit GFLAGS must be used. The command is:
Warning: gflags -i KTJOBO~1.EXE +ust