So you’ve managed to get a dump from a process… now what?
Dump analysis is a skill that requires a bit of knowledge of how processors work, how to read assembly language, how functions are called, what stacks and heaps are, and so on – it’s way beyond the scope of a blog to give you this set of skills. But I can show you the guaranteed requirements to make any sense of the .DMP files…
A .DMP file is food for a debugger, and only a debugger. A more correct term for “debugger” is “debugging tool” – as it is you playing the part of the debugger, such a task is way beyond the capacity of a program.
Microsoft have a free debugger for Windows – the Debugging Tools for Windows comes in x86 and x64 flavours depending on your architecture. In this suite of debuggers the one most commonly used interactively is Windbg.exe, as it has a Windows-style UI and can have its appearance modified to suit through the use of ‘workspaces’.
The debugger itself doesn’t go out of its way to help you as it can be used for so many things – the typical crash dump analysis first step, though, is to enter the command “!analyze –v” to get a summary of the exception information and state of the CPU registers when the program crashed.
If a hang mode dump was produced, I hope it is obvious that !analyze cannot give you anything useful as there is no exception to interpret.
Whether you load a crash or hang dump, Windbg will sit looking at you, waiting for input at a prompt looking something like this: 0:001>
Similar to a command prompt, you have to give it something to do in order to get any use out of it… the inevitable question is “what do I type?” which is countered with the question “what do you want to look at?”.
Before we dig into that, the other vital pre-requisite to perform debugging is to have symbol files…
While processors work on a limited set of (very, very simplistic) instructions, most programming is not done at this level – it would take too long to produce anything useful and maintenance of the code, however well commented, would be practically impossible.
There are plenty of languages that can be used to create programs, for those which are compiled into a native instruction set (e.g. Intel x86/IA32, AMD64, Itanium) there is the option to create a set of “helper” files, called symbols.
A compiled binary does not often contain much to assist with debugging, it just has the raw (native) code and some some initialized variables (such as text strings) – it is not easy to work your way through a compiled version of even a simple program and work out what it is doing, because of how we branch or jump depending on the results of various calls or instructions.
Conversely, the source code written by the programmer gives a perfect idea of the flow of execution – even better if it has comments to let the reader know what was going through the programmer’s mind when he wrote it.
What the symbols provide is a way to debug the compiled binary with some information about where the entry points for functions are, and optionally what arguments they expect to be passed.
The symbols do not help with the execution of the program, so they do not have a place inside the binary – this would bloat them.
In the following examples, the commands entered into the debugger can be seen along with a slightly edited version of their output. Any colouring present is just for emphasis – the default output is rather dull and monochromatic.
When doing a user-mode debug:
“~*” means “for each thread do the following” “k” means “show the call stack”
Here is an attempt to look at the threads in a debug of calc.exe with no symbols:
0:004> ~* k
0 Id: f00.1794 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen Child-SP RetAddr Call Site 00000000`001ed4d8 00000000`76c8c95e USER32!SfmDxSetSwapChainStats+0x1a 00000000`001ed4e0 00000000`ff0b1a8c USER32!GetMessageW+0x2a 00000000`001ed510 00000000`ff0ca00f calc+0x1a8c 00000000`001efc00 00000000`76d8f56d calc+0x1a00f 00000000`001efcc0 00000000`76ec3281 kernel32!BaseThreadInitThunk+0xd 00000000`001efcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 …
2 Id: f00.9e0 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen Child-SP RetAddr Call Site 00000000`041dfb88 000007fe`fcfe10ac ntdll!ZwWaitForSingleObject+0xa 00000000`041dfb90 00000000`ff0b227e KERNELBASE!WaitForSingleObjectEx+0x9c 00000000`041dfc30 00000000`76d8f56d calc+0x227e 00000000`041dfc70 00000000`76ec3281 kernel32!BaseThreadInitThunk+0xd 00000000`041dfca0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 …
Now, after we set a symbols path and then force a reload, we can make a bit more sense of the thread stacks:
0:004> .sympath srv*C:\Symbols.pub*http://msdl.microsoft.com/download/symbols 0:004> .reload /f 0:004> ~* k
0 Id: f00.1794 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen Child-SP RetAddr Call Site 00000000`001ed4d8 00000000`76c8c95e USER32!NtUserGetMessage+0xa 00000000`001ed4e0 00000000`ff0b1a8c USER32!GetMessageW+0x34 00000000`001ed510 00000000`ff0ca00f calc!WinMain+0x1dca 00000000`001efc00 00000000`76d8f56d calc!LDunscale+0x1ea 00000000`001efcc0 00000000`76ec3281 kernel32!BaseThreadInitThunk+0xd 00000000`001efcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d …
2 Id: f00.9e0 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen Child-SP RetAddr Call Site 00000000`041dfb88 000007fe`fcfe10ac ntdll!NtWaitForSingleObject+0xa 00000000`041dfb90 00000000`ff0b227e KERNELBASE!WaitForSingleObjectEx+0x79 00000000`041dfc30 00000000`76d8f56d calc!CTimedCalc::WatchDogThread+0x2e 00000000`041dfc70 00000000`76ec3281 kernel32!BaseThreadInitThunk+0xd 00000000`041dfca0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d …
The symbols path is ready from left to right, and each location is checked for a matching symbols file for each binary the debugger request. In the above example we have a local cache of public symbols in C:\Symbols.pub – if a symbols file is not found here then the debugger requests it from http://msdl.microsoft.com/download/symbols and then copies it into the local cache for next time.
Symbols are also specific to versions of binaries, as a change to the source code almost always modifies the entry points for functions so attempts to use wrong symbols doesn’t necessarily help, but lead to more confusion.
With symbols present it is possible to “browse” the modules, set breakpoints or unassemble the code using labels rather than meaningless memory addresses.
The ‘x’ command is “examine symbols”, and takes a module name followed by a ‘!’ (bang) and a function name – it is fine to use multiple wildcards either side of the bang if you’re not sure of where a function is or its exact name:
0:004> x calc!LDunscale 00000000`ff10d08c calc!LDunscale = <no type information>
0:004> x calc!Win* 00000000`ff0cd31c calc!WinMain = <no type information> 00000000`ff10b0cc calc!WinSqmAddToStream = <no type information> 00000000`ff0c9918 calc!WinSqmAddToStreamEx = <no type information> 00000000`ff10b0d8 calc!WinSqmIncrementDWORD = <no type information> 00000000`ff0cb9b8 calc!WinMainCRTStartup = <no type information>
0:004> x calc!*angle* 00000000`ff125ad0 calc!CContainer::m_iAngle = <no type information> 00000000`ff0c59cc calc!IdcSetAngleTypeDecMode = <no type information> 00000000`ff112080 calc!_imp_GdipFillRectangleI = <no type information> 00000000`ff10b0b4 calc!GdipFillRectangleI = <no type information> 00000000`ff10adf4 calc!cosanglerat = <no type information>
Without symbols there is very little sense to be made of dump files, hence why we can’t provide much help when 3rd party programs are crashing or behaving strangely – without the source or at least the symbols there is not much to go on (especially if code has been optimized for speed or size, making it “not so obvious” to work out what it is doing).
Tools of the Trade
The above information isn’t much, but it gives you the environment to be able to start debugging – without any examples of crashed or hung processes that are not contrived, it is tricky to go much further.
One book I can recommend, however, for those interested in the topic:
Advanced Windows Debugging Mario Hewardt, Daniel Pravat
It has been a few years since I learned assembly language, so I don’t have any specific recommendations for books on this foundation topic, but an introduction to the Intel assembly language would be useful – having a good understanding of the way an Intel-compatible CPU handles code execution and data pointers is essential to be able to debug.