Capturing Application Crash Dumps

Capturing Application Crash Dumps

  • Comments 16
  • Likes

Following on from our posts on the Basic Troubleshooting Toolkit and Basic Debugging of an Application Crash, let's talk about actually capturing Application crash dumps and failures.  Most administrators are familiar with the Dr. Watson for Windows tool that has been around since the days of Windows NT.  An updated version of this tool, DrWtsn32, still exists in Windows XP and Windows Server 2003 - but not in Windows Vista or Windows Server 2008.  So how do we capture user-mode dump files?  We're going to cover several different methods for capturing dump files for User-mode application crashes.

First - let's quickly cover Dr. Watson for Windows XP and Windows Server 2003.  Dr. Watson captures user-mode dump information.  Whenever a user-mode process (such as Internet Explorer or the Print Spooler) crashes, Dr. Watson creates a text file, DrWtsn32.log.  Dr. Watson can also be configured to create a crash dump file that can be loaded into a debugger.  Let's look at the configuration for Dr. Watson.  The first thing we have to do is configure Dr. Watson as our default debugger.  To do this we run the following command: drwtsn32 -i.  What this does is modify two registry values located in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug.  The values are as follows:

  • Value Name = Auto
    Type = String (REG_SZ)
    Data Value = 1 or 0. (Default is 1)
  • Value Name = Debugger
    Type = String (REG_SZ)
    Data Value = drwtsn32 -p %ld -e %ld -g
    NOTE: This data value (drwtsn32 -p %ld -e %ld -g) is specific to Dr. Watson. Alternative debuggers will have their own values and parameters.

So now that Dr. Watson is our default debugger, it's time to go set our parameters.  Run the drwtsn32 command, to bring up the configuration options shown below:

DrWtsn323[1]The first two options are fairly self-explanatory - the location in which Dr. Watson should save the Log File and Crash Dump when they are generated.  By default, this is in the All Users profile path:
drive:\Documents and Settings\All Users.WINNT\Application Data\Microsoft\Dr Watson.  If you change these locations, you must ensure that all users have write permission to the new location.  Otherwise, users will be prompted to select a location in which to save the files.

The "Number of Instructions" parameter specifies how many instructions preceding and following the faulty instruction are included in the disassembly portion of the log file.  The possible values for this parameter range from 0 to 500.  The disassembly portion includes the function being executed when the error occurred, the memory address, raw machine instruction, and decoded machine instruction for each adjacent instruction and an analysis of the faulty instruction.  The default value is 10 (0xA in Hexadecimal).

The "Number of Errors to Save" parameter specified how many errors should be maintained in its application error viewer and in the Event Viewer application log.  By default, this value is 10 (0xA in Hexadecimal).  The possible values range from 0 to 4,294,967,295!  In reality though, you would not want to set the value this high.  When the number of recorded errors reaches the value of this entry, Dr. Watson will continue to add errors to the log file and the dump - but will not add errors to its own log viewer or the Application Event log until it is reset using the "Clear" button or the value is increased.

The "Dump Symbol Table" dumps the symbol table for each module.  Selecting this option can cause log files to become very large!  The "Dump all Thread Contexts" specifies whether Dr. Watson will log a state dump for each thread in the program that failed or only the faulting thread.  The other options are self-explanatory.  Each of the parameters  in this dialog are stored in the HKEY_LOCAL_MACHINE\Software\Microsoft\DrWatson key.  The Log File and Dump File path are not present by default - if you change the location of these files, you will see the Registry values for these options.

So that wraps up Dr. Watson - but what if you're running Windows Vista or Windows Server 2008, or you prefer to use something other than Dr. Watson?  There are two different options to use - DebugDiag or ADPlus.  For pre-Vista Operating Systems, you can also use Userdump.

Let's look at ADPlus first.  ADPlus is a VBScript that is included with the Debugging Tools for Windows.  ADPlus has two different modes - Hang and Crash.  ADPlus is configured through the use of command line switches to specify the parameters.  The basic command line switches are as follows:

Switch Function
-crash Runs ADPlus in crash mode
-hang Runs ADPlus in hang mode
-p <PID> Defines the Process ID to be monitored
-pn <ProcessName> Defines the Process Name to be monitored
-o <directory> Defines the directory where dumps & logs are created

So for example, to run ADPlus to monitor the Print Spooler and output the files to a folder on the D: drive, you would run the following command:  cscript adplus.vbs -crash -pn spoolsv.exe -o d:\Spooler_Dumps. A key feature to remember about ADPlus is that unlike Dr. Watson or DebugDiag, which we are about to discuss, it can capture crash dumps of a failing 64-bit process.  However, there are some caveats with ADPlus.  For example, when you use ADPlus in hang mode, you must wait until the process or processes stop responding before you run the script (unlike crash mode, hang mode is not persistent).  You can find more information regarding ADPlus usage in the Debugging Tools help file and Microsoft KB Article 286350.

Moving on, let's take a look at the Userdump tool.  Similar to ADPlus, Userdump can be used to generate a process dump of an application that has crashed or is hanging.  When you install Userdump, the setup program installs a kernel-mode driver.  Once the program is installed, launch the Process Dumper Applet from the Control Panel.  Click the "New" button and enter the name of the executable to monitor (for example WINWORD.EXE or SPOOLSV.EXE).  You do not need to enter the full path.  Once the application has been added to the monitor it should look like the screenshot on the right. UserDump_Rule1
UserDump_Rule2

Now, highlight the app and click the "Default Settings" button.  Ensure that the "All Exceptions" checkbox is not selected.  Click on the "Select All" button (which is different than selecting the "All Exceptions" checkbox).  Your configuration window should look like the screenshot to the left.

Let's discuss a couple of the other options here.  The option "Bugcheck after Dumping" will cause the machine to create a Kernel dump after creating the user-mode dump.  Be very careful with this option!

The other feature to note here is under the "Exit Monitor" section.  In some instances you will encounter processes that seem to close gracefully for no apparent reason.  If you check the "Monitor Process Exit" box, you will get a dump file created every time the process is closed.  Again - this is an option to use judiciously - even if a user closes the application being monitored deliberately you will create a dump file.  For an application like Internet Explorer, Word, or Excel, this could result in a large number of unnecessary dump files (and quite a bit of wasted disk space!)

Finally, let's look at the Debug Diagnostic Tool (aka DebugDiag).  DebugDiag was originally released as part of the IIS Diagnostic Toolkit.  Version 1.1 is a standalone tool that can be used to troubleshoot hangs or crashes in any 32-bit user-mode process.  Following the initial installation of DebugDiag, the configuration process is wizard-driven.  So let's go ahead and set up a rule to monitor our machine for spooler crashes.

image

Since IIS is not installed on this machine, all of the IIS options are disabled.  After selecting "Crash", and clicking "Next" we are presented with the option to select our Target Type - which could be a specific process (such as Internet Explorer, Excel, Outlook etc), a specific MTS / COM+ application, an IIS web application pool, or a specific NT service.  For our example, we will go ahead and select "A specific NT service" and click "Next".

image

Now we can select our target as shown below:

Select_Spooler

Remember that we can only use DebugDiag against 32-bit user-mode processes.  If you were to try and monitor the spooler on a 64-bit machine, you would receive an error similar to this:

image

After Clicking "Next", we are presented with the Advanced Configurations screen.  For our purposes, we leave this at the default settings, and click "Next", which allows us to name our rule, and select a location for our dump files (see the image below).  Once we have completed these options, we click next and activate our rule.  Now the Spooler process is being monitored for crashes.

Spooler_DumpOptions

If / When the spooler crashes, we will find a new folder created for each instance of the crash in the Userdump location listed above.  This will include the dump file and the log file.

Finally, there is one last option to capture a process dump that is available on Windows Vista.  If you open up Task Manager, you can right-click on an application name, the process name or the service and select the option to "Create Dump File".  This will create a dump of the process, but not terminate it - so you can capture multiple dumps of a running process!

And that brings us to the end of our post on Application Crash Dumps.  Until next time ...

Additional Resources:

 - CC Hameed

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Thank you guys, this is really helpful :)