Thoughts from the EPS Windows Server Performance Team
Useful Microsoft Blogs
Following on from our posts on the Basic Troubleshooting Toolkit and Basic Debugging of an Application Crash, let's talk about actually capturing Application crash dumps and failures. Most administrators are familiar with the Dr. Watson for Windows tool that has been around since the days of Windows NT. An updated version of this tool, DrWtsn32, still exists in Windows XP and Windows Server 2003 - but not in Windows Vista or Windows Server 2008. So how do we capture user-mode dump files? We're going to cover several different methods for capturing dump files for User-mode application crashes.
First - let's quickly cover Dr. Watson for Windows XP and Windows Server 2003. Dr. Watson captures user-mode dump information. Whenever a user-mode process (such as Internet Explorer or the Print Spooler) crashes, Dr. Watson creates a text file, DrWtsn32.log. Dr. Watson can also be configured to create a crash dump file that can be loaded into a debugger. Let's look at the configuration for Dr. Watson. The first thing we have to do is configure Dr. Watson as our default debugger. To do this we run the following command: drwtsn32 -i. What this does is modify two registry values located in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug. The values are as follows:
So now that Dr. Watson is our default debugger, it's time to go set our parameters. Run the drwtsn32 command, to bring up the configuration options shown below:
The first two options are fairly self-explanatory - the location in which Dr. Watson should save the Log File and Crash Dump when they are generated. By default, this is in the All Users profile path:drive:\Documents and Settings\All Users.WINNT\Application Data\Microsoft\Dr Watson. If you change these locations, you must ensure that all users have write permission to the new location. Otherwise, users will be prompted to select a location in which to save the files.
The "Number of Instructions" parameter specifies how many instructions preceding and following the faulty instruction are included in the disassembly portion of the log file. The possible values for this parameter range from 0 to 500. The disassembly portion includes the function being executed when the error occurred, the memory address, raw machine instruction, and decoded machine instruction for each adjacent instruction and an analysis of the faulty instruction. The default value is 10 (0xA in Hexadecimal).The "Number of Errors to Save" parameter specified how many errors should be maintained in its application error viewer and in the Event Viewer application log. By default, this value is 10 (0xA in Hexadecimal). The possible values range from 0 to 4,294,967,295! In reality though, you would not want to set the value this high. When the number of recorded errors reaches the value of this entry, Dr. Watson will continue to add errors to the log file and the dump - but will not add errors to its own log viewer or the Application Event log until it is reset using the "Clear" button or the value is increased.
The "Dump Symbol Table" dumps the symbol table for each module. Selecting this option can cause log files to become very large! The "Dump all Thread Contexts" specifies whether Dr. Watson will log a state dump for each thread in the program that failed or only the faulting thread. The other options are self-explanatory. Each of the parameters in this dialog are stored in the HKEY_LOCAL_MACHINE\Software\Microsoft\DrWatson key. The Log File and Dump File path are not present by default - if you change the location of these files, you will see the Registry values for these options.
So that wraps up Dr. Watson - but what if you're running Windows Vista or Windows Server 2008, or you prefer to use something other than Dr. Watson? There are two different options to use - DebugDiag or ADPlus. For pre-Vista Operating Systems, you can also use Userdump.
Let's look at ADPlus first. ADPlus is a VBScript that is included with the Debugging Tools for Windows. ADPlus has two different modes - Hang and Crash. ADPlus is configured through the use of command line switches to specify the parameters. The basic command line switches are as follows:
So for example, to run ADPlus to monitor the Print Spooler and output the files to a folder on the D: drive, you would run the following command: cscript adplus.vbs -crash -pn spoolsv.exe -o d:\Spooler_Dumps. A key feature to remember about ADPlus is that unlike Dr. Watson or DebugDiag, which we are about to discuss, it can capture crash dumps of a failing 64-bit process. However, there are some caveats with ADPlus. For example, when you use ADPlus in hang mode, you must wait until the process or processes stop responding before you run the script (unlike crash mode, hang mode is not persistent). You can find more information regarding ADPlus usage in the Debugging Tools help file and Microsoft KB Article 286350.
Now, highlight the app and click the "Default Settings" button. Ensure that the "All Exceptions" checkbox is not selected. Click on the "Select All" button (which is different than selecting the "All Exceptions" checkbox). Your configuration window should look like the screenshot to the left.
Let's discuss a couple of the other options here. The option "Bugcheck after Dumping" will cause the machine to create a Kernel dump after creating the user-mode dump. Be very careful with this option!
The other feature to note here is under the "Exit Monitor" section. In some instances you will encounter processes that seem to close gracefully for no apparent reason. If you check the "Monitor Process Exit" box, you will get a dump file created every time the process is closed. Again - this is an option to use judiciously - even if a user closes the application being monitored deliberately you will create a dump file. For an application like Internet Explorer, Word, or Excel, this could result in a large number of unnecessary dump files (and quite a bit of wasted disk space!)
Finally, let's look at the Debug Diagnostic Tool (aka DebugDiag). DebugDiag was originally released as part of the IIS Diagnostic Toolkit. Version 1.1 is a standalone tool that can be used to troubleshoot hangs or crashes in any 32-bit user-mode process. Following the initial installation of DebugDiag, the configuration process is wizard-driven. So let's go ahead and set up a rule to monitor our machine for spooler crashes.
Since IIS is not installed on this machine, all of the IIS options are disabled. After selecting "Crash", and clicking "Next" we are presented with the option to select our Target Type - which could be a specific process (such as Internet Explorer, Excel, Outlook etc), a specific MTS / COM+ application, an IIS web application pool, or a specific NT service. For our example, we will go ahead and select "A specific NT service" and click "Next".
Now we can select our target as shown below:
Remember that we can only use DebugDiag against 32-bit user-mode processes. If you were to try and monitor the spooler on a 64-bit machine, you would receive an error similar to this:
After Clicking "Next", we are presented with the Advanced Configurations screen. For our purposes, we leave this at the default settings, and click "Next", which allows us to name our rule, and select a location for our dump files (see the image below). Once we have completed these options, we click next and activate our rule. Now the Spooler process is being monitored for crashes.
If / When the spooler crashes, we will find a new folder created for each instance of the crash in the Userdump location listed above. This will include the dump file and the log file.
Finally, there is one last option to capture a process dump that is available on Windows Vista. If you open up Task Manager, you can right-click on an application name, the process name or the service and select the option to "Create Dump File". This will create a dump of the process, but not terminate it - so you can capture multiple dumps of a running process!
And that brings us to the end of our post on Application Crash Dumps. Until next time ...
- CC Hameed
Thank you guys, this is really helpful :)