Recently I worked on a case where we needed to capture user dumps (memory dumps) of store.exe. Normally I would suggest using ADPLus.vbs to capture this information, but in this case there would not have been time to have ADPlus work correctly. The behavior we were trying to troubleshoot was the old "Requesting data from the Exchange Server" dialog. All of the performance metrics in Performance Monitor appeared to be within acceptable ranges. Upon further investigation, it appeared that Average RPC Latency increased abnormally when the "Requesting data..." dialog popped up. Part of the problem was that this was not affecting all users at once and it subsided fairly quickly. Any the time the problem was noticed and reported it would be too late to capture a dump or the admin would have to sit waiting for a re-occurrence all day.
After kicking the issue around we decided to try to automate the capturing of dumps utilizing Performance Monitor alerts. Alerts can be configured to monitor one or more existing counters with a threshold value. Depending on whether the alert is set to "over" or "under" several actions can be configured to occur.
a) Log an entry in the Application log. Source: SysMonLog, Event ID 2031 will indicate the specified trigger has fired.b) Send a Network messagec) Start a Performance Monitor logd) Run this program.
Any program can be configured to run when the Alert trigger is fired. This can be useful in automating various processes but I encountered some less than optimal behavior I had not anticipated. For testing purposes I used Processor usage for testing as it was easier to spike the processor than induce high RPC latency in my lab. When the processor usage threshold exceeded 20% I had configured ADPlus.vbs to be run. So when the alert counter went over 50 ADPlus.vbs fired and started dumping the store. I had the Alert configured to pool every 5 seconds (default) and the processor usage continued to rise up to 100%. The pooling interval was short since the window we were trying to capture was small. Each time the pooling interval was reached and the processor usage was over 20% another instance of ADPlus.vbs fired. In the end I had 79 separate instances of the CDB debugger that ADPlus.vbs had called running. That was about 78 concurrent dumps more than I was hoping for.
Credit for the batch file logic must go to my colleague Jason Dool. The trick was we had to have only one dump at a time captured not 79 all at once. Jason came up with a simple batch file to capture a series of 3 dumps. When the alert trigger was reached the batch file was executed. The first thing the batch file does is check for a 0 byte file named "C:\dump.txt" and it this file does not exist then create "dump.txt". If the file "dump.txt" already exists then the batch file exits. Otherwise ADplus.vbs is called three times at a couple of minute intervals. This way as the alert threshold is continuing to rise only one set of dumps will be created and not many instances trying to dump the same process concurrently. Below are the steps necessary to use Performance Monitor alerts to execute ADPlus.vbs to capture a series of timed dumps.
Any executable can be fired using Performance Monitor alerts. I was caught off guard by the results when simply specifying ADPlus.vbs to execute. Not something that would be good in a production environment. As far as this being a typical way to troubleshoot the "Requesting data..." popups, well it is not. 99% of time the problem with these "Requesting data..." popups does not require capturing user dumps, but this particular case had very little of the typical clues to go on.
A batch file can be configured to execute and run ADPlus to capture the desired number of dumps.
A blank file is created and if this file is present the batch file will not execute again.
By preventing the batch file from executing subsequent times an admin can collect the required number of user dumps at configured intervals.
Below is a sample batch file that can be modified to suit specific needs. In order to use this batch file to collect dumps you will need the following installed on the target Exchange server:
1) Install Debugging Tools for Windows on the target server 2) Get a copy of "Sleep.exe" from Windows Resource Kit
Create a BAT or CMD file using the sample below. The sample below will collect three sets of dumps of MAD.exe and STORE.exe. The first time the file executes before creating the first dump set "c:\dumps.txt" is created and the additional dumps are created. If the script is executed again the script checks whether "C:\dumps.txt" exists. If "C:\dumps.txt" exists then the script terminates without capturing any dumps.
====Copy starting below this line=============>REM===========================================================REM COMMENT: This script help you take single or multiple memory dumps ofREM specific process/processes, based on the Performance Monitor triggerREM===========================================================REM You have a royalty-free right to use, reproduce, modify and distributeREM this sample file or any modified version in any way you find useful,REM provided you agree Microsoft has no liability, obligations, warranty, orREM responsibility regarding any result produced by use of this file.REM Copyright (C) 1996-2005 Microsoft CorporationREM===========================================================@echo off REM Checks for file "C:\dumps.txt" and exits if file is presentREM Otherwise collect data being adplus dump. If exist c:\dumps.txt (exit) echo We have successfully ran the dump.bat and created the dumps > c:\dumps.txtecho Initiating dump of Store and System Attendant processes.... REM #Comment 1# The line below calls ADPlus.vbs in hang mode and creates a dump of the Information Store service as well as the System Attendant. The line immediately below can be changed to execute any program.cscript "C:\Program Files\Debugging Tools for Windows\adplus.vbs" -hang -quiet -pn store.exe -pn mad.exe
REM #Comment 2# The lines down to the #End Repeat# can be deleted if the configured application only need to be run once.
echo Waiting for 2 minutes REM - Pause 2 minutes before next dumpsleep 120 echo Initiating second dump of Store and System Attendant processes.... cscript "C:\Program Files\Debugging Tools for Windows\adplus.vbs" -hang -quiet -pn store.exe -pn mad.exe echo Waiting for 3 minutes REM - Pause 3 minutes before next dumpsleep 180 echo Initiating third dump of Store and System Attendant processes.... cscript "C:\Program Files\Debugging Tools for Windows\adplus.vbs" -hang -quiet -pn store.exe -pn mad.exe REM #End Repeat#echo Done collecting process dump information. <========End Copy============
Copy the script and sleep.exe to the Path that the Windows Debugging Tools were installed (by default this sill be "C:\Program Files\Debugging Tools for Windows")
Open Performance Monitor and expand "Performance Logs and Alerts" - Right click on "Alerts" and select "New Alert Settings" - Provide a descriptive name for the alert. - Add appropriate counters that need to be monitored. - Configure a value for which the alert will be activated. - The sample interval can be adjusted as needed. Since this script will only run once 5 seconds is OK. - Switch the "Action" tag. - Select the following:
"Log an entry to the Application Event log" (an Informational event 2031 Source: SysMonLog will be logged when the alert is tripped.)
"Run a Program" and Configure the BAT or CMD file created from the sample to execute. (be sure to include full path to file)
- Click the "Command Line Arguments" button and clear ALL check boxes. - The "Schedule" tab can be configured to activate and deactivate the alert if desired. - Once the preceding configuration has been made click OK. - If the Alert icon in the left pane of "Alerts" is red, then the alert is inactive. - To make the alert active, right click and select "Start".
When the script is executed and starts collecting dumps several CMD windows will appear (possibly up to 7 windows). It is important that these windows be left open. Closing these windows will cause the dump process to terminate prematurely. Once all the configureddumps are complete, the CMD Windows will close.
When ADPlus executes the following Warning may be seen. This is OK:
===========================WARNING! An 'NT_SYMBOL_PATH' environment variable is not set.Please check the application event log or the ADPlus-report.txtfor more details.==========================
This is simply a warning indicating that symbols have not been configured. Symbols are typically not required in order to capture necessary information using this method. In order to collect a second set of dumps, the "C:\dumps.txt" file must be deleted. There is no useful information in this file as it is only meant to serve as a flag to prevent multiple instances of the batch file from executing.
When a process is being dumped the process is frozen until the dump is complete. During that time the Information Store service mailbox and public folder access will be unavailable. Clients will receive... you guessed it, the "Requesting data...." dialog.
In the above example, three consecutive dumps of the Information Store and System Attendant are being initiated. The reason that we wanted to capture three dumps is to examine how threads were changing over time. Typically the only time you will want to capture memory dumps is at the recommendation of Support Services.
Hope this was helpful!
- Ashley Webb