Introduction

The goal of this post is to show how DebugDiag 1.2 can assist you identifying a potential source of bottleneck on a scenario where TMG user mode process (wspsrv.exe) is consuming high amount of CPU.

Data Gathering

First part is to make sure you collect the user mode dump while the issue is happening. To do that, use the approach that I explain in the following post:

http://blogs.technet.com/b/yuridiogenes/archive/2010/05/01/how-to-capture-a-manual-dump-of-the-wspsrv-exe-process-on-tmg-2010.aspx

Data Analysis

Once you have the data you can use DebugDiag to analyze the dump. Follow the steps below in order to perform this analysis:

1. After installing Debug Diag (64 bits edition in this case), launch it and cancel the first window.

2. Click Advanced Analysis tab.

3. Click Add Data Files button and choose the dump file that was previously collected.

4. Choose the scenario that applies to this issue in the top pane. In this case the scenario is Crash/Hang Analyzers as shown below:

5. Click Start Analysis.

6. Wait until the report is generated.

Reviewing the Report

Don’t go too far on the report before reviewing the first part of it, which is the Analysis Summary. Here it is the example for this scenario:

In this case the warning message says:

Detected a possible critical section related problem in wspsrv.dmp
Lock at 0x015e7c70 is Unlocked
Impact analysis
0.67% of threads blocked
(Threads 78)
The following functions are involved in the root cause
GapaEngine_1cc44e8_bace5e90+10e22

The thread number has a hyperlink on it, when you click on this hyperlink you will see the stack that it is referring to:

ntdll!ZwWaitForSingleObject+a
ntdll!RtlpWaitOnCriticalSection+e8
ntdll!RtlEnterCriticalSection+d1
GapaEngine_1cc44e8_bace5e90+10e22
0x454b64d8
0x0300e000
0x015ccbe8
0x4b80e418
0x015ccbe8
GapaEngine_1cc44e8_bace5e90+ff44
0x00004441`014dd475
0x00000010

The recommendation that DebugDiag gives is:

The following vendors were identified for follow up based on root cause analysis
Unknown vendor for module C:\Program Files\Microsoft Forefront Threat Management Gateway\IPS\GapaEngine_1cc44e8_bace5e90.dll
Please follow up with the vendors identified above

In other words, it is telling me to investigate further this module. Now what? Well, now you have an initial path to follow, you know that GAPA Engine is involved, which means that you can start doing some tests, such as:

• Does the issue still happening if you disable NIS?
• If it does not, than you might be experiencing the issue described in this post http://blogs.technet.com/b/yuridiogenes/archive/2011/03/20/another-case-of-high-cpu-utilization-by-wspsrv-exe-on-forefront-tmg-2010.aspx
• If it does not, than you can try to narrow it down more by using the procedures from Forefront TMG 2010 Performance issues Cheat Sheet.

It is important to remember that troubleshooting performance issue can be a long process and DebugDiag can assist you to find the root cause. However, sometimes finding the culprit doesn’t fix the issue, just show who is causing the problem, in this case further investigation is needed to find out how to really fix the issue.