Mfartura's blog : Debug

Tip: Kernel Debugging a VPC Server

Marcelo Fartura — Wed, 20 Jun 2007 10:16:00 GMT

Here is little tip for you that want to practice kernel mode debugging but either don’t have 2 machine machines to play the TARGET and HOST roles or simply don’t want to play with (for any reason) with the old null modem serial cable used to connect the HOST to the TARGET: It’s possible to use a Virtual PC machine to play the TARGET role, so all you would need is the Virtual PC image correctly configured to boot up in DEBUG mode (through the regular boot.ini options - /DEBUG /DEBUGPORT=COM# /BAUDRATE=115200) and map its serial COM port to a named pipe.

How to do so:

Supposing you already have a Virtual PC image set up (it can be an image of Windows 2000, Windows XP or Windows 2003), you just need to follow the step by step list below:

1. Login as admin in the VPC image and edit the c:\boot.ini file to include a new boot up OS option as following:

A default boot.ini file will likely have the following content:

[boot loader]

timeout=3

default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[operating systems]

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect

Edit the boot.ini and add the highlighted line below:

[boot loader]

timeout=3

default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[operating systems]

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional – DEBUG" /fastdetect / DEBUG /DEBUGPORT=COM1 /BAUDRATE=115200

2. Before rebooting the VPC image, go the VPC console, select the option “Settings” under the menu “Edit”. Find the configuration item for the COM1 port, select the radio button “Named Pipe” and type the string \\.\pipe\COM1 as the picture below illustrates.

This VPC image (the virtual machine) will be the TARGET system, or the machine whose the kernel will be debugged. Your real machine (the one hosting the VPC image) will be the HOST machine or the one which will run the debugger that will be attached to the TARGET’s system kernel.

3. Start a Windbg (from the Debugging Tools For Windows) instance with the following command line options:

windbg -k com:pipe,port=\\.\pipe\COM1,resets=0,reconnect

The Windbg will open up it’s command window and will wait to connect through the named pipe mapped port as soon as VPC OS starts up. The picture below illustrates how the Windbg will start up with these command line parameters:

4. Now reboot the VPC image and select the new option you’ve included in the boot.ini in the step 1 of this procedure. During the boot process, as soon as the kernel gets loaded the Windbg will attach to it. All you need to do now if break the debugger at the point you want to start debugging and have fun J.

Real Case: IIS and Asp.Net hanging or poor performance (intermediary)

Marcelo Fartura — Thu, 26 Oct 2006 04:15:00 GMT

This week I'm working on this case related to IIS6 server hosting an Asp.Net application (it's a C# app) whose the performance is poor even when a very low number of users is hitting it. The architecture is the classical web layer accessing the backend DB servers and my goal is to isolate the problem, i.e pinpoint where the problem resides and try to find out why.

The first step after understanding the environment and being aware of all relevant servers and infrastructure devices, is to actually see the problem happening to have a better understanding of what is going on. We had one single user accessing the application and there are indeed some actions the user takes on the pages that take a long time to return, so the next step is to establish a baseline and agree on a SLA for the performance the application is supposed to have.

Another very important relevant point to this scenario is that since the environment counts on a NLB composed by multiple IIS servers, we need first to isolate which IIS server the client will connect to when we start trying to repro the problem. So after checking their NLB configuration and making sure they have the affinity set to single we can just open the IIS logs of the servers and search the client IP at the c-ip field. Since the affinity is single the server which has been serving the client will continue to be the same.

Once we know which server we will work with during the tests, my favorite tool to create the baseline is the System Monitor (perfrmon.exe) and we can also make some basic analysis of the IIS logs. So I set the perfmon with some specific counters that will help me to establish a baseline on how their app is currently behaving and also made sure they have the IIS log fields I need to work with when establishing the baseline.

The counters I used in the perfmon were the following:

ASP.NET v1.1.4322\Applications Running
ASP.NET v1.1.4322\Request Execution Time
ASP.NET v1.1.4322\Request Wait Time
ASP.NET v1.1.4322\Requests Current
ASP.NET v1.1.4322\Requests Queued
ASP.NET v1.1.4322\State Server Sessions Active
ASP.NET v1.1.4322\Worker Process Restarts
ASP.NET v1.1.4322\Worker Processes Running

The fields I needed to make sure are being included in the IIS logs are the following:

c-bytes

s-bytes

cs-uri-stem

cs-uri-query

time-taken

After that we reproduced the problem, which by the way was really happening with only one user accessing it which already eliminates any hypothesis of the problem being related to the amount of load hitting the server, we started working on the baseline.

The counters in perfmon pretty much confirm what we see in the IIS logs: Their main application page takes an average time longer than 40 seconds to execute. The perfmon counter “ASP.NET v.1.1.4322\Request Execution Time” will report that as well as the IIS logs if you use the LogParser tool and run the following query against them:

First we need to consolidate all the IIS logs in a single CSV file:

LogParser "select date,time,cs-uri-stem,cs-uri-query,sc-status,sc-win32-status,time-taken,sc-bytes,cs-bytes from ex061024.log, ex061025.log to merged.csv" -o:CSV

Now we run the following query to get the average time-taken for the URLs in the logs and send the output to a new CSV file named AvgTimeTaken.txt

logparser "select cs-uri-stem,count(cs-uri-stem), avg(time-taken) from merged.csv to AvgTimeTaken.txt where time-taken>=10000 GROUP BY cs-uri-stem ORDER BY avg(time-taken) DESC" -o:CSV -i:CSV

The result is as following:

cs-uri-stem	COUNT(ALL cs-uri-stem)	AVG(ALL time-taken)
/App/MainPage.aspx	15	48353
/App/MainPage.aspx	3	37738

Ok, now that we see the problem happening and have our baseline it’s time to start working on isolating problem.

We don’t know what exactly the user does in order to have the mainpage.aspx taking so long to execute, so what we do is try to reproduce the problem again and keep following the perfmon counter “ASP.NET v1.1.4322\Requests Current”. This method of tracking the “Requests Current” counter will not work for all scenarios, since when you have the server under some load you will likely not be able to tell when a specific request is taking long or not because there will be many requests at the same time and they will keep changing the value of “Requests Current”. For this cases we would need to have the collaboration of the user (which would tell us when the page is taking long to return) or look at other counters like the “Request Queued”. However for this specific scenario where we have the variables under control and there is a single user testing the system for us, it will work just fine.

When the “Requests Current” shows 1 request and don’t get back to zero after, let’s say, 10 seconds, that’s the point where we know this request is taking too long to execute. Remember that our baseline is around 40 seconds… Now it’s right time to generate a hung dump of the worker process. To do that we first run the iisapp.vbs script in order to identify the PID of the worker process hosting our application pool and after that we can use any tool to simply generate multiple hang dumps (like Debugging Tools For Windows or Debug Diagnostics Tool (DebugDiag)) some seconds apart from each other. In this case I used the adplus.vbs from Debugging Tools For Windows with the following sintaxe:

Adplus.vbs –hang –p 2364 –quiet –o c:\dumps

This will generate a memory dump of the w3wp.exe whose the PID is 2364 and store it at the c:\dumps folder.

The dumps analysis will require just some basic .Net debugging steps (I used the Windbg from Debugging Tools For Windows to analyze them) as below:

This is a managed application (it’s aspx) so we want to make sure we have, besides the proper symbols, the sos.dll extension properly loaded:

0:000> .chain

Extension DLL chain:

aspvbscriptext.dll: API 1.0.0, built Fri Feb 06 18:20:44 2004

[path: C:\debuggers\aspvbscriptext.dll]

mdacexts.dll: image 2004.05.11.1, API 1.0.0, built Tue May 11 17:46:12 2004

[path: C:\debuggers\mdacexts.dll]

webdbg: API 3.5.5, built Thu Nov 15 18:26:28 2001

[path: C:\debuggers\webdbg.dll]

sieext: API 1.0.0, built Fri Nov 05 16:07:40 2004

[path: C:\debuggers\winext\sieext.dll]

dbghelp: image 6.6.0005.0, API 6.0.6, built Fri Apr 14 17:03:13 2006

[path: C:\debuggers\dbghelp.dll]

ext: image 6.6.0005.0, API 1.0.0, built Fri Apr 14 17:02:35 2006

[path: C:\debuggers\winext\ext.dll]

exts: image 6.6.0005.0, API 1.0.0, built Fri Apr 14 17:02:30 2006

[path: C:\debuggers\WINXP\exts.dll]

uext: image 6.6.0005.0, API 1.0.0, built Fri Apr 14 17:02:37 2006

[path: C:\debuggers\winext\uext.dll]

ntsdexts: image 6.0.5357.0, API 1.0.0, built Fri Apr 14 17:21:31 2006

[path: C:\debuggers\WINXP\ntsdexts.dll]

clr10\sos: image 6.6.0005.0, API 1.0.0, built Fri Apr 14 17:02:47 2006

[path: C:\debuggers\clr10\sos.dll]

Ok, so once we know we have the right extension loaded, let’s start finding which threads are being used to process the Asp.Net mainpage.aspx. We expect to see only one thread here since our user is simply accessing this single page. By looking at the first dump we see the following:

0:000> !aspxpages

Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"

Going to print all threads with ASP.NET Pages running.

Thread 11: /App/Mainpage.aspx

Ok, as we suspected there is only one thread processing the Asp.Net request and it’s the thread 11. Now let’s look on what this thread is currently doing:

0:011> !clrstack
Thread 11
ESP EIP
0x0604f4dc 0x7ffe0304 [FRAME: NDirectMethodFrameStandalone] [DEFAULT] UI2 System.Data.Common.UnsafeNativeMethods/Dbnetlib.ConnectionRead(ValueClass System.Runtime.InteropServices.HandleRef,SZArray UI1,UI2,UI2,UI2,ByRef I)
0x0604f4f8 0x069cf5e4 [DEFAULT] [hasThis] Void System.Data.SqlClient.TdsParser.ReadNetlib(I4) ß it seems we have some SQL stuff on the stack…
0x0604f558 0x069cf531 [DEFAULT] [hasThis] Void System.Data.SqlClient.TdsParser.ReadBuffer()
0x0604f560 0x069cf463 [DEFAULT] [hasThis] UI1 System.Data.SqlClient.TdsParser.ReadByte()
0x0604f568 0x069ced7a [DEFAULT] [hasThis] Boolean System.Data.SqlClient.TdsParser.Run(ValueClass System.Data.SqlClient.RunBehavior,Class System.Data.SqlClient.SqlCommand,Class System.Data.SqlClient.SqlDataReader)
0x0604f5b0 0x06aa3c79 [DEFAULT] [hasThis] Void System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
0x0604f5c4 0x06aa3c0e [DEFAULT] [hasThis] SZArray Class System.Data.SqlClient._SqlMetaData System.Data.SqlClient.SqlDataReader.get_MetaData()
0x0604f5cc 0x06aa1be7 [DEFAULT] [hasThis] Class System.Data.SqlClient.SqlDataReader System.Data.SqlClient.SqlCommand.ExecuteReader(ValueClass System.Data.CommandBehavior,ValueClass

<cut for brevity>

So, the managed stack for the thread 11 seems to be waiting on SQL connection. The native stack would be at a ntdll!NtWaitingForSingleObject() function which basically means this thread is doing nothing but waiting. So let’s examine a little further to see if we can identify which results are we waiting from the SQL Server:

0:011> !dso
Thread 11
ESP/REG Object Name
eax 0x10073b34 System.Collections.Hashtable
ecx 0x10073b34 System.Collections.Hashtable
0x604f4c8 0x1801dad4 System.Byte[]
0x604f4d0 0x28061840 System.Data.SqlClient.TdsParser
0x604f4d4 0x1801dad4 System.Byte[]
0x604f4f8 0x28061840 System.Data.SqlClient.TdsParser
0x604f500 0x28090704 System.Data.SqlClient.SqlDataReader
0x604f504 0x28061840 System.Data.SqlClient.TdsParser
0x604f51c 0x28061840 System.Data.SqlClient.TdsParser
0x604f54c 0x2808fd28 System.Data.SqlClient.SqlCommand ß this is what we’re looking for

<cut for brevity>

Let’s taker a closer look at this specific object:

0:011> !do 0x2808fd28
Name: System.Data.SqlClient.SqlCommand
MethodTable 0x06854dd8
EEClass 0x0657fb60
Size 64(0x40) bytes
GC Generation: 0
mdToken: 0x02000194 (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x068545c4
MT Field Offset Type Attr Value Name
0x79b80d44 0x40000b6 0x4 CLASS instance 0x00000000 __identity
0x0649824c 0x4000502 0x8 CLASS instance 0x00000000 site
0x0649824c 0x4000503 0xc CLASS instance 0x00000000 events
0x0649824c 0x4000501 0 CLASS shared static EventDisposed
>> Domain:Value 0x00099230:NotInit 0x000dfad8:0x100144d8 <<
0x06854dd8 0x4000eac 0x10 CLASS instance 0x10032b90 cmdText ß this string seems promising
0x06854dd8 0x4000ead 0x24 System.Int32 instance 4 cmdType
0x06854dd8 0x4000eae 0x28 System.Int32 instance 3600 _timeout
0x06854dd8 0x4000eaf 0x2c System.Int32 instance 3 _updatedRowSource
0x06854dd8 0x4000eb0 0x38 System.Boolean instance 0 _inPrepare
0x06854dd8 0x4000eb1 0x30 System.Int32 instance -1 _handle
0x06854dd8 0x4000eb2 0x14 CLASS instance 0x28090450 _parameters
0x06854dd8 0x4000eb3 0x18 CLASS instance 0x2808f624 _activeConnection
0x06854dd8 0x4000eb4 0x39 System.Boolean instance 0 _dirty
0x06854dd8 0x4000eb5 0x3a System.Byte instance 0 _execType
0x06854dd8 0x4000eb6 0x1c CLASS instance 0x00000000 _cachedMetaData
0x06854dd8 0x4000eb7 0x34 System.Int32 instance -1 _rowsAffected
0x06854dd8 0x4000eb8 0x3b System.Boolean instance 0 designTimeVisible
0x06854dd8 0x4000eb9 0x20 CLASS instance 0x00000000 _transaction

Looking at such an instance:

0:011> !do 0x10032b90
Name: System.String
MethodTable 0x79b7daf8
EEClass 0x79b7de44
Size 80(0x50) bytes
GC Generation: 2
mdToken: 0x0200000f (c:\windows\microsoft.net\framework\v1.1.4322\mscorlib.dll)
String: dbo.[spGenerateOptions] ß Bingo!!! This is the stored procedure we’ve been waiting on to return
FieldDesc*: 0x79b7dea8
MT Field Offset Type Attr Value Name
0x79b7daf8 0x4000013 0x4 System.Int32 instance 31 m_arrayLength
0x79b7daf8 0x4000014 0x8 System.Int32 instance 30 m_stringLength
0x79b7daf8 0x4000015 0xc System.Char instance 0x64 m_firstChar
0x79b7daf8 0x4000016 0 CLASS shared static Empty
>> Domain:Value 0x00099230:0x28000224 0x000dfad8:0x28000224 <<
0x79b7daf8 0x4000017 0x4 CLASS shared static WhitespaceChars
>> Domain:Value 0x00099230:0x28000238 0x000dfad8:0x2000364c <<

So we know that at the point this dump was taken we were just waiting at the stored procedure spGenerateOptions to complete it’s execution at a SQL Server.

We are going now to look at the subsequent dump (taken 30 seconds later) to verify whether we have any progress or not on that thread, however let’s take a loot at one last thing on this dump before starting with the other one (this can save us a lot of time here):

0:011> !runaway

User Mode Time

Thread Time

13:d5c 0 days 0:00:37.500

14:c00 0 days 0:00:24.718

16:9c8 0 days 0:00:16.406

15:9e8 0 days 0:00:13.562

11:bb8 0 days 0:00:12.015 ß This is the CPU time spent by the thread 11 at this point

So we know that the thread 11 have had the CPU for the aprox. 12.015 seconds when this dump was generated. So after opening the second dump, before jumping on any deeper analysis, lets just run the same command and verify if the thread 11 made any progress.

0:011> !runaway

User Mode Time

Thread Time

13:d5c 0 days 0:00:37.500

14:c00 0 days 0:00:24.718

16:9c8 0 days 0:00:16.406

15:9e8 0 days 0:00:13.562

11:bb8 0 days 0:00:12.015 ß The time is exactly the same 30 seconds later

Ok, so there is not a single additional millisecond being spent by the thread 11 on the second dump. At this point we don’t even need to take a look at the thread stack to know it’s exactly the same. In other words this thread has been doing nothing but waiting on that stored procedure to execute at the SQL Server.

The problem has been isolated as a database performance issue (mission accomplished!!! J). The next step is work on the database server (running in another box) to understand why this sp is taking too long to execute and, of course, properly addressing the issue to provide a definitive solution. IIS ended up just being the server where the symptom is faced and not the one where the root cause of the problem is really happening.

A Virus Infection: Contolling the Outbreak (tip)

Marcelo Fartura — Tue, 24 Oct 2006 08:48:00 GMT

This weekend I had the opportunity to work along our security specialists (from PSSSec group) in what we can call the hottest situation involving hundreds of production servers either stopped or severely impacted by a new variation of a worm. As a new variation, the AV signature file wasn't updated with the proper signature and once the first machine got inffected it immediately started spreading the worm accros the customer production environment...

Details like how the first machine got infected, how the worm spreads itself through the network or even which worm is the variation of, will be ommited here as this scenario will be usefull only to illustrate what the tip is about.

So, direct to the point:

While on site helping to control the situation, one of the steps in our action plan while the signature file for the AV was being built was to control the infection or in other words, stop the infected machines to keep infecting the others. Clean the machines was something we knew we wouldn't be able to accomplish unless we could build an AV engine overnight :)

We isolated the worm itself to 2 executable files and a DLL. The executables are dropped during the infection and a registry key is changed in order to load them whenever the machine is booted. They also run as Local System, so you can't just open the task manager, right click their processes, and select the "End process" option as this would cause a access denied. What you can do though, is to use the utility kill.exe together with the tlist.exe from Debugging Tools For Windows , build a script that will filter the tlist.exe's output trying to find the specific processes names and then run kill.exe to kill them, and then you use the at.exe command or the Task Scheduler to schedule such a script to run. By using the Task Scheduler service your script will run as Local System and should be able to kill the processes running under the same security context.

Ok, but this does not prevent the process to get spawned again (for instance in the next boot) and even deleting the registry entries that could cause that, since the AV is not effective, this machine can get infected again and you will never know when to run the script again to stop the worm's processes. We needed to find a way to prevent these specific processes to keep getting respawned...

So here is the tip for something you can use in different OS versions from Windows NT to Windows 2003 to accomplish that: There is a registry key called "Image File Execution Options" under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion which is normally used for debugging purposes. There is a specific string value that we can use to accomplish our demand of preventing the processes to run. It's the "Debugger" SZ value. This value is used to include a debugger that should launch the process whenever there is a demand for the OS to spawn it. For instance: If you create a key named "Calc.exe" under "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options" and add a SZ value called "Debugger" and set it to, for instance, a debugger like cdb.exe (Debugger = c:\debuggers\cdb.exe) the effect will be such that whenever you try to execute the program calc.exe (regardless of user and regardless of how) the debugger cdb.exe will be called instead and it will spawn the calc.exe itself within the debugger.

All right, so now we just need to be creative and find a way to, instead of run a debugger, just prevent the process to be spawned. Here is one way:

1. Create a registry key with the name of the process you want to prevent to execute. Ex.: calc.exe

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\calc.exe

2. Under this new key you've just created, create a SZ value called "Debugger" and set it to the following value:

SZ Debugger = "cmd.exe /c echo %time% %date% >> c:\ExecBlocked.log"

That's it. You don't need to restart anything or reboot the machine. From now on you will not only be unable to run calc.exe as whenever you try to do it the file ExecBlocked.log will get the attempt to execute recorded with the date/time information.

To enable the process the run again, simply remove the registry key.

Lab: Win32 Application causing 100% CPU condition (Advanced)

Marcelo Fartura — Mon, 18 Sep 2006 21:32:00 GMT

This is lab for simulating a real 100% CPU condition being caused by a generic Win32 application when there are no symbols available for such application.

Let’s start supposing the situation is currently occurring. So, the first step is to confirm the symptoms, which we can easily do by using, for instance the task manager. As we see in the picture below we have a 50% utilization on a dual core machine which represents 100% of one CPU being utilized.

If we navigate to the process tab in the task manager we will be able to identify which process is causing the problem as shown in the picture below:

Ok, so it’s the MyApp.exe that is keeping one of our CPUs totally busy. It's time to bring the debugger and look what’s happening under the hood…

By using the Windbg from the Debugging Tools For Windows package, we can choose, by pressing F6, which process we want to attach the debugger to:

After attaching to the MyApp.exe process, the debugger will, by default, list all the modules loaded at the address space as well as it will break the execution through the break instruction exception (80000003). Now, since the execution is broken, our CPU is no longer busy and we can start investigating what is going on here.

Microsoft (R) Windows Debugger Version 6.6.0005.0

Executable search path is:

ModLoad: 00400000 00446000 C:\Temp\MyApp.exe

ModLoad: 7c900000 7c9b0000 C:\WINDOWS\system32\ntdll.dll

ModLoad: 7c800000 7c8f4000 C:\WINDOWS\system32\kernel32.dll

ModLoad: 77d40000 77dd0000 C:\WINDOWS\system32\USER32.dll

ModLoad: 77f10000 77f57000 C:\WINDOWS\system32\GDI32.dll

ModLoad: 76b40000 76b6d000 C:\WINDOWS\system32\WINMM.dll

ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll

ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll

ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll

ModLoad: 76c30000 76c5e000 C:\WINDOWS\system32\WINTRUST.dll

ModLoad: 77a80000 77b14000 C:\WINDOWS\system32\CRYPT32.dll

ModLoad: 77b20000 77b32000 C:\WINDOWS\system32\MSASN1.dll

ModLoad: 76c90000 76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll

ModLoad: 72d20000 72d29000 C:\WINDOWS\system32\wdmaud.drv

ModLoad: 72d10000 72d18000 C:\WINDOWS\system32\msacm32.drv

ModLoad: 77be0000 77bf5000 C:\WINDOWS\system32\MSACM32.dll

ModLoad: 77bd0000 77bd7000 C:\WINDOWS\system32\midimap.dll

ModLoad: 5ad70000 5ada8000 C:\WINDOWS\system32\uxtheme.dll

ModLoad: 74720000 7476b000 C:\WINDOWS\system32\MSCTF.dll

ModLoad: 77120000 771ac000 C:\WINDOWS\system32\OLEAUT32.DLL

ModLoad: 774e0000 7761d000 C:\WINDOWS\system32\ole32.dll

ModLoad: 10000000 1001e000 C:\Program Files\Babylon\CAPTLIB.DLL

ModLoad: 77f60000 77fd6000 C:\WINDOWS\system32\SHLWAPI.dll

ModLoad: 74c80000 74cac000 C:\WINDOWS\system32\OLEACC.dll

ModLoad: 76080000 760e5000 C:\WINDOWS\system32\MSVCP60.dll

(8a8.c5c): Break instruction exception - code 80000003 (first chance)

eax=7ffd6000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005

eip=7c901230 esp=00fcffcc ebp=00fcfff4 iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246

ntdll!DbgBreakPoint:

7c901230 cc int 3

Let’s make sure you have the right symbol path so we can load at least the MS public symbols.

0:003> .sympath

Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols

We have 4 threads as below:

0:003> ~

0 Id: 8a8.da8 Suspend: 1 Teb: 7ffdf000 Unfrozen

1 Id: 8a8.f1c Suspend: 1 Teb: 7ffde000 Unfrozen

2 Id: 8a8.75c Suspend: 1 Teb: 7ffdd000 Unfrozen

. 3 Id: 8a8.c5c Suspend: 1 Teb: 7ffdc000 Unfrozen

Our current context is set to thread 03.

The more reliable way to find out which thread is consuming the processor time would be use the System Monitor tool with the right objects and counters (process and thread objects) as described in the article Troubleshooting IIS 100% CPU issues - Step by Step. However we will try something different here. We will use the extension command !uext.runaway or simply !runaway (since the extension uext.dll is loaded by default when debugging user mode apps from Windbg.exe):

0:003> !uext.runaway

User Mode Time

Thread Time

0:da8 0 days 0:03:36.703

3:c5c 0 days 0:00:00.000

2:75c 0 days 0:00:00.000

1:f1c 0 days 0:00:00.000

The !runaway extension command returns a list containing all the current threads and the processor time spent by them. As we see in the result above our thread 0 has to be the one we are interested in since it's not only the one with more time, but is also the only one with any processor time.

So let’s change the context to thread 0 in order to verify what has been executed by it:

0:003> ~0s

*** WARNING: Unable to verify checksum for C:\Temp\MyApp.exe

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Temp\MyApp.exe -

eax=00000000 ebx=00000000 ecx=0012fd7c edx=7c90eb94 esi=00402150 edi=0012fe40

eip=00401a73 esp=0012fd74 ebp=0012fd74 iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246

MyApp!AboutDlgProc+0xa73:

00401a73 5d pop ebp

There is a message warning us about the lack of symbols. Let's see what we can find about what this thread is doing even with no symbols for the application. Looking at the call stack we have the following:

0:000> kL

ChildEBP RetAddr

WARNING: Stack unwind information not available. Following frames may be wrong.

0012fd74 00401a69 MyApp!AboutDlgProc+0xa73

0012fd80 0040248f MyApp!AboutDlgProc+0xa69

0012fdd8 77d48734 MyApp!WndProc+0x33f

0012fe04 77d48816 USER32!InternalCallWinProc+0x28

0012fe6c 77d489cd USER32!UserCallWinProcCheckWow+0x150

0012fecc 77d496c7 USER32!DispatchMessageWorker+0x306

0012fedc 00401244 USER32!DispatchMessageA+0xf

0012ff30 00402cd3 MyApp!AboutDlgProc+0x244

0012ffc0 7c816fd7 MyApp!WndProc+0xb83

0012fff0 00000000 kernel32!BaseProcessStart+0x23

This just tells us, based on the first frame, that this is the first thread and also it has some MyApp.exe code being executed (we have a chain of at least 5 MyApp.exe functions in the stack). But with no symbols this won't help us any longer since the function names for MyApp.exe are not reliable. Time to look at the asm being executed...

The current instruction pointer (represented by the CPU register EIP) is pointing to this asm:

0:000> u eip

MyApp!AboutDlgProc+0xa73:

00401a73 5d pop ebp

00401a74 c3 ret

00401a75 cc int 3

As we can see, it's just taking the top value on the stack and saving it to the CPU register EBP (pop EBP) and the next instruction is setting the EIP to the return address (this will load the EIP with the instruction located at the address at EBP + 4 in the memory). We can identify this asm block as part of epilog which means we are at the end of it's execution. So let's look a little bit before in the asm to verify what was being executed before the epilog block:

0:000> u eip -13 eip+1

MyApp!AboutDlgProc+0xa60:

00401a60 55 push ebp

00401a61 8bec mov ebp,esp

00401a63 51 push ecx

00401a64 e805000000 call MyApp!AboutDlgProc+0xa6e (00401a6e)

00401a69 8945fc mov dword ptr [ebp-4],eax

00401a6c ebf6 jmp MyApp!AboutDlgProc+0xa64 (00401a64)

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

Now we got the complete block since the execution of the prologue (the push EBP; mov EBP, ESP; push ECX instructions). By analyzing the block above we see that after the prolog execution, we make a function call to the instrution at 00401a6e (call MyApp!AboutDlgProc+0xa6e), so let's verify what should be executed when doing that:

0:000> u 00401a6e

MyApp!AboutDlgProc+0xa6e:

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

00401a74 c3 ret

All right, so what we see here, after the prolog block, is an exclusive OR (XOR) operation being executed using EAX as both operands, and then the epilog block is executed - This epilog block is the same one (the instruction at the address 00401a73) our EIP is currently set to (notice the first asm block at the beginning of this analysis).

The XOR operation is just being used to set the EAX register to 0 (the result of the operation is loaded to the destination operand, in this case the EAX itself). If we have a chance to look at the application source code, this is likely something like "MyLocalVar = 0;". So it comes the question: Why not make a "mov EAX,0" instead of a "xor EAX,EAX"? The answer is simple: it's a matter of cost. The instruction XOR costs less than the instruction MOV to the CPU, so it's faster to execute a XOR since in this case it will produce the same desired result.

So what we know so far is that we have a function being called and the only thing this function does is set a variable to 0 and returns. So let's keep going from the return of this function in the following block:

MyApp!AboutDlgProc+0xa60:

00401a60 55 push ebp

00401a61 8bec mov ebp,esp

00401a63 51 push ecx

00401a64 e805000000 call MyApp!AboutDlgProc+0xa6e (00401a6e)

00401a69 8945fc mov dword ptr [ebp-4],eax

00401a6c ebf6 jmp MyApp!AboutDlgProc+0xa64 (00401a64)

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

We've just returned from the instruction "call MyApp!AboutDlgProc+0xa6e". So the next instruction to be executed will the one at the address 00401a69 which is just moving the EAX value (which we already know at this point should be 0) to the location of EBP - 4. EBP - 4 is known to be the location of the first local variable of the current function at the call stack, so we're basically copying the EAX value to one or our local variables.

The next instrutcion (jmp MyApp!AboutDlgProc+0xa64) will cause the execution to "jump" to the address 00401a64 which, at this point, is already familiar to us. We know that the instruction at 00401a64 will make a function call to the function MyApp!AboutDlgProc+0xa6e which we analyzed before and know it's just setting the EAX to 0 and returning.

The conclusion now is obvious. We have an infinite loop being executed and this is the reason we have a 100% CPU condition happening. This has been proven to be a bug within the MyApp.exe application. The next step is to contact the MyApp.exe vendor and let them know about the bad news J

Troubleshooting IIS 100% CPU issues - Step by Step (intermediary)

Marcelo Fartura — Fri, 15 Sep 2006 02:17:00 GMT

Processes consuming 100% of a system's CPU time is such a situation everybody has faced at least once (likely much more than once :)), and normally as solution we just "kill the murderer" by stopping the culprit process. But what was really causing that process to behave like that? If you can't properly answer this question you might be convicted to repeat the "kill the murderer" process forever...

The procedures we will see below are general enough for troubleshoot any application causing the 100% CPU situation, however I will use IIS as an example here since, as a custom application's host, it is one of the products whereby this situation happens very often.

Notice: This has been classified as an intermediary level content, so it presumes you understand concepts like memory dumps, crashes, hangs, applications, processes, threads, call stacks etc, and also have some experience with debugging.

Before the problem happens...

Step 1:
Make sure you are familiar with and have the proper tools we will need for the further steps. We will need the System Monitor (perfmon.exe), the Debugging Tools for Windows package installed (preferably installed at the c:\debuggers folder) and if possible the proper symbol files.

You will also need some basic understanding about the IIS architecture.

Step 2:
Run the System Monitor (perfmon.exe), in a remote machine if possible, and create a new Counter Log: Chose any name, lets say "Performance_log". After confirming the name you've just chosen, in the General Tab, click in Add Objects and choose "Select counter objects from comupter". In the combo list, select or type the remote computer name (the one where the problem happens and which you're going to monitor)

Select the following objects::counters:

Processor(*)\*
Process(*)\*
Threads(*)\*

Start monitoring and wait until the server gets the high CPU utilization issue

When the problem happens...

Step 3:
First confirm that is an IIS process (any w3wp.exe for IIS6, inetinfo.exe or any dllhost.exe for IIS5/5.1 of inetinfo.exe or any MTX.exe - IIS4) that is consuming the CPU time. You can use even the task manager (CPU and Process columns) to confirm that.

After identifying which process is consuming the CPU time, take a note of the Process ID of such a process from task manager. You can also use other tools like tasklist.exe (builtin for Windows XP and Windows 2003) or tlist.exe from the Debugging Tools for Windows. For IIS5 servers, if you want to know what application is being hosted by the process (in case of a DLLHOST.EXE) you can use the Component Services mmc or simply run the following command from a command prompt:

c:\debuggers\>tlist -k | find /i "dllhost.exe"

It will list all the DLLHOST.EXE process and the respective applications being hosted by the processes in following output format:

C:\Debuggers>tlist -k | find /i "dllhost"
3496 dllhost.exe     MyCom+ Application 01
4892 dllhost.exe     MyCom+ Applicatoin 02
4900 dllhost.exe     IIS Out_Of_Proc_Pooled
3840 dllhost.exe     Mts:   System Application

For the IIS4, the same assumption is valid except that you have MTS Explorer instead and the surrogate name will change from dllhost.exe to mtx.exe.
For the IIS6 however, you can't use Component Services (IIS6 doesn't rely on COM+ for creating the surrogates as IIS5 does) nor tlist.exe to identify the applications being hosted by the processes. You will need to use the script iisapp.vbs instead (it's built in script for Windows 2003). Run iisapp from a command prompt with no parameters and it will return a list with the w3wp.exe processes, their PID and the application pools being hosted by them.

Step 4:
Once you've identified the PID (process ID) the CPU consumer you're ready to generate a hang memory dump (please note that the dump will contain useful information only, and if only, it was taken while the problem was happening). To generate the dump execute the following commands from a command prompt:

C:\Debuggers\>cscript adplus.vbs -hang -p <PID> -quiet -o c:\dumps

Adplus will spawn a debugger (cdb.exe) which will attach to the process, generate the hang dump and dettach itself. If you're familiar with the Debug Diagnostics Tool package (DebugDiag), feel free to use it. It doesn't matter which tool you will use as long as you generate a good full dump.

Generally is a good practice to generate more 1 or 2 dumps repeating above procedure 1 minute apart from each other. This might be useful when analyzing the problem cause.

Once the adplus.vbs is done, stop the perfmon log. At this point you can also apply the workaround (likely kill the process) you've been applying to make your server reponsive again.

Analyzing the data...

Step 5:
Open the perfmon log:

Run perfmon , select System Monitor on the left navigation bar and type Ctrl + Q to open the properties window.
Click in the Source tab and select Log Files. Click add and select the c:\perflogs\Performance_log000001.blg file and click ok.

Press Ctrl + I, or press the "+" button, to add objects and counters.
Select Process object, and the %Processor Time counter, select all the IIS related processes, click add then close.

The system monitor window will show the %CPU utilization of each process. Press Ctrl+ B to change the view to histogram. Determine the highest bar and double click it.

Press Ctrl+ H to confirm you selected the highest bar on the histigram view. Determine which process is the one that owns the highest bar. It must to be the same as the one you identified on the task manager as the culprit.

Press Ctrl + E to clean the histogram window and press CTRL + I to include new counters.

Select thread object, select %Processor Time counter and select all the instances of the process you determined as the culprit one.

Repeat the same process as before to determine which thread is the one consuming CPU - tip: Histogram is the best view to do that.

Once you know the process and thread within the process that is consuming the CPU, clean the perfmon System Monitor window one more time - Ctrl + E

Press Ctrl + I and select the Thread object again. Select the ID thread counter and select the instance of the previous determined thread. Click add and close. See how to simplify the Thread and Process ID identification in the perfmon in this article.

In the System Monitor Window, determine the thread ID of thread by looking at Min, Max and Avg values. THey should be all the same. If they are not the same is because you have monitored for a longer period than the high CPU utilization one. If that happened press Ctrl + Q, go to source tab and move the Time WIndow slide in order to get only the period of high utilization CPU, then go back to System Monitor Tool and chek for thread ID again.

Convert the Thread ID value from decimal to hexadecimal and take a note of this hex value.

How about some debugging now?

Step 6:
Open the Windbg.exe from the c:\debuggers folder.
Press Ctrl + D and select the hang dump (.dmp file) you generated earlier by uging adplus.vbs (they should be placed in the folder c:\dumps based on the option "-o c:\dumps" used before)

Close the Disassembly window if it opens (some versions of Windbg will do it autommaticaly)

Make sure your symbol path is properly set (remember to use the command !symfix or set it manually to http://msdl.microsoft.com/download/symbols) and run the .reload /s command if you needed to change the symbol path.

On the Windbg's command prompt type ~ and press enter. A thread list will show up as the example below:

0:00> ~
   0 Id: f4.f3 Suspend: 1 Teb: 7ffde000 Unfrozen
   1 Id: f4.f7 Suspend: 1 Teb: 7ffdd000 Unfrozen
   2 Id: f4.10a Suspend: 1 Teb: 7ffdb000 Unfrozen
   3 Id: f4.10f Suspend: 1 Teb: 7ffda000 Unfrozen
   4 Id: f4.110 Suspend: 1 Teb: 7ffd9000 Unfrozen
   5 Id: f4.111 Suspend: 1 Teb: 7ffd8000 Unfrozen
   6 Id: f4.112 Suspend: 1 Teb: 7ffd7000 Unfrozen
   7 Id: f4.115 Suspend: 1 Teb: 7ffd6000 Unfrozen
   8 Id: f4.116 Suspend: 1 Teb: 7ffd5000 Unfrozen
   9 Id: f4.117 Suspend: 1 Teb: 7ffd4000 Unfrozen
10 Id: f4.11a Suspend: 1 Teb: 7ffae000 Unfrozen
11 Id: f4.11b Suspend: 1 Teb: 7ffad000 Unfrozen
12 Id: f4.129 Suspend: 1 Teb: 7ffac000 Unfrozen
13 Id: f4.12a Suspend: 1 Teb: 7ffab000 Unfrozen
14 Id: f4.12e Suspend: 1 Teb: 7ffaa000 Unfrozen

The Id: identificator above is followed by the PID.TID where PID is the hex representation of the process ID and TID is the hex representation of the thread ID.

Find the one which has the thread ID you determined before. Tip: you can convert from hex to decimal by the the "?" in the debugger's command prompt. Ex.: "? f4" will generate the following output:

0:00> ? f4
Evaluate expression: 244 = 000000f4

Once you identified the thread in the list you have two options to set this thread's context the active one. You run the following command with using the TID:

~~[TID]s

Or you run the following command using the corresponding thread number (#):

~#s

Both command will produce the same result which is set the thread's context the active one in the debugger. Now you can check the thread stack to see what is causing the high CPU utilization by just typing one of stack commands (k). For example, the kb command will show you the stack with the some additional information like the three first arguments passed through the functions as below:

0:065> kb
ChildEBP RetAddr Args to Child
07c7bc20 623016d6 60a27460 07c7bff8 00000200 oran8!nnciqdn+0xab
07c7bd3c 60b2695a 60a27460 00298880 07c7bff8 orantns8!nnftboot+0x6d6
07c7bd68 60b25881 00298880 07c7bff8 00000200 oran8!nnfgqdn+0x3a
07c7bdc0 60a10acf 60a27460 07c7e630 00000000 oran8!nnfgrne+0x2c1
07c7c418 60b6e222 60a27460 07c7e630 00000016 oranl8!nlolgobj+0x6df
07c7c458 60b27be0 60a27460 07c7e630 00298930 oran8!sntseltst+0xc072
07c7c488 60b16472 60a27460 07c7e630 000000ff oran8!nnfsn2a+0x30
07c7d600 60b60ebc 60a27460 00001803 07c7e630 oran8!niqname+0x252
07c7f76c 604b0c93 07c7e630 00000007 00001803 oran8!osncon+0x23c
07c7f94c 604b849e 07012e44 14414d22 00000007 ORACLIENT8!xaolog+0x1f533
07c7f970 604b84ea 14414df8 14414d22 00000007 ORACLIENT8!xaolog+0x26d3e
07c7f98c 604ba15d 14414df8 14414d22 00000007 ORACLIENT8!xaolog+0x26d8a
07c7fe44 604bb128 14414df8 14414cf6 0000002b ORACLIENT8!xaolog+0x289fd
07c7fe84 6047fed4 14414df8 14414cf6 ffffffff ORACLIENT8!xaolog+0x299c8
07c7febc 604614a0 14414df8 14414cf6 ffffffff ORACLIENT8!upilog+0x64
07c7fef0 10002642 14414db8 14414df8 14414cf6 ORACLIENT8!ocirlo+0xb0
07c7ff84 7800265a 14414b90 01b0ed14 7800ef03 ociw32!olog+0x62
07c7ffb8 77f04ef0 00297850 01b0ed14 7800ef03 MSVCRT!beginthreadex+0xb2
07c7ffec 00000000 780025ff 00297850 00000000 KERNEL32!lstrcmpiW+0xbe

By looking at thread stack you should be able to identify what is causing the high CPU utilization problem. You can always unassembly (u) from the intruction pointer to try digging a little bit more and obtain some additional information. Ex.:

0:065> u eip
oran8!nnciqdn+0xab:
60b1fefb 8b10            mov     edx,dword ptr [eax]
60b1fefd 83c004          add     eax,4
60b1ff00 52              push    edx
60b1ff01 50              push    eax
60b1ff02 8b4510          mov     eax,dword ptr [ebp+10h]
60b1ff05 53              push    ebx
60b1ff06 50              push    eax
60b1ff07 56              push    esi

I will post a real case analysis very soon in this blog...