Mfartura's blog : Assembler

Real Case: Random apps running 100% CPU (advanced)

Marcelo Fartura — Wed, 28 Feb 2007 05:18:00 GMT

This is another 100% CPU issue. This time I’m not working on any customer case as the issue is happening on my own Windows XP SP2 machine. So I resolved trying to troubleshoot that just for fun J.

First, let me explain the scenario:

Random interactive applications, like Outlook, Word, Excel, Explorer, Internet Explorer etc, were unexpectedly consuming 100% CPU. Once they started eating up all the CPU they get completely hung and do not stop using all the CPU until I actually kill their processes. Some times after killing their process and restarting the application the issue was gone, some times the issue comes back right after the application restarts and I need to kill it again…

Ok, I made my part playing as an end user by just killing the processes for some time but it started buzzing me as it was becoming more frequent. So now it’s time have some fun and go to the core of the issue.

All I need to do is follow the steps in the post Lab: Win32 Application Causing 100% Condition. So let’s go start doing that:

Once the problem happened next time, it was in the infopath.exe application, I attached the Windbg (from the Debugging Tools For Windows) and executed the command below in order to find out which thread I should look at:

Ok, so now I know the thread 0 is the one I’m looking for since it’s the one who’s been consumed the CPU for the last 40 seconds. You might be asking “what if there were other threads also consuming the CPU?” Well, for this specific issue I knew we would find only one single thread since my machine is a dual CPU and whenever the problem happens the app takes only one of the CPUs.

With that said, let’s take a look at the call stack of the thread 0:

Ok, so we don’t have that much… the return address is 0x0 and ebp register is 0x0 so since we don’t have the proper symbols to work with this I guess we won’t be able to rebuild the stack. So we will have even more fun diving in the ASM from now on!!!

We already know the application nview.dll is the one loaded in the call stack. Let’s see what is it doing or trying to do. Our current instruction pointer tells us this:

So all we need to do to follow the assembler execution path is to get the current CPU registers value and start reproducing the effects the real instructions would have. The current value of the CPU registers is below:

Notice that both EAX and EDX (the two registers used by the first operation) are 0x0. So the first instruction mov eax, edx won’t change anything. It will make a copy of the EDX register to the EAX but since they are both 0x0 this will result in no changes at our registers. Let’s take a look at the subsequent instructions and their effects:

and eax, 01000000h

It will make an AND logical operation between EAX and the value 0x1000000 and load the result at the EAX register. Again, this won’t change the EAX value since 0x0 AND 0x1000000 results in 0x0 which is already the current EAX value.

xor ebp,ebp

It will make an Exclusive-OR operation, or XOR, between the stack base pointer EBP and itself and load the result at the EBP (which is the destination operand on this operation). Needless to say this would load EBP with 0x0, however as we can notice below from the current register values, EBP is already 0x0:

or eax,ebp

It will make an OR logical operation between EAX and EBP and load the result at the EAX. We already know this will be the equivalent of making 0x0 OR 0x0 which will result in guess what… Exactly! – It will result in 0x0 which doesn’t change the original value of EAX.

je nview!NVLoadDatabase+0xc8a (100020ea)

This will cause the instruction pointer EIP to “jump” to 100020ea just in case the operands of the previous OR operation were equal in value. Well, EAX is indeed equal, in value, to EBP since they’re both 0x0. So the EIP is loaded with the value 100020ea and our assembly execution path is properly changed as below:

So, let’s continue from the instruction located at 100020ea.

shl edx,8

It will make a shift left on the EDX value of 8 positions. EDX is a 32-bit representation of the value 0x0, so shifting it 8 positions wouldn’t change anything. EDX remains 0x0 after the operation as we can see below:

mov eax,edx

This one we already know. It won’t change anything as EAX = EDX = 0x0.

sar eax,8

It will a shift right on the EAX of 8 positions. As we’ve seen before with the EDX and the shift left operation, this won’t change anything as EAX = 0x0.

add eax,esi

Finally EAX changed its original value of 0x0 to the result of the sum between itself and ESI register. ESI = 0003fff8 and so EAX will from now on be EAX = 0003fff8 as can see below:

sar edx,1Fh

Again, a shift righ operation on the EDX with 31 positions. EDX will remain with 0x0 after that as we’ve seen before.

cdq

This will convert the value of EAX to a 64-bit value. But it will keep it original value of 0003fff8.

mov eax, esi

It will copy the value of ESI to EAX, however as we noticed before EAX was made equal to ESI by the operation add eax, esi previously executed (since EAX was 0x0 by that time). So this operation doesn’t change anything either.

lea eax,[ecx+esi]

This will load EAX with the value resultant from the operation ECX + ESI. So let’s make a little math here… ECX + ESI = 00c90000 + 0003fff8 = 00ccfff8. So EAX will be from now on = 00ccfff8 as we see below:

cmp eax, edi

This will compare the values of EAX and EDI and change the proper flags accordingly.

mov dword ptr [esp+14h],edx

This will copy the EDX value to the memory location referred by the stack pointer plus 20 bytes, or in other words, this will copy EDX value to 20 bytes down on the stack. So, doing a little math again we will have that such a position would be the ESP = 0012c6c0 plus 20 bytes, so 0012c6c0 + 0x14 = 0012c6d4. Let’s see what is current content of 0012c6d4:

All right, so it’s currently 0x0 and we are copying the value of EDX which is also 0x0 to that location in the memory. So, we won’t actually change anything here. Let’s move on.

jb nview!NVLoadDatabase+0xc60 (100020c0)

This will make our execution path jumping to 100020c0 just in case the result of the operation cmp previously executed, tells us the value of EAX is lower (or below) EDI. By the time the cpm operation was executed, EAX was 00xxfff8 and EDI was 00cd0000 so EAX is indeed below EDI and will therefore jump in the execution path as below:

So we moved back some bytes on our execution path. Let’s keep moving…

mov edx, dword prt [eax]

It will copy the value referred by EAX to the register EDX. The value currently referred by EAX is the following:

So we’re just copying 0x0 to EDX which is already 0x0, so it doesn’t change our register’s value.

mov eax, dword ptr [eax+4]

This will copy the value referred by the EAX plus 4 bytes to EAX. The valued pointed by EAX + 4 is the value pointed by 00ccfffc which is 0x0. So now we’re basically making EAX = 0x0 again – As we had in the very beginning of our execution path.

mov dword ptr [esp+14h],eax

This will just copy the EAX value to a specific position on the stack (20 bytes under the top pointed by the ESP). So it doesn’t change any register and it actually doesn’t change the stack value either since EAX is currently 0x0 once again and we had already saved the value 0x0 at that same position on the stack when executed the instruction located at 100020ff.

Now hold on: The next instruction is exactly the one we executed in the very first step of all this investigation. And guess what? The registers and the stack have exactly the same values as we had in the very beginning. So we don’t need to move forward on this to conclude that we will end up again in the same place, and again, and again, and again… indefinitely! This turns out to be an infinite loop!!

Conclusion:

This explains why this thread is using 100% of our CPU and concludes our investigation. Unfortunately this is something we can’t fix since it’s happening within the nview.dll’s code and this is not a MS product.

NVIEW.DLL is a NVidia module apparently used by the NVidia Desktop Manager software. I’ve done some research through the NVidia web site and I haven’t found any reference to this so I believe there is no definitive solution for this yet. As a workaround I simply disabled the Desktop Manager feature and everything seems to be fine now.

Lab: Win32 Application causing 100% CPU condition (Advanced)

Marcelo Fartura — Mon, 18 Sep 2006 21:32:00 GMT

This is lab for simulating a real 100% CPU condition being caused by a generic Win32 application when there are no symbols available for such application.

Let’s start supposing the situation is currently occurring. So, the first step is to confirm the symptoms, which we can easily do by using, for instance the task manager. As we see in the picture below we have a 50% utilization on a dual core machine which represents 100% of one CPU being utilized.

If we navigate to the process tab in the task manager we will be able to identify which process is causing the problem as shown in the picture below:

Ok, so it’s the MyApp.exe that is keeping one of our CPUs totally busy. It's time to bring the debugger and look what’s happening under the hood…

By using the Windbg from the Debugging Tools For Windows package, we can choose, by pressing F6, which process we want to attach the debugger to:

After attaching to the MyApp.exe process, the debugger will, by default, list all the modules loaded at the address space as well as it will break the execution through the break instruction exception (80000003). Now, since the execution is broken, our CPU is no longer busy and we can start investigating what is going on here.

Microsoft (R) Windows Debugger Version 6.6.0005.0

Executable search path is:

ModLoad: 00400000 00446000 C:\Temp\MyApp.exe

ModLoad: 7c900000 7c9b0000 C:\WINDOWS\system32\ntdll.dll

ModLoad: 7c800000 7c8f4000 C:\WINDOWS\system32\kernel32.dll

ModLoad: 77d40000 77dd0000 C:\WINDOWS\system32\USER32.dll

ModLoad: 77f10000 77f57000 C:\WINDOWS\system32\GDI32.dll

ModLoad: 76b40000 76b6d000 C:\WINDOWS\system32\WINMM.dll

ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll

ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll

ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll

ModLoad: 76c30000 76c5e000 C:\WINDOWS\system32\WINTRUST.dll

ModLoad: 77a80000 77b14000 C:\WINDOWS\system32\CRYPT32.dll

ModLoad: 77b20000 77b32000 C:\WINDOWS\system32\MSASN1.dll

ModLoad: 76c90000 76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll

ModLoad: 72d20000 72d29000 C:\WINDOWS\system32\wdmaud.drv

ModLoad: 72d10000 72d18000 C:\WINDOWS\system32\msacm32.drv

ModLoad: 77be0000 77bf5000 C:\WINDOWS\system32\MSACM32.dll

ModLoad: 77bd0000 77bd7000 C:\WINDOWS\system32\midimap.dll

ModLoad: 5ad70000 5ada8000 C:\WINDOWS\system32\uxtheme.dll

ModLoad: 74720000 7476b000 C:\WINDOWS\system32\MSCTF.dll

ModLoad: 77120000 771ac000 C:\WINDOWS\system32\OLEAUT32.DLL

ModLoad: 774e0000 7761d000 C:\WINDOWS\system32\ole32.dll

ModLoad: 10000000 1001e000 C:\Program Files\Babylon\CAPTLIB.DLL

ModLoad: 77f60000 77fd6000 C:\WINDOWS\system32\SHLWAPI.dll

ModLoad: 74c80000 74cac000 C:\WINDOWS\system32\OLEACC.dll

ModLoad: 76080000 760e5000 C:\WINDOWS\system32\MSVCP60.dll

(8a8.c5c): Break instruction exception - code 80000003 (first chance)

eax=7ffd6000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005

eip=7c901230 esp=00fcffcc ebp=00fcfff4 iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246

ntdll!DbgBreakPoint:

7c901230 cc int 3

Let’s make sure you have the right symbol path so we can load at least the MS public symbols.

0:003> .sympath

Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols

We have 4 threads as below:

0:003> ~

0 Id: 8a8.da8 Suspend: 1 Teb: 7ffdf000 Unfrozen

1 Id: 8a8.f1c Suspend: 1 Teb: 7ffde000 Unfrozen

2 Id: 8a8.75c Suspend: 1 Teb: 7ffdd000 Unfrozen

. 3 Id: 8a8.c5c Suspend: 1 Teb: 7ffdc000 Unfrozen

Our current context is set to thread 03.

The more reliable way to find out which thread is consuming the processor time would be use the System Monitor tool with the right objects and counters (process and thread objects) as described in the article Troubleshooting IIS 100% CPU issues - Step by Step. However we will try something different here. We will use the extension command !uext.runaway or simply !runaway (since the extension uext.dll is loaded by default when debugging user mode apps from Windbg.exe):

0:003> !uext.runaway

User Mode Time

Thread Time

0:da8 0 days 0:03:36.703

3:c5c 0 days 0:00:00.000

2:75c 0 days 0:00:00.000

1:f1c 0 days 0:00:00.000

The !runaway extension command returns a list containing all the current threads and the processor time spent by them. As we see in the result above our thread 0 has to be the one we are interested in since it's not only the one with more time, but is also the only one with any processor time.

So let’s change the context to thread 0 in order to verify what has been executed by it:

0:003> ~0s

*** WARNING: Unable to verify checksum for C:\Temp\MyApp.exe

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Temp\MyApp.exe -

eax=00000000 ebx=00000000 ecx=0012fd7c edx=7c90eb94 esi=00402150 edi=0012fe40

eip=00401a73 esp=0012fd74 ebp=0012fd74 iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246

MyApp!AboutDlgProc+0xa73:

00401a73 5d pop ebp

There is a message warning us about the lack of symbols. Let's see what we can find about what this thread is doing even with no symbols for the application. Looking at the call stack we have the following:

0:000> kL

ChildEBP RetAddr

WARNING: Stack unwind information not available. Following frames may be wrong.

0012fd74 00401a69 MyApp!AboutDlgProc+0xa73

0012fd80 0040248f MyApp!AboutDlgProc+0xa69

0012fdd8 77d48734 MyApp!WndProc+0x33f

0012fe04 77d48816 USER32!InternalCallWinProc+0x28

0012fe6c 77d489cd USER32!UserCallWinProcCheckWow+0x150

0012fecc 77d496c7 USER32!DispatchMessageWorker+0x306

0012fedc 00401244 USER32!DispatchMessageA+0xf

0012ff30 00402cd3 MyApp!AboutDlgProc+0x244

0012ffc0 7c816fd7 MyApp!WndProc+0xb83

0012fff0 00000000 kernel32!BaseProcessStart+0x23

This just tells us, based on the first frame, that this is the first thread and also it has some MyApp.exe code being executed (we have a chain of at least 5 MyApp.exe functions in the stack). But with no symbols this won't help us any longer since the function names for MyApp.exe are not reliable. Time to look at the asm being executed...

The current instruction pointer (represented by the CPU register EIP) is pointing to this asm:

0:000> u eip

MyApp!AboutDlgProc+0xa73:

00401a73 5d pop ebp

00401a74 c3 ret

00401a75 cc int 3

As we can see, it's just taking the top value on the stack and saving it to the CPU register EBP (pop EBP) and the next instruction is setting the EIP to the return address (this will load the EIP with the instruction located at the address at EBP + 4 in the memory). We can identify this asm block as part of epilog which means we are at the end of it's execution. So let's look a little bit before in the asm to verify what was being executed before the epilog block:

0:000> u eip -13 eip+1

MyApp!AboutDlgProc+0xa60:

00401a60 55 push ebp

00401a61 8bec mov ebp,esp

00401a63 51 push ecx

00401a64 e805000000 call MyApp!AboutDlgProc+0xa6e (00401a6e)

00401a69 8945fc mov dword ptr [ebp-4],eax

00401a6c ebf6 jmp MyApp!AboutDlgProc+0xa64 (00401a64)

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

Now we got the complete block since the execution of the prologue (the push EBP; mov EBP, ESP; push ECX instructions). By analyzing the block above we see that after the prolog execution, we make a function call to the instrution at 00401a6e (call MyApp!AboutDlgProc+0xa6e), so let's verify what should be executed when doing that:

0:000> u 00401a6e

MyApp!AboutDlgProc+0xa6e:

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

00401a74 c3 ret

All right, so what we see here, after the prolog block, is an exclusive OR (XOR) operation being executed using EAX as both operands, and then the epilog block is executed - This epilog block is the same one (the instruction at the address 00401a73) our EIP is currently set to (notice the first asm block at the beginning of this analysis).

The XOR operation is just being used to set the EAX register to 0 (the result of the operation is loaded to the destination operand, in this case the EAX itself). If we have a chance to look at the application source code, this is likely something like "MyLocalVar = 0;". So it comes the question: Why not make a "mov EAX,0" instead of a "xor EAX,EAX"? The answer is simple: it's a matter of cost. The instruction XOR costs less than the instruction MOV to the CPU, so it's faster to execute a XOR since in this case it will produce the same desired result.

So what we know so far is that we have a function being called and the only thing this function does is set a variable to 0 and returns. So let's keep going from the return of this function in the following block:

MyApp!AboutDlgProc+0xa60:

00401a60 55 push ebp

00401a61 8bec mov ebp,esp

00401a63 51 push ecx

00401a64 e805000000 call MyApp!AboutDlgProc+0xa6e (00401a6e)

00401a69 8945fc mov dword ptr [ebp-4],eax

00401a6c ebf6 jmp MyApp!AboutDlgProc+0xa64 (00401a64)

00401a6e 55 push ebp

00401a6f 8bec mov ebp,esp

00401a71 33c0 xor eax,eax

00401a73 5d pop ebp

We've just returned from the instruction "call MyApp!AboutDlgProc+0xa6e". So the next instruction to be executed will the one at the address 00401a69 which is just moving the EAX value (which we already know at this point should be 0) to the location of EBP - 4. EBP - 4 is known to be the location of the first local variable of the current function at the call stack, so we're basically copying the EAX value to one or our local variables.

The next instrutcion (jmp MyApp!AboutDlgProc+0xa64) will cause the execution to "jump" to the address 00401a64 which, at this point, is already familiar to us. We know that the instruction at 00401a64 will make a function call to the function MyApp!AboutDlgProc+0xa6e which we analyzed before and know it's just setting the EAX to 0 and returning.

The conclusion now is obvious. We have an infinite loop being executed and this is the reason we have a 100% CPU condition happening. This has been proven to be a bug within the MyApp.exe application. The next step is to contact the MyApp.exe vendor and let them know about the bad news J