Mfartura's blog

Marcelo Fartura's blog about debugging and general troubleshooting

Kernel dump analysis - Bugcheck 0xA (IRQL_NOT_LESS_OR_EQUAL)

Kernel dump analysis - Bugcheck 0xA (IRQL_NOT_LESS_OR_EQUAL)

  • Comments 2
  • Likes

Yet another kernel memory dump to be analyzed - The bugcheck this time is the 0xA - IRQL_NOT_LESS_OR_EQUAL.   To better understand what this message means we would need a little background on Windows Internals but basically when executing anything at a interrupt request level (IRQL) = 2 or higher (in normal circumstances instructions get executed at IRQL = 0) we can't page fault.   On the situation of this specific dump, we would crash even on IRQL = 0 since we're trying to write to the memory address 0x0 which is in user mode address space and is reserved (the first useable address in the user mode address space is 0x00010000)

 

Let's jump to the analysis.

  

Bugcheck info:

0: kd> .bugcheck

Bugcheck code 0000000A

Arguments 00000000 00000002 00000001 8087bb19

                                |              |                    |                        |à This is the failing instruction address

                                |              |                    |à 0x1 means WRITE, 0x0 means READ, so it crashed when trying to WRITE to 0x0

                                |              |à This is the IRQL at which problem happened (IRQL=2)

                                |à This is the memory address where the problem happened

 

So it crashed basically because it failed to access a memory address (which at a lower IRQLs would result in a page fault) on the IRQL = 2.  At IRQL = 2 and above page faults are not allowed to happen.

 

Current thread stack:

 

0: kd> kvnL

 # ChildEBP RetAddr  Args to Child             

00 b7d6ebbc 8087bb19 badb0d00 00000000 00000000 nt!_KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ b7d6ebbc)

01 b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f (FPO: [Non-Fpo]) (CONV: stdcall)

02 b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d (FPO: [Non-Fpo]) (CONV: stdcall)

03 b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)

04 b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67 (FPO: [Non-Fpo]) (CONV: fastcall)

WARNING: Stack unwind information not available. Following frames may be wrong.

05 b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada

06 b7d6ecf0 b775b520 f82cde28 f82cde28 8b572548 naiavf5x+0x737a

07 b7d6ed04 8081dcdf 8b478990 f82cde28 f82cde38 naiavf5x+0x2520

08 b7d6ed18 808f8be4 00000000 00000000 f7ca8448 nt!IofCallDriver+0x45 (FPO: [Non-Fpo]) (CONV: fastcall)

09 b7d6ed50 80932ca2 00ca8460 f7ca8460 00000001 nt!IopDeleteFile+0x13a (FPO: [Non-Fpo]) (CONV: stdcall)

0a b7d6ed68 80932cec f7ca8460 00000001 8ad47770 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)

0b b7d6ed80 8087f92f 00000000 00000000 8ad47770 nt!ObpProcessRemoveObjectQueue+0x36 (FPO: [1,0,0]) (CONV: stdcall)

0c b7d6edac 80948bd0 00000000 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo]) (CONV: stdcall)

0d b7d6eddc 8088d4e2 8087f844 80000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo]) (CONV: stdcall)

0e 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

 

The current frame being executed is a post-exception frame, I mean, it's just handling the situation.  The problem really happened here:

 

0: kd> .trap b7d6ebbc

ErrCode = 00000002

eax=00000000 ebx=ef4d0848 ecx=00000000 edx=00000000 esi=f7ca7008 edi=00000000

eip=8087bb19 esp=b7d6ec30 ebp=b7d6ec40 iopl=0         nv up ei ng nz na po nc

cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010282

nt!ExDeleteResourceLite+0x1f:

8087bb19 8901            mov     dword ptr [ecx],eax  ds:0023:00000000=????????

0: kd> kvnL

  *** Stack trace for last set context - .thread/.cxr resets it

 # ChildEBP RetAddr  Args to Child             

00 b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f (FPO: [Non-Fpo]) (CONV: stdcall)

01 b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d (FPO: [Non-Fpo]) (CONV: stdcall)

02 b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)

03 b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67 (FPO: [Non-Fpo]) (CONV: fastcall)

WARNING: Stack unwind information not available. Following frames may be wrong.

04 b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada

05 b7d6ecf0 b775b520 f82cde28 f82cde28 8b572548 naiavf5x+0x737a

06 b7d6ed04 8081dcdf 8b478990 f82cde28 f82cde38 naiavf5x+0x2520

07 b7d6ed18 808f8be4 00000000 00000000 f7ca8448 nt!IofCallDriver+0x45 (FPO: [Non-Fpo]) (CONV: fastcall)

08 b7d6ed50 80932ca2 00ca8460 f7ca8460 00000001 nt!IopDeleteFile+0x13a (FPO: [Non-Fpo]) (CONV: stdcall)

09 b7d6ed68 80932cec f7ca8460 00000001 8ad47770 nt!ObpRemoveObjectRoutine+0xde (FPO: [Non-Fpo]) (CONV: stdcall)

0a b7d6ed80 8087f92f 00000000 00000000 8ad47770 nt!ObpProcessRemoveObjectQueue+0x36 (FPO: [1,0,0]) (CONV: stdcall)

0b b7d6edac 80948bd0 00000000 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo]) (CONV: stdcall)

0c b7d6eddc 8088d4e2 8087f844 80000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo]) (CONV: stdcall)

0d 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

 

So it's crashing when trying to write to the address pointed by ECX - mov dword ptr [ecx], eax - This is the assembly code of the function where it crashed from the start to the crash instruction :

 

0: kd> u nt!ExDeleteResourceLite nt!ExDeleteResourceLite+0x1f + 1

nt!ExDeleteResourceLite [d:\nt\base\ntos\ex\resource.c @ 2250]:

8087bafa 8bff            mov     edi,edi

8087bafc 55              push    ebp

8087bafd 8bec            mov     ebp,esp

8087baff 83ec0c          sub     esp,0Ch

8087bb02 56              push    esi

8087bb03 8d55f4          lea     edx,[ebp-0Ch]

8087bb06 b9c0418b80      mov     ecx,offset nt!ExpResourceSpinLock (808b41c0)

8087bb0b ff1508118080    call    dword ptr [nt!_imp_KeAcquireInStackQueuedSpinLock (80801108)]

8087bb11 8b7508          mov     esi,dword ptr [ebp+8]  ßESI is obtaining its value from the content of EBP+8

8087bb14 8b4e04          mov     ecx,dword ptr [esi+4]  ß ECX is obtaining its value from the content of ESI+4

8087bb17 8b06            mov     eax,dword ptr [esi]

8087bb19 8901            mov     dword ptr [ecx],eax  ß This is where it crashed

 

 

ECX is getting its value from the address pointed by ESI + 4, and ESI is getting its value from the address pointed by EBP + 8 which is a pointer to the first parameter passed from the previous function on the stack.  Confirming:

 

0: kd> dd esi+4 l1

f7ca700c  00000000

0: kd> dd poi(ebp+8) l1

f7ca7008  00000000

 

Next step is look to the previous function on the stack.  The assembly for the next function up to the point it calls the current one is this:

 

 

0: kd> u nt!SepTokenDeleteMethod nt!SepTokenDeleteMethod+0x9d

nt!SepTokenDeleteMethod [d:\nt\base\ntos\se\token.c @ 2652]:

80972a66 8bff            mov     edi,edi

80972a68 55              push    ebp

80972a69 8bec            mov     ebp,esp

80972a6b 51              push    ecx

80972a6c 51              push    ecx

80972a6d 56              push    esi

80972a6e 8b7508          mov     esi,dword ptr [ebp+8] ß ESI is coming from the content  EBP + 8  (first parameter passed from the previous function)

80972a71 f6868800000020  test    byte ptr [esi+88h],20h

80972a78 57              push    edi

80972a79 7538            jne     nt!SepTokenDeleteMethod+0x4d (80972ab3)

80972a7b 8bbe94000000    mov     edi,dword ptr [esi+94h]

80972a81 53              push    ebx

80972a82 8d4f0c          lea     ecx,[edi+0Ch]

80972a85 eb0f            jmp     nt!SepTokenDeleteMethod+0x30 (80972a96)

80972a87 8d42ff          lea     eax,[edx-1]

80972a8a 8bd8            mov     ebx,eax

80972a8c 8bc2            mov     eax,edx

80972a8e f00fb119        lock cmpxchg dword ptr [ecx],ebx

80972a92 3bc2            cmp     eax,edx

80972a94 741c            je      nt!SepTokenDeleteMethod+0x4c (80972ab2)

80972a96 8b11            mov     edx,dword ptr [ecx]

80972a98 83fa01          cmp     edx,1

80972a9b 75ea            jne     nt!SepTokenDeleteMethod+0x21 (80972a87)

80972a9d 8b4704          mov     eax,dword ptr [edi+4]

80972aa0 8945f8          mov     dword ptr [ebp-8],eax

80972aa3 8b4708          mov     eax,dword ptr [edi+8]

80972aa6 8945fc          mov     dword ptr [ebp-4],eax

80972aa9 8d45f8          lea     eax,[ebp-8]

80972aac 50              push    eax

80972aad e890ecffff      call    nt!SepDeReferenceLogonSession (80971742)

80972ab2 5b              pop     ebx

80972ab3 8d4638          lea     eax,[esi+38h]

80972ab6 8b08            mov     ecx,dword ptr [eax]

80972ab8 0b4804          or      ecx,dword ptr [eax+4]

80972abb 6a00            push    0

80972abd 5f              pop     edi

80972abe 7407            je      nt!SepTokenDeleteMethod+0x61 (80972ac7)

80972ac0 57              push    edi

80972ac1 50              push    eax

80972ac2 e8b5140000      call    nt!SepModifyTokenPolicyCounter (80973f7c)

80972ac7 8b4678          mov     eax,dword ptr [esi+78h]

80972aca 3bc7            cmp     eax,edi

80972acc 7407            je      nt!SepTokenDeleteMethod+0x6f (80972ad5)

80972ace 57              push    edi

80972acf 50              push    eax

80972ad0 e86ff8f1ff      call    nt!ExFreePoolWithTag (80892344)

80972ad5 8b868c000000    mov     eax,dword ptr [esi+8Ch]

80972adb 3bc7            cmp     eax,edi

80972add 7406            je      nt!SepTokenDeleteMethod+0x7f (80972ae5)

80972adf 50              push    eax

80972ae0 e85766ffff      call    nt!SeFreeCapturedObjectTypeList (8096913c)

80972ae5 8b8690000000    mov     eax,dword ptr [esi+90h]

80972aeb 3bc7            cmp     eax,edi

80972aed 7407            je      nt!SepTokenDeleteMethod+0x90 (80972af6)

80972aef 57              push    edi

80972af0 50              push    eax

80972af1 e84ef8f1ff      call    nt!ExFreePoolWithTag (80892344)

80972af6 8b4630          mov     eax,dword ptr [esi+30h]  ß EAX is coming from the content of ESI + 0x30

80972af9 3bc7            cmp     eax,edi

80972afb 740f            je      nt!SepTokenDeleteMethod+0xa6 (80972b0c)

80972afd 50              push    eax  ß EAX is pushed on the stack as the first parameter

80972afe e8f78ff0ff      call    nt!ExDeleteResourceLite (8087bafa)

 

So once again we need to look at the previous function since this one is also receiving the bad value as parameter.  By revisiting the stack we see the first parameter it passes is also received as below:

 

0: kd> kbL5

ChildEBP RetAddr  Args to Child             

b7d6ec40 80972b03 f7ca7008 80a56be4 00000000 nt!ExDeleteResourceLite+0x1f

b7d6ec5c 80932ca2 ef4d0860 ef4d0848 00000000 nt!SepTokenDeleteMethod+0x9d

b7d6ec74 8086c1a5 ef4d0860 00000000 8b478df0 nt!ObpRemoveObjectRoutine+0xde

b7d6ec94 b7765ada f7ca3444 b7764007 f7ca8460 nt!ObfDereferenceObject+0x67

WARNING: Stack unwind information not available. Following frames may be wrong.

b7d6ecb0 b776037a f7cbf004 b7d6ece0 f82cde28 naiavf5x+0xcada

 

 

As we can go directly to the function nt!ObfDereferenceObject+0x67 and try to see where its setting up that parameter from.  The assembly code from this function to the point it calls the next one is this:

 

0: kd> u  nt!ObfDereferenceObject nt!ObfDereferenceObject+0x67 ß This is a fastcall function so the first two parameters are passed through the registers ECX and EDX instead of using the stack (EBP as a reference)

nt!ObfDereferenceObject [d:\nt\base\ntos\ob\obref.c @ 2441]:

8086c13e 8bff            mov     edi,edi

8086c140 55              push    ebp

8086c141 8bec            mov     ebp,esp

8086c143 51              push    ecx

8086c144 803de0088a8000  cmp     byte ptr [nt!ObpTraceEnabled (808a08e0)],0

8086c14b 53              push    ebx

8086c14c 56              push    esi

8086c14d 57              push    edi

8086c14e 894dfc          mov     dword ptr [ebp-4],ecx ß The location pointed by EBP-0x4 which is local variable is receiving the value of ECX which is the first parameter passed to this function

8086c151 8d71e8          lea     esi,[ecx-18h]

8086c154 7408            je      nt!ObfDereferenceObject+0x20 (8086c15e)

8086c156 6a00            push    0

8086c158 56              push    esi

8086c159 e8c8fcffff      call    nt!ObpPushStackInfo (8086be26)

8086c15e 83cbff          or      ebx,0FFFFFFFFh

8086c161 f00fc11e        lock xadd dword ptr [esi],ebx

8086c165 4b              dec     ebx

8086c166 7547            jne     nt!ObfDereferenceObject+0x71 (8086c1af)

8086c168 8b3d90108080    mov     edi,dword ptr [nt!_imp__KeGetCurrentIrql (80801090)]

8086c16e ffd7            call    edi

8086c170 64a124010000    mov     eax,dword ptr fs:[00000124h]

8086c176 6683787200      cmp     word ptr [eax+72h],0

8086c17b 752c            jne     nt!ObfDereferenceObject+0x6b (8086c1a9)

8086c17d ffd7            call    edi

8086c17f 3c01            cmp     al,1

8086c181 7326            jae     nt!ObfDereferenceObject+0x6b (8086c1a9)

8086c183 803de0088a8000  cmp     byte ptr [nt!ObpTraceEnabled (808a08e0)],0

8086c18a 740f            je      nt!ObfDereferenceObject+0x5d (8086c19b)

8086c18c 833dd8078a8000  cmp     dword ptr [nt!ObpTraceNoDeregister (808a07d8)],0

8086c193 7506            jne     nt!ObfDereferenceObject+0x5d (8086c19b)

8086c195 56              push    esi

8086c196 e805fcffff      call    nt!ObpDeregisterObject (8086bda0)

8086c19b 6a00            push    0

8086c19d ff75fc          push    dword ptr [ebp-4] ß Here it is pushing the content of EBP-4 which we know is coming from ECX (first parameter) as a first parameter to the next function

8086c1a0 e81f6a0c00      call    nt!ObpRemoveObjectRoutine (80932bc4)  ß this is the function call to the next function

 

So we know the bad value is coming from the previous function on the stack which is naiavf5x+0xcada.  By looking at the assembly for that we have this:

 

0: kd> u b7765ac1-10 b7765ad5+2

naiavf5x+0xcab1:

b7765ab1 ff1550a476b7    call    dword ptr [naiavf5x+0x11450 (b776a450)]

b7765ab7 5f              pop     edi

b7765ab8 8bc6            mov     eax,esi

b7765aba 5e              pop     esi

b7765abb 5b              pop     ebx

b7765abc 5d              pop     ebp

b7765abd c20800          ret     8

b7765ac0 56              push    esi

b7765ac1 8bf1            mov     esi,ecx  ß ESI is coming from ECX

b7765ac3 8d86a8000000    lea     eax,[esi+0A8h]

b7765ac9 50              push    eax

b7765aca e8818affff      call    naiavf5x+0x5550 (b775e550)

b7765acf 85c0            test    eax,eax

b7765ad1 7516            jne     naiavf5x+0xcae9 (b7765ae9)

b7765ad3 8bce            mov     ecx,esi ß ECX which will be the first parameter to the next function is coming from ESI

b7765ad5 e8b6f7ffff      call    naiavf5x+0xc290 (b7765290)

 

 

CONCLUSION:

So at this point we can’t move further since we have no symbols for the NAIAVF5X.SYS driver – which is a file system filter driver from McAfee.  The conclusion would be that this driver is bad until the  McAfee analyze this same dump and prove us wrong J.  Here is some addition info about the driver:

 

0: kd> lmvm naiavf5x

start    end        module name

b7759000 b776d440   naiavf5x   (no symbols)          

    Loaded symbol image file: naiavf5x.sys

    Image path: \SystemRoot\system32\drivers\naiavf5x.sys

    Image name: naiavf5x.sys

    Timestamp:        Mon Aug 04 20:08:00 2003 (3F2F0370) ß By the way, this is old stuff.  They might have already corrected this on newer versions of the driver and I wouldn’t be surprised to find some documentation about  this on McAfee’s web site.

 

    CheckSum:         00020D5B

    ImageSize:        00014440

    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

 

By the way, for this dump, “!analyze –v” does the job and points to the right guilty driver.

 

Until the next dump analysis, I mean, next post :)

Comments
  • Hello Marcelo,

    I am trying to apply your debugging steps to a few 0xA dumps we have had over the last few weeks, all were 'probably caused' by TDI.sys. (so I'm guessing something network related)

    The last function called before the stack was: hal!KeAcquireSpinLockRaiseToSynch+0xe

    Any tips on further analyzing this problem? I really find this stuff interesting but I'm guessing I would have to aquire some Assembler knowledge to be able to debug these kind of issues :S

    # ChildEBP RetAddr  Args to Child              

    00 b7104a04 80a5a3de badb0d00 00000000 80a5da73 nt!KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ b7104a04)

    01 b7104a74 b4bb1b6c b7104ae0 8ab6ae48 00000000 hal!KeAcquireSpinLockRaiseToSynch+0xe (FPO: [0,0,0])

    02 b7104a8c b4bb750d 89351002 00000000 00000029 tcpip!TCPDataRequestComplete+0x40 (FPO: [Non-Fpo])

    03 b7104aac b4bb74ad b7104ae0 00003700 88e3e1a8 tcpip!CompleteSends+0x31 (FPO: [Non-Fpo])

    04 b7104aec b4bb6cbc 00000029 3b806af8 8c51e700 tcpip!TcpFastReceive+0x390 (FPO: [Non-Fpo])

    05 b7104bc8 b4bb32bf 8cb63f48 0100007f 0100007f tcpip!TCPRcv+0x723 (FPO: [Non-Fpo])

    06 b7104c28 b4bb14de 00000024 8cb63f48 b4bb6a6d tcpip!DeliverToUser+0x189 (FPO: [Non-Fpo])

    07 b7104cb8 b4bbd163 8cb63f48 b7104d30 00000014 tcpip!IPRcvPacket+0x686 (FPO: [Non-Fpo])

    08 b7104d64 f7658064 b4bf2ee0 8cb63f48 8cf50660 tcpip!LoopXmitRtn+0x195 (FPO: [Non-Fpo])

    09 b7104d80 8088043d 8cb63f48 00000000 8cf50660 TDI!CTEpEventHandler+0x32 (FPO: [Non-Fpo])

    0a b7104dac 80949b7c b4bf2ee0 00000000 00000000 nt!ExpWorkerThread+0xeb (FPO: [Non-Fpo])

    0b b7104ddc 8088e062 80880352 00000001 00000000 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])

    0c 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

    3: kd> .trap b7104a04

    ErrCode = 00000002

    eax=00000002 ebx=00000000 ecx=00000026 edx=00000000 esi=89351040 edi=00000002

    eip=80a5a3de esp=b7104a78 ebp=b7104a8c iopl=0         nv up ei pl nz na po nc

    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010202

    hal!KeAcquireSpinLockRaiseToSynch+0xe:

    80a5a3de f00fba2900      lock bts dword ptr [ecx],0   ds:0023:00000026=????????

    Sincerely,

    Enrico Klein

  • I had the exact same problem on <a href="www.articlewritingclicks.com/.../a> on a McAffee Internet Security version.I switched to another antiv and it never came back.I wonder if they know about it?

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment