Brad Rutkowski's Blog : Debugging

Some useful debugging commands

Brad Rutkowski — Wed, 02 Apr 2008 00:47:00 GMT

All of these are for kernel mode, these are just commands I use often that don't troubleshoot a particular problem, but are helpful in getting a general picture of the system. If you have a specific issue you're trying to understand, drop a note and I'll see if there is a command to help you out.

Vertarget:

Lists Version information for the machine/dump you're debugging. You can also use "version" to tell you about the debugger bits.

1: kd> vertarget
Windows Kernel Version 6001 (Service Pack 1) MP (4 procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Built by: 6001.18000.amd64fre.longhorn_rtm.080118-1840
Kernel base = 0xfffff800`0160c000 PsLoadedModuleList = 0xfffff800`017d1db0
Debug session time: Tue Apr 1 14:29:22.553 2008 (GMT-7)
System Uptime: 0 days 0:03:14.328

!sysinfo

Good utility to check the CPU revs, BIOS revs, etc

1: kd> !sysinfo machineid
Machine ID Information [From Smbios 2.3, DMIVersion 35, Size=3752]
BiosVendor = American Megatrends Inc.
BiosVersion = 080002
BiosReleaseDate = 10/01/2007
SystemManufacturer = Microsoft Corporation
SystemProductName = Virtual Machine
SystemVersion = 5.0
BaseBoardManufacturer = Microsoft Corporation
BaseBoardProduct = Virtual Machine
BaseBoardVersion = 5.0

1: kd> !sysinfo cpuinfo
[CPU Information]
~MHz = REG_DWORD 2660
Component Information = REG_BINARY 0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
Configuration Data = REG_FULL_RESOURCE_DESCRIPTOR ff,ff,ff,ff,ff,ff,ff,ff,0,0,0,0,0,0,0,0
Identifier = REG_SZ Intel64 Family 6 Model 15 Stepping 6
ProcessorNameString = REG_SZ Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
Update Signature = REG_BINARY 0,0,0,0,0,0,0,0
Update Status = REG_DWORD 8
VendorIdentifier = REG_SZ GenuineIntel
MSR8B = REG_QWORD 0

Getting the server name from the dump:

It's quite a bit easier to do internally, but this will get it done too. Good to know you're debugging the right server. :)

1: kd> x srv!SrvComputerName
fffffa60`04024500 srv!SrvComputerName = <no type information>
1: kd> dq fffffa60`04024500
fffffa60`04024500 00000000`00180018 fffff880`04ccd8c0
fffffa60`04024510 00000000`00000000 00000000`00000000
fffffa60`04024520 00000000`00000000 00000000`00000000
fffffa60`04024530 00000000`000c000a fffff880`04a0fc60
fffffa60`04024540 fffffa60`04024540 fffffa60`04024540
fffffa60`04024550 00000000`00060001 fffffa60`04024558
fffffa60`04024560 fffffa60`04024558 00000000`ffffffff
fffffa60`04024570 00000000`00000000 00000000`00000000
1: kd> du fffff880`04ccd8c0
fffff880`04ccd8c0 "BRAD-LHDC-01?"

!running -ti

This will dump the stacks of each thread that is running on each processor

1: kd> !running -ti

System Processors f (affinity mask)
Idle Processors f
All processors idle.

Prcb Current Next
0 fffff80001780680 fffff80001785b80 ................

Child-SP RetAddr Call Site
fffff800`026bb8d0 fffffa60`00a066da nt!KeSetTimer+0x89
fffff800`026bb920 fffffa60`00a06aca NETIO!WfpStartTimerForLeftTime+0x8a
fffff800`026bb970 fffffa60`00a06585 NETIO!WfppLeastRecentlyUsedTimerRoutine+0x1aa
fffff800`026bb9c0 fffffa60`00a067ff NETIO!WfpTimerWheelTimeoutHandler+0x175
fffff800`026bba40 fffff800`016698b3 NETIO!WfpSysTimerNdisCallback+0x4f
fffff800`026bba70 fffff800`0166a238 nt!KiTimerListExpire+0x333
fffff800`026bbca0 fffff800`0166aa9f nt!KiTimerExpiration+0x1d8
fffff800`026bbd10 fffff800`0166bb72 nt!KiRetireDpcList+0x1df
fffff800`026bbd80 fffff800`018395c0 nt!KiIdleLoop+0x62
fffff800`026bbdb0 00000000`fffff800 nt!zzz_AsmCodeRange_End+0x4

1 fffffa60005f3180 fffffa60005fcd40 ................

Child-SP RetAddr Call Site
fffffa60`0171bb08 fffff800`016b03d7 nt!RtlpBreakWithStatusInstruction
fffffa60`0171bb10 fffff800`0165afef nt! ?? ::FNODOBFM::`string'+0x356a
fffffa60`0171bb50 fffffa60`026867a2 nt!KiSecondaryClockInterrupt+0x11f
fffffa60`0171bce8 fffffa60`02685685 intelppm!C1Halt+0x2
fffffa60`0171bcf0 fffff800`0167c7c8 intelppm!C1Idle+0x9
fffffa60`0171bd20 fffff800`0166bb31 nt!PoIdle+0x148
fffffa60`0171bd80 fffff800`018395c0 nt!KiIdleLoop+0x21
fffffa60`0171bdb0 00000000`fffffa60 nt!zzz_AsmCodeRange_End+0x4

!stacks

This is a great utility to check what threads are waiting on for each process. Find out more in the debuggers chm.

1: kd> !stacks 2
Proc.Thread .Thread Ticks ThreadState Blocker

Max cache size is       : 1048576 bytes (0x400 KB)
Total memory in cache   : 0 bytes (0 KB)
Number of regions cached: 0
0 full reads broken into 0 partial reads
    counts: 0 cached/0 uncached, 0.00% cached
    bytes : 0 cached/0 uncached, 0.00% cached
** Prototype PTEs are implicitly decoded
                            [fffffa8000c77950 System]
   4.000008 fffffa8000c774c0 ffffe94b GATEWAIT   nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForGate+0x22a
                                        nt!MmZeroPageThread+0x162
                                        nt!Phase1Initialization+0xe
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16
   4.000010 fffffa8000ca0720 ffffff8c Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        nt!PopIrpWorkerControl+0x22
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16
   4.000014 fffffa8000c78bb0 fffffcb0 Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        nt!PopIrpWorker+0x164
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16

<SNIP>

!PCR

Command will show you some useful info from the processor control block. Like the current thread, next, DPQ queues (Can run !dpcs).

1: kd> !pcr
KPCR for Processor 1 at fffffa60005f3000:
    Major 1 Minor 1
        NtTib.ExceptionList: fffffa60005fd280
            NtTib.StackBase: fffffa60005f6cc0
           NtTib.StackLimit: 000000000554f578
         NtTib.SubSystemTib: fffffa60005f3000
              NtTib.Version: 00000000005f3180
          NtTib.UserPointer: fffffa60005f37f0
              NtTib.SelfTib: 000007fffff8a000

                    SelfPcr: 0000000000000000
                       Prcb: fffffa60005f3180
                       Irql: 0000000000000000
                        IRR: 0000000000000000
                        IDR: 0000000000000000
              InterruptMode: 0000000000000000
                        IDT: 0000000000000000
                        GDT: 0000000000000000
                        TSS: 0000000000000000

              CurrentThread: fffffa60005fcd40
                 NextThread: 0000000000000000
                 IdleThread: fffffa60005fcd40

DpcQueue: 0xfffffa800124dc70 0xfffffa6000e7abe0 [Normal] tcpip!TcpPeriodicTimeoutHandler

1: kd>

!LMI <driver>

When I want to find out ifno about a particular driver in the dump, i use "lm n t" to get all of them, but then !lmi to drill into one. I use it quite often to see if I have the private or public symbol loaded

1: kd> !lmi srv.sys
Loaded Module Info: [srv.sys]
         Module: srv
   Base Address: fffffa6004007000
     Image Name: srv.sys
   Machine Type: 34404 (X64)
     Time Stamp: 47919135 Fri Jan 18 21:57:09 2008
           Size: 94000
       CheckSum: 70fe5
Characteristics: 22 perf
Debug Data Dirs: Type Size     VA Pointer
             CODEVIEW    20, 142c8,   136c8 RSDS - GUID: {D3FD3BA3-615D-437E-83B9-D339ED15DEE3}
               Age: 2, Pdb: srv.pdb
                CLSID     4, 142c4,   136c4 [Data not mapped]
     Image Type: MEMORY   - Image read successfully from loaded memory.
    Symbol Type: PDB      - Symbols loaded successfully from symbol server.
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb
    Load Report: public symbols , not source indexed
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb

Dial in your debugging skills with this book.

Brad Rutkowski — Fri, 25 Jan 2008 04:15:57 GMT

So this book has been getting mentioned around by a lot of reputable contacts internally so my teammate picked up a couple copies and I've been browsing it since. Browsing? Like all books in this category (Windows Internals, C++ Programming, etc) I don't read it cover to cover, if you do you should earn a merit badge.

I've noticed that when I run into a situation, like a memory leak or server hung and have exhausted all I can think of, I picked up this book and learned a few new tricks which came in handy (!address -summary). This happened a few times and then the book became the first place I checked when running into a situation I had not debugged very often or ever.

For example one "trick" they tell you about early on is when you have a usermode debug session piped to kernel (ntsd -d), but for whatever reason you cant use .breakin to get into the kernel from the user mode debug session. They mention using .sleep 1000 to sleep the user mode debugger and then you can use ctrl+c to break into kernel and snoop around, when you're done snooping around, g the remote and then sleep command will complete and you'll fall back into the user mode remote. (If this doesn't make sense well get this book then!).

It's tidbits of information like this and solid debugging skills that come from years on the job and can be found in this book. If you want to get into debugging, or have been been debugging for a while and want to learn new techniques, this is a great find. All in all if you debug either user mode or kernel mode, you're going to find some great information in this book.

The only other book I use frequently on the job is the Windows Internals book that most of you know and love (excluding the debugger.chm file).

I get no kick backs for my recommendation, I just like the book and I'm sure to fine more gems as I continue to run into issues.

Technorati Tags: Windows,Debugging

These are a few of my favorite things... (Part 4)

Brad Rutkowski — Thu, 22 Nov 2007 02:09:02 GMT

Just some more tricks/tools I use frequently...

Scale-to-Fit in Perfmon

If you deal a lot with perfmon you know you can have a ton of different counters in one line graph or in one bar graph and that the scale is usually 0-100 which really isn't applicable in some cases. Now you can just alt click the graph and select "Scale selected counter" which will then fit them all in one graph. As an aside, you can also just drag in perfmon collections (.html, .blg, .csv, or .tsv)into the MMC now and have the data displayed.

Handle.exe

Need to know what process/user is holding a file open on your server? Use Handle:

C:\>handle -u S:\Public\UserA\DCChkWeb\dcchk_default_new.htm

dcChk.exe pid: 7440 BRADDOM\userb S:\Public\UserA\DCChkWeb\dcchk_default_new.htm

Need to know what type of handles a particular process has open? Use Handle:

C:\Users\UserB\Desktop>handle.exe -p 620 -s

Handle type summary:
                  : 52
Desktop         : 1
Directory       : 2
Event           : 6229
File            : 3210
IoCompletion    : 17
Key             : 150
KeyedEvent      : 2
Mutant          : 10
Process         : 58
Process         : 94
Section         : 12
Semaphore       : 6169
Thread          : 630
Timer           : 8
Token           : 4927
TpWorkerFactory : 2
WindowStation   : 2
Total handles: 21575

Tlist.exe

Great tool to dump all the processes running on your system.

Two main arguments I use with Tlist:

"-v" to dump the verbose output which will show the arguments that were passed to the process:

2 32 5116 AcroRd32.exe Title: sw
Command Line: "C:\Program Files (x86)\Adobe\Acrobat 7.0\Reader\AcroRd32.exe" /o /eo /l

"-s" to dump what services run in each process. As you probably know a lot of services are just called with "svchost -netsvcs" so how do you knwo which one WINMGMT lives in? Use -s.

C:\localbin>tlist -s
   0 System Process
   4 System
460 smss.exe
548 csrss.exe
580 wininit.exe
632 services.exe
652 lsass.exe       Svcs: KeyIso,Netlogon,ProtectedStorage,SamSs
660 lsm.exe
808 svchost.exe     Svcs: DcomLaunch,PlugPlay
916 svchost.exe     Svcs: RpcSs
988 svchost.exe     Svcs: WinDefend
408 svchost.exe     Svcs: AudioSrv,Dhcp,Eventlog,lmhosts,p2pimsvc,wscsvc
512 svchost.exe     Svcs: AudioEndpointBuilder,CscService,EMDMgmt,Netman,PcaSvc,SysMain,TrkWks,UmRdpService,UxSms,WdiSystemHost,WPDBusEnum,wudfsvc
540 svchost.exe     Svcs: AeLookupSvc,BITS,CertPropSvc,gpsvc,hkmsvc,IKEEXT,iphlpsvc,LanmanServer,MMCSS,ProfSvc,RasMan,Schedule,seclogon,SENS,SessionEnv,ShellHWDetection,Themes,Winmgmt,wuauserv
796 audiodg.exe
<SNIP>

SPLInfo.exe

SplInfo is a command-line tool that collects information from the print spooler and displays it.

C:\Localbin>splinfo \\prn-machine

Number Remote Printers 490 on \\prn-machine

Windows Version 6.0 Build 6001 (Service Pack 1, v.275) FREE

Number of Processors 4 PROCESSOR_INTEL Level 6

Total Jobs Spooled 3,650

Total Bytes Printed 7,243,275,903

Total GDI Pages Printed 11,690

Average Bytes/Job 1,984,459

Average Pages/Job 3

Average Bytes/Page 619,612

Browse List Requested 0

Browse Printer Added 0

Queues with Jobs 20

# Queues with # Jobs:

1 114

1 10

1 6

1 5

1 4

Spooler Up Time 1 Day 21:46:46

Server Up Time 22 Days 05:48:32

Technorati tags: Vista, Longhorn, Windows 2008

Got a handle leak? Use !Htrace to help find the leaking stacks non-invasively.

Brad Rutkowski — Tue, 13 Nov 2007 21:10:57 GMT

So when your an app developer or someone in my position where you need to track down memory leaks one of the tools to use is Htrace once you've identified it's a handle leak.

I just wanted to put this post out there to show that I found you can use Htrace against a usermode process like LSASS below without being invasive! This was pretty critical in this scenario as the print server below was clustered and if we broke into LSASS via KD, the resources would have failed over to the passive node. Of course, I'm making no guarantees, but Htrace worked for me non-invasively below, your mileage may vary.

More about non-invasive debugging in a previous post here.

Before using Htrace you need to use application verifier to track handles for you for whatever process is leaking.

C:\>cdb -p 708 -pvr -y http://msdl.microsoft.com/download/symbols //Using PVR to be non-invasive for LSASS.

*** wait with pending attach
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
WARNING: Process 708 is not attached as a debuggee
         The process can be examined but debug events will not be received
...........................................................................
(2c4.2cc): Wake debugger - code 80000007 (first chance)
eax=00000000 ebx=00000000 ecx=025bf200 edx=00000000 esi=00000000 edi=000005a4
eip=77848254 esp=025bf64c ebp=025bf69c iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
77848254 c3              ret
0:000> !htrace -enable //Enables tracing of handles to start. by enabling you take a snapshot as well.
Handle tracing enabled.
Handle tracing information snapshot successfully taken.
0:000> !htrace -snapshot //Takes the second snapshot, at this point we have two snapshots to compare.
Handle tracing information snapshot successfully taken.
0:000> !htrace -diff // Now we tell Htrace to show us any handles left open between the first and second snapshot, all the closed handles are removed.
Handle tracing information snapshot successfully taken.
0x20d new stack traces since the previous snapshot.
Ignoring handles that were already closed...
Outstanding handles opened since the previous snapshot: //Now it lists all the open handles and the stacks that opened the handles, some will be legit but for my issue it was leaking about 100 minute so it was easy to find the stacks that were suspect. Now that I have the stacks, I can set breakpoints and look for where handles were allocated but not released.

--------------------------------------
Handle = 0x00022060 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x77846c2c: ntdll!ZwDuplicateToken+0x4c
0x74e6160c: LSASRV!LsapInitializeSessionToken+0x44
0x74e5e5b1: LSASRV!LsapSetSessionToken+0x4f
0x74e64352: LSASRV!LsapCreateTokenEx+0x28
0x74c86301: kerberos!KerbCreateTokenFromTicket+0x8d
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xff
0x74e639de: LSASRV!WLsaAcceptContext+0x18
0x74e930a0: LSASRV!NegHandleClientRequest+0x5e
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0xe4
0x74e639de: LSASRV!WLsaAcceptContext+0x8e
0x74e637bf: LSASRV!LpcAcceptContext+0x15
0x74e511de: LSASRV!DispatchAPI+0x80
0x74e510da: LSASRV!LpcHandler+0x2b
--------------------------------------
Handle = 0x00030ca4 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x778468cc: ntdll!ZwCreateSemaphore+0x4c
0x77824d77: ntdll!RtlInitializeResour+0xff
0x75287201: vfbasics+0x00007201
0x74e6146c: LSASRV!LsapCreateLsaLogonSess+0x46
0x74e61544: LSASRV!LsapCreateLogonSession+0xf8
0x74c861b5: kerberos!KerbCreateTokenFromTicket+0x0d
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xff
0x74e639de: LSASRV!WLsaAcceptContext+0x18
0x74e930a0: LSASRV!NegHandleClientRequest+0xeb
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0x3e
0x74e639de: LSASRV!WLsaAcceptContext+0x8e
0x74e637bf: LSASRV!LpcAcceptContext+0x57
0x74e511de: LSASRV!DispatchAPI+0x80
--------------------------------------
Handle = 0x0000d36c - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x778468cc: ntdll!ZwCreateSemaphore+0x4c
0x77824d4f: ntdll!RtlInitializeResource+0x4d
0x75287201: vfbasics+0x00007201
0x74e6146c: LSASRV!LsapCreateLsaLogonSession+0xf6
0x74e61544: LSASRV!LsapCreateLogonSession+0x28
0x74c861b5: kerberos!KerbCreateTokenFromTicket+0xad
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xf3
0x74e639de: LSASRV!WLsaAcceptContext+0x34
0x74e930a0: LSASRV!NegHandleClientRequest+0x43
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0x04
0x74e639de: LSASRV!WLsaAcceptContext+32
0x74e637bf: LSASRV!LpcAcceptContext+044
0x74e511de: LSASRV!DispatchAPI+0x3
--------------------------------------
Handle = 0x0000da98 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

<SNIP>

You can see all about using Htrace by watching this video on Channel 9: http://channel9.msdn.com/ShowPost.aspx?PostID=341851

Technorati tags: debugging, memory leak, handle

Not getting kernel memory dumps in Windows Vista or Windows 2008?

Brad Rutkowski — Tue, 16 Oct 2007 19:36:11 GMT

Backstory:

With the advent of Windows Vista there are changes made in how the operating system determines if it can take a kernel memory dump or not. Starting in Vista the amount of memory allocated for kernel mode could vary dynamically. If the pagefile is not big enough, switching to minidump at dump time can’t be done easily. So the dump stack initialization is happening at the time of boot where this check for the pagefile size is done.

What does this mean? It means if you don't have a pagefile as large as physical memory at boot, and your system is configured for a kernel dump, you'll end up getting a minidump. If you permit me to opine, this makes sense in the client space where a valid dump is more critical than a corrupted kernel dump, as the results usually would get uploaded to Microsoft via WERCON or another mechanism. If further triage is needed MSFT could contact you with the ability to setup a kernel capture.

In the Server world though, it's different. We have thousands of x64 systems with 16GBs of RAM and there is no way we could have a 16GB page file as the system either does not have the space (on C), or it does not make fiscal sense with regards to disk space. We have all our systems configured to take kernel dumps in case we crash the server via debugger/keyboard. We dogfood our beta operating systems, and a hung server is a normal site to see, and sometimes we can't break in via the debugger and a crash dump is our last and only resort. Crashing a box and ending up with a minidump does not suffice in our role.

The change in Vista SP1 RC0/Windows 2008 RC0 on:

Starting with the release of RC0, there is a new registry key that can be set which will tell the OS to ignore the page file check on boot up and you'll take your chances getting a valid kernel dump. We've tested this internally and all works as expected. So if you need kernel dumps on your large memory systems, this might be something to remember for your bag of tricks.

Key: HKLM\System\CurrentControlSet\Control\CrashControl

Value: IgnorePagefileSize

Type: DWORD

Data: 1

Technorati tags: Windows 2008, Vista, Debugging

Kernel stack not resident (Using .pagein)

Brad Rutkowski — Thu, 30 Aug 2007 02:36:10 GMT

You might find yourself debugging an issue and a thread you are interested in is paged out. Here's the steps to use to page in the stack for the kernel side and user side... Be careful when doing this on a live machine that you want to release after debugging as paging in certain section of memory can cause it to bugcheck...

2: kd> !thread fffffa8004415460
THREAD fffffa8004415460 Cid 087c.0acc Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa80044157f0 Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff88018c943f0
Impersonation token: fffff8801d302060 (Level Impersonation)
Owning Process            fffffa80046e5610       Image:         snmp.exe
Wait Start TickCount      367059906      Ticks: 15906005 (2:20:55:35.268) //Been waiting a while.
Context Switch Count      13819416
UserTime                  00:00:38.173
KernelTime                00:02:33.972
Win32 Start Address 0x000007fefa7724bc
Stack Init fffffa600440ddb0 Current fffffa600440d6e0
Base fffffa600440e000 Limit fffffa6004408000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Kernel stack not resident. // We can't see what the stack looks like as it been waiting so long its been paged out.

2: kd> .pagein fffffa600440d6e0 //Grab Current from above... This will get us the kernel side...
You need to continue execution (press 'g' <enter>) for the pagein to be brought in. When the debugger breaks in again, the page will be present.
2: kd> g
Break instruction exception - code 80000003 (first chance)
nt!DbgBreakPointWithStatus:
fffff800`0163e1d0 cc              int     3
1: kd> !thread fffffa8004415460
THREAD fffffa8004415460 Cid 087c.0acc Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa80044157f0 Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff88018c943f0
Impersonation token: fffff8801d302060 (Level Impersonation)
Owning Process            fffffa80046e5610       Image:         snmp.exe
Wait Start TickCount      367059906      Ticks: 15906070 (2:20:55:36.282)
Context Switch Count      13819416
UserTime                  00:00:38.173
KernelTime                00:02:33.972
Win32 Start Address 0x000007fefa7724bc
Stack Init fffffa600440ddb0 Current fffffa600440d6e0
Base fffffa600440e000 Limit fffffa6004408000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffffa60`0440d720 fffff800`01647abe : fffffa60`0440da88 fffff880`18c943f0 fffffa60`0440da88 fffff880`18c943f0 : nt!KiSwapContext+0x7f
fffffa60`0440d860 fffff800`016484c5 : 00000000`00303cb0 fffffa60`0440da88 00000000`00000009 00000000`00000001 : nt!KiSwapThread+0x12e
fffffa60`0440d8c0 fffff800`01681067 : 00000000`00000000 00000000`00000011 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x5f5
fffffa60`0440d940 fffff800`018be424 : fffffa60`0440da88 00000000`00303cb0 fffffa80`04415460 00000000`00000000 : nt!AlpcpSignalAndWait+0x97
fffffa60`0440d980 fffff800`018be868 : 00000000`00000000 00000000`00000000 00000000`00303cb0 00000000`00300318 : nt!AlpcpReceiveSynchronousReply+0x44
fffffa60`0440d9e0 fffff800`018a834f : fffffa80`04352e60 fffffa80`00020000 00000000`00303cb0 00000000`00300318 : nt!AlpcpProcessSynchronousRequest+0x251
fffffa60`0440db00 fffff800`016437b3 : fffffa80`04415460 fffffa60`0440dca0 00000000`00000280 fffff800`0189c654 : nt!NtAlpcSendWaitReceivePort+0x19f
fffffa60`0440dbb0 00000000`77af4dca : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffffa60`0440dc20)
00000000`016aebc8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77af4dca

1: kd> .pagein /p fffffa80046e5610 00000000`016aebc8 //We take the process ID of the thread and the usermode address at the bottom of the stack.
You need to continue execution (press 'g' <enter>) for the pagein to be brought in. When the debugger breaks in again, the page will be present.
1: kd> g
Break instruction exception - code 80000003 (first chance)
nt!DbgBreakPointWithStatus:
fffff800`0163e1d0 cc              int     3

1: kd> !thread fffffa8004415460 //Viola! Now we have the whole stack, you might need to do a .reload for symbols.
THREAD fffffa8004415460 Cid 087c.0acc Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa80044157f0 Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff88018c943f0
Impersonation token: fffff8801d302060 (Level Impersonation)
Owning Process            fffffa80046e5610       Image:         snmp.exe
Wait Start TickCount      367059906      Ticks: 15906135 (2:20:55:37.296)
Context Switch Count      13819416
UserTime                  00:00:38.173
KernelTime                00:02:33.972
Win32 Start Address 0x000007fefa7724bc
Stack Init fffffa600440ddb0 Current fffffa600440d6e0
Base fffffa600440e000 Limit fffffa6004408000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffffa60`0440d720 fffff800`01647abe : fffffa60`0440da88 fffff880`18c943f0 fffffa60`0440da88 fffff880`18c943f0 : nt!KiSwapContext+0x7f
fffffa60`0440d860 fffff800`016484c5 : 00000000`00303cb0 fffffa60`0440da88 00000000`00000009 00000000`00000001 : nt!KiSwapThread+0x12e
fffffa60`0440d8c0 fffff800`01681067 : 00000000`00000000 00000000`00000011 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x5f5
fffffa60`0440d940 fffff800`018be424 : fffffa60`0440da88 00000000`00303cb0 fffffa80`04415460 00000000`00000000 : nt!AlpcpSignalAndWait+0x97
fffffa60`0440d980 fffff800`018be868 : 00000000`00000000 00000000`00000000 00000000`00303cb0 00000000`00300318 : nt!AlpcpReceiveSynchronousReply+0x44
fffffa60`0440d9e0 fffff800`018a834f : fffffa80`04352e60 fffffa80`00020000 00000000`00303cb0 00000000`00300318 : nt!AlpcpProcessSynchronousRequest+0x251
fffffa60`0440db00 fffff800`016437b3 : fffffa80`04415460 fffffa60`0440dca0 00000000`00000280 fffff800`0189c654 : nt!NtAlpcSendWaitReceivePort+0x19f
fffffa60`0440dbb0 00000000`77af4dca : 000007fe`fea5c72b 00000000`00001000 00000000`016aee90 00000000`01460058 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffffa60`0440dc20)
00000000`016aebc8 000007fe`fea5c72b : 00000000`00001000 00000000`016aee90 00000000`01460058 00000000`0030ed80 : ntdll!NtAlpcSendWaitReceivePort+0xa
00000000`016aebd0 000007fe`fea6c592 : 00000000`00302b50 00000000`016aef30 000007fe`fe95c8b8 00000000`00001000 : RPCRT4!LRPC_CCALL::SendReceive+0xbb
00000000`016aec50 000007fe`fea6c5e2 : 00000000`016aed00 00000000`00000000 00000000`00000000 00000000`01460058 : RPCRT4!I_RpcSendReceive+0x42
00000000`016aec80 000007fe`feafad2c : 00000000`016aef30 00000000`00000000 00000000`00000000 00000000`0030ed80 : RPCRT4!NdrSendReceive+0x32
00000000`016aecb0 000007fe`feafaef0 : 00000000`00000000 000007fe`fe95d090 00000000`00000011 00000000`016aece0 : RPCRT4!NdrpClientCall3+0x11c
00000000`016aef00 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!NdrClientCall3+0x7c

1: kd>

How to catch shutdowns that don’t go to debugger or cause bugchecks

Brad Rutkowski — Sat, 25 Aug 2007 07:29:29 GMT

Sometimes a system just goes offline or reboots and we cannot catch it in the debugger or it is not a stop error (bugcheck) so that machine just reboots from underneath you. One technique you can use is to set a breakpoint on nt!NtSetSystemPowerState. This function is the last thing called during ANY type of reboot. So you will see the calling function by using this.

Most of the time when I use this its a system that is stuck in a reboot loop on boot and it keeps loading the OS and then reboots again. I think the last time I hit this it was because someone copied over an x64 binary onto an x86 system for NTDSAI.dll so LSASS was tanking.

Here's an example:

kd> bp nt!NtSetSystemPowerState //Set the BP

kd> g
Breakpoint 0 hit
nt!NtSetSystemPowerState:
809c7c84 68bc000000 push 0BCh
kd> k
ChildEBP RetAddr
f6667d44 809b297d nt!NtSetSystemPowerState
f6667d58 808234cb nt!NtShutdownSystem+0x32
f6667d58 7c8285ec nt!KiFastCallEntry+0xf8
0134ff74 7c827b9b ntdll!KiFastSystemCallRet
0134ffb8 77e64829 ntdll!ZwShutdownSystem+0xc
0134ffa0 0101f1ce kernel32!BaseThreadStart+0x34

kd> .reload
Connected to Windows Server 2003 3790 x86 compatible target, ptr64 FALSE
Loading Kernel Symbols
..........................................................................................
Loading User Symbols
..........................................................................
Loading unloaded module list
...................
kd> k
ChildEBP RetAddr
f6667d44 809b297d nt!NtSetSystemPowerState
f6667d58 808234cb nt!NtShutdownSystem+0x32
f6667d58 7c8285ec nt!KiFastCallEntry+0xf8
0134ff74 7c827b9b ntdll!KiFastSystemCallRet
0134ff78 0102ad63 ntdll!ZwShutdownSystem+0xc
0134ffa0 0101f1ce winlogon!ShutdownThread+0x18c
0134ffb8 77e64829 winlogon!LogoffThreadProc+0x36
0134ffec 00000000 kernel32!BaseThreadStart+0x34

kd> !pcr
KPCR for Processor 0 at ffdff000:
    Major 1 Minor 1
        NtTib.ExceptionList: f666764c
            NtTib.StackBase: 00000000
           NtTib.StackLimit: 00000000
         NtTib.SubSystemTib: 80042000
              NtTib.Version: 0002492d
          NtTib.UserPointer: 00000001
              NtTib.SelfTib: 7ffa5000

                    SelfPcr: ffdff000
                       Prcb: ffdff120
                       Irql: 00000000
                        IRR: 00000000
                        IDR: ffff24f0
              InterruptMode: 00000000
                        IDT: 8003f400
                        GDT: 8003f000
                        TSS: 80042000

              CurrentThread: 82aa03a8
                 NextThread: 00000000
                 IdleThread: 8089fd80

                  DpcQueue:
kd> !thread 82aa03a8 //Looks like this instance it was Winlogon that was shutting the system down.
THREAD 82aa03a8 Cid 0160.046c Teb: 7ffa5000 Win32Thread: e1032868 RUNNING on processor 0
Not impersonating
DeviceMap                 e1001358
Owning Process            82b36548       Image:         winlogon.exe
Wait Start TickCount      176590         Ticks: 0
Context Switch Count      175                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.290
Win32 Start Address winlogon!LogoffThreadProc (0x0101f198)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f6668000 Current f6667a84 Base f6668000 Limit f6663000 Call 0
Priority 15 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr Args to Child
f6667d44 809b297d 00000005 00000004 c0000004 nt!NtSetSystemPowerState (FPO: [Non-Fpo])
f6667d58 808234cb 00000001 0134ffa0 7c8285ec nt!NtShutdownSystem+0x32 (FPO: [Non-Fpo])
f6667d58 7c8285ec 00000001 0134ffa0 7c8285ec nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f6667d64)
0134ff74 7c827b9b 0102ad63 00000001 00002000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0134ff78 0102ad63 00000001 00002000 0007ab38 ntdll!ZwShutdownSystem+0xc (FPO: [1,0,0])
0134ffa0 0101f1ce 0134ffc0 00000000 00000000 winlogon!ShutdownThread+0x18c (FPO: [Non-Fpo])
0134ffb8 77e64829 00003907 00000000 00000000 winlogon!LogoffThreadProc+0x36 (FPO: [Non-Fpo])
0134ffec 00000000 0101f198 012ffa68 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

Did you know there is live noninvasive debugging?

Brad Rutkowski — Fri, 13 Apr 2007 09:24:34 GMT

Say you want to check something out in KD but you don't have a debugger attached, like !vm perhaps, or you want to attach to a user mode process like LSASS and dump the threads running and their current state? Well there's is solution, obviously everything doesn't work when its noninvasive, but you can still screw the server up so be careful. Also it works on XP and later, this isn't just a vista thing.

I find myself constantly using this technique to peer into processes when I don't want to break in and take down a service. Also quite helpful to use KD -KL to look at memory consumption quickly without getting a debugger hooked up.

Usermode I like -pvr which is noninvasive and nonsuspending:

C:\debuggers_public>cdb -pvr -p 3976 -y SRV**http://msdl.microsoft.com/download/symbols

*** wait with pending attach
Symbol search path is: SRV**http://msdl.microsoft.com/download/symbols
Executable search path is:
WARNING: Process 3976 is not attached as a debuggee
The process can be examined but debug events will not be received
...............................................................
(f88.ce8): Wake debugger - code 80000007 (first chance)
eax=0000003c ebx=00000002 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=77c2aec5 esp=0013f76c ebp=0013f804 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!ZwWaitForMultipleObjects+0x15:
77c2aec5 c21400 ret 14h

0:000> vertarget
Windows Vista Version 6000 UP Free x86 compatible
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.0.6000.16386 (vista_rtm.061101-2205)
Debug session time: Thu Apr 12 23:10:39.539 2007 (GMT-7)
System Uptime: 1 days 4:36:26.108
Process Uptime: 0 days 3:48:12.445
Kernel time: 0 days 0:00:00.234
User time: 0 days 0:00:00.203

For kernel debugging well we use kd -kl:

C:\Debuggers>kd -kl -y SRV**http://msdl.microsoft.com/download/symbols

***** WARNING: Your debugger is probably out-of-date.

Connected to Windows Server 2003 3790 x86 compatible target, ptr64 FALSE
Symbol search path is: SRV**http://msdl.microsoft.com/download/symbols
Executable search path is:
*******************************************************************************
WARNING: Local kernel debugging requires booting with kernel
debugging support (/debug or bcdedit -debug on) to work optimally.
*******************************************************************************
Windows Server 2003 Kernel Version 3790 (Service Pack 1) MP (2 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_sp1_gdr.070304-2232
Kernel base = 0x80800000 PsLoadedModuleList = 0x808af988
Debug session time: Thu Apr 12 23:14:37.845 2007 (GMT-7)
System Uptime: 0 days 11:28:10.734
lkd> !vm

*** Virtual Memory Usage ***
Physical Memory: 262017 ( 1048068 Kb)
Page File: \??\C:\pagefile.sys
Current: 1572864 Kb Free Space: 1484016 Kb
Minimum: 1572864 Kb Maximum: 1572864 Kb
Cannot read pte range @ 808af740
Available Pages: 105545 ( 422180 Kb)
ResAvail Pages: 195302 ( 781208 Kb)
Locked IO Pages: 135 ( 540 Kb)
Free System PTEs: 43267 ( 173068 Kb)
Free NP PTEs: 32766 ( 131064 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 179 ( 716 Kb)
Modified PF Pages: 162 ( 648 Kb)
NonPagedPool Usage: 4459 ( 17836 Kb)
NonPagedPool Max: 51711 ( 206844 Kb)
PagedPool 0 Usage: 7720 ( 30880 Kb)
PagedPool 1 Usage: 1096 ( 4384 Kb)
PagedPool 2 Usage: 1104 ( 4416 Kb)
PagedPool 3 Usage: 1080 ( 4320 Kb)
PagedPool 4 Usage: 1089 ( 4356 Kb)
PagedPool Usage: 12089 ( 48356 Kb)
PagedPool Maximum: 70656 ( 282624 Kb)
Shared Commit: 4266 ( 17064 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 6783 ( 27132 Kb)
PagedPool Commit: 12095 ( 48380 Kb)
Driver Commit: 1459 ( 5836 Kb)
Committed pages: 140881 ( 563524 Kb)
Commit limit: 633232 ( 2532928 Kb)

For more information look up noninvasive debugging in the debugger chm file.

-Brad

Technorati tags: windows, debugging, vista, windows 2003, XP

Debugging Terminal Service not listening. (Isolating an instance of SVCHOST)

Brad Rutkowski — Fri, 06 Apr 2007 21:37:16 GMT

Ran into another issue today where I needed to set and IFEO for the particular instance of SVCHOST.exe running terminal service. Here is the easiest way of doing so:

1) Make a copy of SVCHOST.exe on the server, name it Mysvchost.exe and leave it in %Systemroot%\system32\

2) Open regedit and go to HKLM\System\CurrentControlSet\Services\TermService (this could be whatever service you're trying to isolate)

3) Edit "ImagePath", "%SystemRoot%\system32\svchost.exe -k termsvcs" and change svchost.exe to "Mysvchost.exe"

4) Then I set the IFEO for Mysvchost.exe with NTSD -d which send the output tot he kernel debugger.

5) Reboot

6) In Business

Simple and effective...

Let me drive! Using remote.exe to connect to a client.

Brad Rutkowski — Wed, 04 Apr 2007 06:48:03 GMT

Some might call me a control freak, but when I have the opportunity to investigate a machine myself or provide instructions via the phone/mail to the end-user you know what I'm going to choose. One of the easier ways that I find is to use remote.exe which is part of the debugging package. I'm just going to show you two examples of using remote.exe:

How to share out a command prompt. If Vista have the user open an elevated command prompt and then type this:

C:\debuggers_public>remote /S "CMD" USERNAME
**************************************
*********** REMOTE ************
*********** SERVER ************
**************************************
To Connect: Remote /C TRENCHTOWN "USERNAME" //Connect using this.

Now you can connect to this command prompt via the command above. You are officially on the remote machine at their command prompt and can continue your investigation using any tools that can be accessed at the cmd prompt.

You might also find an instance when you have a dump that you are investigating and you'd like to share it with others:

C:\debuggers_public>remote /S "cdb -z iexplore.dmp -y SRV**http://msdl.microsoft.com/download/symbols" IE_DUMP
**************************************
*********** REMOTE ************
*********** SERVER ************
**************************************
To Connect: Remote /C TRENCHTOWN "IE_DUMP" //You would connect to this remote via this command

Loading Dump File [C:\debuggers_public\iexplore.dmp]
User Mini Dump File with Full Memory: Only application data is available

Windows Vista Version 6000 UP Free x64
Product: WinNt, suite: SingleUserTS
Debug session time: Tue Apr 3 20:30:04.000 2007 (GMT-7)
System Uptime: 5 days 1:31:39.135
Process Uptime: 0 days 0:00:36.000
Symbol search path is: SRV**http://msdl.microsoft.com/download/symbols

<snip>

One other thing to note, to exit the remote without killing the remote itself use @q. Type the @ symbol in the remote to see other syntax.

Hope that helps.

Technorati tags: debugging, windows

The case of sidebar.exe not starting. Oh Snap!

Brad Rutkowski — Fri, 30 Mar 2007 01:20:00 GMT

Ran into a case today where each time we tried to start sidebar.exe it would fail silently. No crash to investigate, no error, so where to next?

First I set an IFEO for sidebar.exe to launch windbg.exe when started, by doing this it stops at the initial breakpoint.

Secondly I enabled loader snaps to show me more information on module loads: c:\debuggers\gflags.exe -i sidebar.exe +sls.

The show loader snaps flag captures detailed information about the loading and unloading of executable images and their supporting library modules and displays the data in the debugger.

After doing this and g'ing the debugger we could see that right before the process terminated it failed to load one particular dll:

LDR: LdrLoadDll, loading C:\Program Files\AVOne\AVOne 3GP Video Converter\atl.dll from C:\Program Files\AVOne\AVOne 3GP Video Converter;C:\Windows\system32
LDR: LdrpSearchPath - Looking for C:\Program Files\AVOne\AVOne 3GP Video Converter\atl.dll in C:\Program Files\AVOne\AVOne 3GP Video Converter;C:\Windows\system32
LDR: LdrpSearchPath - Unable to locate C:\Program Files\AVOne\AVOne 3GP Video Converter\atl.dll in C:\Program Files\AVOne\AVOne 3GP Video Converter;C:\Windows\system32: 0xc0000135
LDR: LdrpCheckForLoadedDll - Unable To Locate C:\Program Files\AVOne\AVOne 3GP Video Converter\atl.dll: 0xc0000135

Looking up the error code with err.exe:

C:\Debuggers\x86_ver>err 0xc0000135
# for hex 0xc0000135 / decimal -1073741515
STATUS_DLL_NOT_FOUND
# {Unable To Locate Component}
# This application has failed to start because %hs was not
# found. Re-installing the application may fix this problem.
# 1 matches found for "0xc0000135"

Once I had the user install the missing application (AVOne 3GP Video Converter) everything worked as expected.

Technorati tags: debugging, windows, windbg, debug

My computer is hard hung, now what can I do?

Brad Rutkowski — Mon, 23 Oct 2006 21:41:40 GMT

You may run into instances where a machine becomes "hard hung", I usually can tell when a machine is in this state because the "num lock" and "caps lock" key don't work. So what can you do in these instances? Well Windows has the ability to create a manual crash dump. Manual crash dumps will always have the bugcheck code of 0xE2. If you check out this article, you'll find all the info about this technique but I'm just going to give you the meat of it.

1.
Start Registry Editor.

2.
Locate the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters

3.
On the Edit menu, click Add Value, and then add the following registry entry:

Name: CrashOnCtrlScroll
Data Type: REG_DWORD
Value: 1

Now when you hit CTRL+SCROLL LOCK+SCROLL LOCK, it will cause your machine to bugcheck, if this does not make your machine bugcheck, then you either have a typo in the registry key, or more than likely this is a hardware issue and you should check your driver/BIOS versions and make sure everything is up to date.

We see a lot of manual crash dumps internally here at MSFT and most of the time its due to some driver that has a bug in it and not Windows OS code. At this point you have a dmp file that you can either send to MSFT for analysis, or dig into yourself. To find out more about analyzing a hung system you can use !analyze -hang, !vm, !locks, etc. Check out the debugger chm file for other techniques. If you don't know how to view dmp files check out my first post.

Technorati tags: BSOD, DMP, Bugcheck, Bugcode, Windows, Active Directory

IceRocket tags: BSOD, DMP, Bugcheck, Bugcode, Windows