Mfartura's blog

How to manually translate virtual addresses into physical ones

2008-11-21T02:03:00Z

In previous posts we talked about virtual address space and how virtual memory is managed. I've never posted anything about virtual address translation though, and for the ones interested on the details behing this operation I recommend reading the chapter 7 - Memory Management (specifically the Address Translation section) of the Windows Internals 4th Edition book. So I won't go too deep on the details of how the system does the translation itself, my goal here is just how to show you how easily...(read more)

A new phase...

2008-11-21T01:44:00Z

Even though this has been a strictly technical channel (I've never posted about anything else other than technical stuff) I know a lot of friends read it and so I decided to open an exception on this post and use it to publish something about me that also has a direct effect on how often and what type of content I will be able to keep publishing here.... Effective this week I started a new phase on my career at Microsoft assuming a non-technical role. I will be working as a people manager for the...(read more)

Kernel dump analysis - Bugcheck 0xA (IRQL_NOT_LESS_OR_EQUAL)

2008-06-26T00:28:00Z

Yet another kernel memory dump to be analyzed - The bugcheck this time is the 0xA - IRQL_NOT_LESS_OR_EQUAL. To better understand what this message means we would need a little background on Windows Internals but basically when executing anything at a interrupt request level (IRQL) = 2 or higher (in normal circumstances instructions get executed at IRQL = 0) we can't page fault. On the situation of this specific dump, we would crash even on IRQL = 0 since we're trying to write to the memory address...(read more)

Kernel dump analysis - Bugcheck 1E (KMODE_EXCEPTION_NOT_HANDLED)

2008-06-18T02:39:00Z

It’s been a long time since my last post, but for some reason lately I’ve been receiving so many nice feedbacks about the blog and the other posts that I feel really motivated again to post a new article. See how important your feedback is for me? J Ok, normally I post about user mode debugging, but last week I was working on developing some content and I got this kernel memory dump I decided to take the risk of analyzing it, even considering Kernel debugging is not my specialty. Normally the first...(read more)

How to attach a debugger from the creation time of the Worker Process (w3wp.exe)

2007-08-09T23:59:00Z

Normally the answer for this would be as simple as use the file image execution options (through direct editing the registry or using gflags.exe from Debugging Tools For Windows ) and set the “debugger” option to you preferred debugger tool and this would do the job. At least this is the way we do for troubleshooting services startup in general. There is one detail though which is the fact the services are likely loaded in a different winstation (they would have not interactivity with the desktop)...(read more)

Real case: Application Pool’s worker process (w3wp.exe) crashing during recycling

2007-08-09T23:34:00Z

I was teaching a workshop in London last week and one of the students brought a very interesting issue he had been working on for some time and based on the dump analysis he was not able to point to anything different from our own code (Microsoft’s modules) as the ones causing the crash. He had done some research and found out that some crashes when recycling the app pool have been reported around the internet due to a PHP ISAPI problem. He had such ISAPI loaded in the crashing app pool so he suspected...(read more)

Tip: How many users are hitting my web site?

2007-07-31T20:00:00Z

This is a question we hear very often from our customers, in forums or through distribution lists. The reason we keep hearing the same question over the time is very simple: There isn’t a definitive answer for that – At least not for IIS since it’s not exactly driven by the number of users hitting it. It would make more sense to monitor the number of connections and requests being made through these than simply working around a number of users. Why I think this way? This is a subject for another...(read more)

Logparser fails when using the ChartType option

2007-07-30T00:39:00Z

I'm posting this as I couldn't find this solution documented anywhere (including within MSFT through the discussions around logparser). Problem: After I installed the Office 2007, my logparser queries which use the parameter "ChartType" no longer work returning the following error: Error: invalid parameter "chartType" The solution? Logparser depends on the Office Web Components to create the charts. Just intall them (I just reinstalled the OWC for Office 2003 ) and everything will work again....(read more)

How to Extract SQL info from a thread stack

2007-07-13T22:40:00Z

In the post about troubleshooting Asp.Net poor performance I showed you how to identify one possible cause for high response times when processing Asp.Net pages – the cause we explored on that post was a database server taking too long to respond. We were able to verify all the information about the SQL side of the scenario through analyzing the Asp.Net thread stack and using the SOS.dll specific commands. So the question now is: What about when I’m processing a Classic Active Server Page (ASP) page instead of a Asp.Net one?

Well in this case, as we already know, the SOS.dll won’t help, so let’s see how to do it manually.

First step is to identify the thread and look at it call stack as below:

0:016> kb10

ChildEBP RetAddr Args to Child

00f2e1dc 7c822124 77e6baa8 000006b8 00000000 ntdll!KiFastSystemCallRet

00f2e1e0 77e6baa8 000006b8 00000000 00f2e224 ntdll!NtWaitForSingleObject+0xc

00f2e250 77e6ba12 000006b8 00004e20 00000000 kernel32!WaitForSingleObjectEx+0xac

00f2e264 74cd2e3f 000006b8 00004e20 00004e20 kernel32!WaitForSingleObject+0x12

00f2e290 6d56db96 00c1d2f8 01ab4270 00000009 DBmsLPCn!ConnectionRead+0xaf

00f2e2b0 6d56871b 01ab6290 01ab4270 00000009 DBNETLIB!WrapperRead+0x2c

00f2e308 4e252d91 01ab6290 01ab4270 01ab4270 DBNETLIB!ConnectionRead+0x519

00f2e33c 4e252df0 01ab6290 01ab4270 00000009 SQLOLEDB!CDataSource::ConnectionRead+0x35

00f2e388 4e2522b6 01ab403e 00000001 00000000 SQLOLEDB!CDBConnection::GetBytes+0x269

00f2e3d4 4e259f63 015c4e08 00000088 0000001e SQLOLEDB!CDBConnection::ProcessTDSStream+0x157

00f2e490 4e25a030 015c5a00 00000014 015c4970 SQLOLEDB!CStmt::ExecDirect+0x786

00f2e4a8 4e2596d1 015c5a00 00000014 00000000 SQLOLEDB!CStmt::SQLExecDirect+0x28

00f2e4d8 4e259b1c 00000000 4e2590b4 00000014 SQLOLEDB!CCommand::ExecuteHelper+0x157

00f2e55c 4e2595ea 015c5a68 00000000 4bbe9658 SQLOLEDB!CCommand::Execute+0x76b

00f2e594 4bbe952e 015c4d60 00000000 4bbe9658 SQLOLEDB!CImpICommandText::Execute+0xdd

00f2e5d4 4bc0c9db 015c12b0 015c4a04 015c1638 msado15!CConnection::Execute+0x9d

00f2e7a4 4bbe9388 015c5660 00000003 015c48a8 msado15!_ExecuteAsync+0x19f

00f2e7b8 4bbe90e0 015c5660 00f2e988 00000000 msado15!ExecuteAsync+0x23

00f2e8a0 4bbe9303 00000003 015c1a00 00000000 msado15!CQuery::Execute+0xa5e

00f2e90c 4bbe6128 015c48a8 00000003 015c1a00 msado15!CCommand::_Execute+0x153

00f2e9a0 4bbe6400 ffffffff 015c1a00 00f2eb34 msado15!CRecordset::_Open+0x30f

00f2eb18 11004397 005c1638 00000008 00000000 msado15!CRecordset::Open+0x5dc

So, that’s a classic OLEDB thread stack. To obtain the SQL command being send to the SQL server identify the function call SQLOLEDB!CStmt::SQLExecDirect above. Grab the first parameter and run the following command:

0:016> du 015c5a00

015c5a00 "SELECT * FROM TITLES"

Of course you still want to know which SQL server this thread holds a connection with. So how about checking the connection string? Just identify the the function call msado15!CConnection::Execute, grab the first parameter and execute the following:

0:016> du poi(015c12b0+64)

015c1aa8 "Provider=SQLOLEDB.1;Integrated S"

015c1ae8 "ecurity=SSPI;Persist Security In"

015c1b28 "fo=False;Initial Catalog=pubs;Da"

015c1b68 "ta Source=(local)"

So now you can call the DBA in charge of the server “local” and ask him to collaborate with you on this low performance problem your ASP app has been unfairly blamed about J

Have fun!

How to identify the process and thread being called in a COM call from a thread stack

2007-07-13T22:23:00Z

I’ve just published a post on how to manually identify which classic ASP page is being processed by a specific thread. Following the same idea you can also identify COM calls being made by a thread. As always, first identify the thread doing the COM call you want investigate and then look at the thread stack as below:

<0:032>

0:032> kb100

ChildEBP RetAddr Args to Child

0235e93c 7c822114 77e6711b 00000002 0235e98c ntdll!KiFastSystemCallRet

0235e940 77e6711b 00000002 0235e98c 00000001 ntdll!NtWaitForMultipleObjects+0xc

0235e9e8 7739cd08 00000002 0235ea10 00000000 kernel32!WaitForMultipleObjectsEx+0x11a

0235ea44 77697483 00000001 0235ea8c 000003e8 USER32!RealMsgWaitForMultipleObjectsEx+0x141

0235ea6c 776974f2 0235ea8c 000003e8 0235ea9c ole32!CCliModalLoop::BlockFn+0x8

0235ea94 7778866b ffffffff 0235eb94 000e513c ole32!ModalLoop+0x5b

0235eab0 77788011 00000000 00000000 00000000 ole32!ThreadSendReceive+0xa0

0235eacc 77787ed7 0235eb94 0011825c 0235ebf0 le32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x112

0235ebac 776975b8 0011825c 0235ecc0 0235ecb0 ole32!CRpcChannelBuffer::SendReceive2+0xc1

0235ebc8 7769756a 0235ecc0 0235ecb0 0011825c ole32!CCliModalLoop::SendReceive+0x1e

0235ec34 776c4eee 0011825c 0235ecc0 0235ecb0 ole32!CAptRpcChnl::SendReceive+0x6f

0235ec88 77ce127e 00000001 0235ecc0 0235ecb0 ole32!CCtxComChnl::SendReceive+0x91

0235eca4 77ce13ca 0011750c 0235ecec 00000000 RPCRT4!NdrProxySendReceive+0x43

0235f08c 77d0c947 77d12028 77d150c2 0235f0ac RPCRT4!NdrClientCall2+0x206

0235f0a4 77d0c911 0011750c 60030001 73464c18 OLEAUT32!IDispatch_RemoteInvoke_Proxy+0x1c

0235f364 73469f71 0011750c 60030001 73464c18 OLEAUT32!IDispatch_Invoke_Proxy+0xb6

0235f3b8 73468f37 01ef0100 0011750c 60030001 vbscript!CatchIDispatchInvoke+0x46

0235f3f8 73468ea6 01ef0288 0011750c 60030001 vbscript!IDispatchInvoke2+0xaf

0235f434 73469000 01ef0288 0011750c 60030001 vbscript!IDispatchInvoke+0x59

0235f548 73467bb6 01ef0288 00000000 60030001 vbscript!InvokeDispatch+0x13a

0235f56c 73467cad 01ef0288 0011750c 60030001 vbscript!InvokeByName+0x42

0235f848 73464940 00000000 00000000 01ef0288 vbscript!CScriptRuntime::Run+0x1331

0235f940 73464cd2 00000000 00000000 00000000 vbscript!CScriptEntryPoint::Call+0x5c

0235f998 73465522 01ef0768 00000000 00000000 vbscript!CSession::Execute+0xb4

0235f9e8 7346189b 00000000 00000000 709e19b4 vbscript!COleScript::ExecutePendingScripts+0x13e

0235fa04 709e2f5a 01612040 01612040 01112700 vbscript!COleScript::SetScriptState+0x150

0235fa30 709e2f1a 00000000 709e19b4 0235fb38 asp!CActiveScriptEngine::TryCall+0x19

0235fa6c 709e2e50 00000000 647246fe 014123e8 asp!CActiveScriptEngine::Call+0x31

0235fa88 709e2d54 0235fb0c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x5b

0235fadc 709e2c7f 01911e90 00000000 0235fb68 asp!ExecuteRequest+0x17e

0235fb44 709e2a4d 01911e90 014123e8 0235fb68 asp!Execute+0x249

0235fb98 709e271a 00000000 00000000 00114c28 asp!CHitObj::ViperAsyncCallback+0x3e8

0235fbb4 75bd748e 01493048 00097be8 0235fd74 asp!CViperAsyncRequest::OnCall+0x92

0235fbd0 7770f153 00114c28 000a4178 00000000 COMSVCS!CSTAActivityWork::STAActivityWorkHelper+0x32

0235fc1c 7770fba0 00000000 000a4178 75bd745c ole32!EnterForCallback+0xc4

0235fd7c 777100aa 0235fc54 75bd745c 00114c28 ole32!SwitchForCallback+0x1a3

0235fda8 7769408c 000a4178 75bd745c 00114c28 ole32!PerformCallback+0x54

0235fe40 77712865 00097be8 75bd745c 00114c28 ole32!CObjectContext::InternalContextCallback+0x159

0235fe60 75bd7831 00097be8 75bd745c 00114c28 ole32!CObjectContext::DoCallback+0x1c

0235fecc 75bd7b95 00114e18 00114df8 001147c4 COMSVCS!CSTAActivityWork::DoWork+0x12d

0235fee4 75bd852e 00114c28 00000001 00114df8 COMSVCS!CSTAThread::DoWork+0x18

0235ff04 75bd897a 00000000 01eefcb0 00039608 COMSVCS!CSTAThread::ProcessQueueWork+0x37

0235ff84 77bcb530 00114df8 00000000 00000000 COMSVCS!CSTAThread::WorkerLoop+0x17c

0235ffb8 77e66063 00039608 00000000 00000000 msvcrt!_endthreadex+0xa3

0235ffec 00000000 77bcb4bc 00039608 00000000 kernel32!BaseThreadStart+0x34

Now identify the function call ole32!CRpcChannelBuffer::SendReceive2 above. Grab the first parameter and run the following command:

0:032> dd poi(0011825c +18)+8 l2

000de8d0 00000d8c 00000ddc

The output above represents the process ID (in this case the hex number 0xd8c) of the server process for this COM call and the thread ID (in this case the hex number 0xddc).

Notice that the output values are in hex format and your PID (the one you identify by using task manager, tlist, tasklist etc) info will probably be in decimal format so you still need to make the convertion:

In this case the PID of the server process being called in this thread would be 3462 = 00000d86.

How to identify which ASP page is being processed on a specific thread

2007-07-13T22:00:00Z

I’ve been teaching one of our workshops offerings (Workshop Plus: IIS6 Critical Problem Management) and when we go through some debugging samples, one thing I like to demo is how to find which Asp.Net pages are being processed in which threads. Well, the SOS.dll extension supports the “!aspxpages” command that will make your life really easier in a sense that all you need to do is run the command and it will return a list of the threads and their respective IDs as well as which Asp.Net page is being processed by such threads (aspx, asmx).

And then the students try to use the same command to identify classic Active Server Pages (.asp). This will not work since the commands implemented by the SOS.dll are specifically for managed code (.Net) and ASP pages will be processed by threads running native code. Well, the easiest way to accomplish that would be just feedback the memory dump to the Debug Diagnostics Tool (DebugDiag) and run the hang/crash analysis scripts against it. But how about when you don’t have Debug Diagnostics Tool (DebugDiag) or any other extension available for some reason to do that for you? Can you still do that manually through the Windbg from Debugging Tools For Windows? The answer is YES! And you don’t even need symbols for that ;-).

Here is what you can do to find out what ASP page is being processed in a specific thread:

First verify the thread call stack:

0:032> kb

ChildEBP RetAddr

0235fa88 709e2d54 0235fb0c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x5b

0235fadc 709e2c7f 01911e90 00000000 0235fb68 asp!ExecuteRequest+0x17e

0235fb44 709e2a4d 01911e90 014123e8 0235fb68 asp!Execute+0x249