Debugging In Progress...

This blog is intended to provide information on debug issues encountered in Microsoft Support, such as STOP errors or hanging servers.

[RESOLVED] Win2008R2 RTM/SP1: STOP 0x3B in rdbss!RxFsdCommonDispatch+ad7

[RESOLVED] Win2008R2 RTM/SP1: STOP 0x3B in rdbss!RxFsdCommonDispatch+ad7

  • Comments 4
  • Likes

Status: Resolved

Update 120531: Yes! The Private is approved. We are releasing the hotfix in the July Release Cycle, under KB2719704.

Update 120510: We are now testing a private at several customers... fingers crossed this fixes this long-runner! :D

Update 120417: Still ongoing, we are now awaiting verifier-enabled dumps for the crash. If you have this issue too, please enable "verifier /volatile /flags 0x9 /adddriver rdpdr.sys rdbss.sys" and send me the resulting dump.

Update 120217: Yesterday I received an instrumented rdpdr.sys to implement on affected machines. To be able to obtain this and help us find the root cause of this, please create a support case and contact me.

Update 120120: Recently I've filed an RFC on this issue, to get Engineering assistance on resolving this. I expect some update later next week.

Update 111115: Two more customers reported this. Since we know that when we enable full Redirector (Rdr) tracing the problem goes away, I figured out some "light-weight" Rdr tracing. Also, we enabled RdpDr tracing since we suspect a race condition between Rdr and RdpDr. If you run into this problem please enable the tracing and contact me:

tracelog.exe -start RdrMup -guid #fc4b0d39-e8be-4a83-a32f-c0c7c4f61ee4 -b 256 -min 256 -max 256 -f %systemroot%\system32\LogFiles\mup.etl -flag 0x0f00000f -level 5
logman create trace RdpDr -ow -o %SYSTEMROOT%\system32\LogFiles\rdpdr.etl -p {73BFB78F-12B5-4738-A66C-A77BCD55FA12} 0x1 0xf -nb 256 256 -bs 256 -mode 0x2 -f bincirc -max 100 -ets

Note: Tracelog comes with the WDK, LogMan is included in the OS.

Update 111109: The first customer hitting this came back: they had the problem again... awaiting further information now.

Update 111108: This issue went silent for some time... Two customers that encountered this stopped having the problem after implementing KBs 2579362 & 2521220. Today, a customer running SP1 contacted me, also encountering this issue.

Update 110808: Still no crash with the instrumentation and tracing running... customer notified me that their environment is still not running at full load because of the holiday season. Now we wait...

Recently, this case came to my attention, although it's been going on for quite some time. Perhaps there are more people out there who have servers experiencing this issue... :) If so, please let me know!

The stack of this problem shows:

2: kd> knL
 # Child-SP          RetAddr           Call Site
00 fffff880`03b30398 fffff800`018ccca9 nt!KeBugCheckEx
01 fffff880`03b303a0 fffff800`018cc5fc nt!KiBugCheckDispatch+0x69
02 fffff880`03b304e0 fffff800`018f340d nt!KiSystemServiceHandler+0x7c
03 fffff880`03b30520 fffff800`018faa90 nt!RtlpExecuteHandlerForException+0xd
04 fffff880`03b30550 fffff800`019079ef nt!RtlDispatchException+0x410
05 fffff880`03b30c30 fffff800`018ccd82 nt!KiDispatchException+0x16f
06 fffff880`03b312c0 fffff800`018cabb4 nt!KiExceptionDispatch+0xc2
07 fffff880`03b314a0 fffff880`02e172e0 nt!KiBreakpointTrap+0xf4
08 fffff880`03b31630 fffff880`02e35b74 rdbss!RxFsdCommonDispatch+0xad8
09 fffff880`03b31720 fffff880`04b8acd7 rdbss!RxFsdDispatch+0x224
0a fffff880`03b31790 fffff880`018b8271 rdpdr!DrPeekDispatch+0x31f
0b fffff880`03b317e0 fffff880`018b6138 mup!MupiCallUncProvider+0x161
0c fffff880`03b31850 fffff880`018b6b0d mup!MupStateMachine+0x128
0d fffff880`03b318a0 fffff880`013006af mup!MupFsdIrpPassThrough+0x12d
0e fffff880`03b318f0 fffff880`018f794d fltmgr!FltpDispatch+0x9f
0f fffff880`03b31950 fffff880`013006af mfehidk!DEVICEDISPATCH::DispatchPassThrough+0x105
10 fffff880`03b319b0 fffff800`01be8707 fltmgr!FltpDispatch+0x9f
11 fffff880`03b31a10 fffff800`01be8f66 nt!IopXxxControlFile+0x607
12 fffff880`03b31b40 fffff800`018cc993 nt!NtDeviceIoControlFile+0x56
13 fffff880`03b31bb0 00000000`7721f72a nt!KiSystemServiceCopyEnd+0x13

We hit a Debug Breakpoint (0x80000003) because of an APC issue. Currently, we are working on instrumentation that will tell us where the APC problem is located. As soon as we know that, we can start thinking of a fix.

To be continued...

Comments
  • Same exact stack here:

    Child-SP          RetAddr           Call Site

    fffff880`326df458 fffff800`05ac7ca9 nt!KeBugCheckEx

    fffff880`326df460 fffff800`05ac75fc nt!KiBugCheckDispatch+0x69

    fffff880`326df5a0 fffff800`05aee40d nt!KiSystemServiceHandler+0x7c

    fffff880`326df5e0 fffff800`05af5a90 nt!RtlpExecuteHandlerForException+0xd

    fffff880`326df610 fffff800`05b029ef nt!RtlDispatchException+0x410

    fffff880`326dfcf0 fffff800`05ac7d82 nt!KiDispatchException+0x16f

    fffff880`326e0380 fffff800`05ac5bb4 nt!KiExceptionDispatch+0xc2

    fffff880`326e0560 fffff880`03797d08 nt!KiBreakpointTrap+0xf4

    fffff880`326e06f0 fffff880`037b5614 rdbss!RxFsdCommonDispatch+0xad8

    fffff880`326e07e0 fffff880`0618ecd7 rdbss!RxFsdDispatch+0x224

    fffff880`326e0850 fffff880`017d7271 rdpdr!DrPeekDispatch+0x31f

    fffff880`326e08a0 fffff880`017d5138 mup!MupiCallUncProvider+0x161

    fffff880`326e0910 fffff880`017d5b0d mup!MupStateMachine+0x128

    fffff880`326e0960 fffff880`013a56af mup!MupFsdIrpPassThrough+0x12d

    fffff880`326e09b0 fffff800`05de1547 fltmgr!FltpDispatch+0x9f

    fffff880`326e0a10 fffff800`05de1da6 nt!IopXxxControlFile+0x607

    fffff880`326e0b40 fffff800`05ac7993 nt!NtDeviceIoControlFile+0x56

    fffff880`326e0bb0 00000000`7717f72a nt!KiSystemServiceCopyEnd+0x13

    This seems to happen for us when Win2k8R2 terminals servers are under a load of around 200 users.  The server sometimes bluescreens, sometimes the UmRDPService stops responding.  

  • Same here but only about 10 users

    Is this an IO issue?

    We are also getting large numbers of 7011s in the system log.

  • @George: sorry for the delay! :$

    This is not an IO-issue afaik, it happens with redirecting folders. However, it does appear that the issue is load-related.

    Can you please enable "verifier /flags 0x9 /driver rdpdr.sys rdbss.sys" and create a case with us?

  • I am getting the same issue and I am suspecting that this KB can fix it support.microsoft.com/.../2525246.

    Dumps:

    RetAddr           : Args to Child                                                           : Call Site

    fffff800`0207eb29 : 00000000`0000003b 00000000`80000003 fffff880`02e17d07 fffff880`076c4ca0 : nt!KeBugCheckEx

    fffff800`0207e47c : fffff880`076c5438 fffff880`076c4ca0 00000000`00000000 fffff800`020adb50 : nt!KiBugCheckDispatch+0x69

    fffff800`020a52dd : fffff800`022a2bd0 fffff800`021c8a70 fffff800`0200f000 fffff880`076c5438 : nt!KiSystemServiceHandler+0x7c

    fffff800`020ac950 : fffff800`021cf198 fffff880`076c45d8 fffff880`076c5438 fffff800`0200f000 : nt!RtlpExecuteHandlerForException+0xd

    fffff800`020b98cf : fffff880`076c5438 fffff880`076c4ca0 fffff880`00000000 01cd3a11`dccb7d62 : nt!RtlDispatchException+0x410

    fffff800`0207ec02 : fffff880`076c5438 01cd3a11`efbea7ac fffff880`076c54e0 fffff780`00000014 : nt!KiDispatchException+0x16f

    fffff800`0207ca34 : 00000000`000007ff 00000000`00000650 fffffa80`34b97a60 fffff880`076c5560 : nt!KiExceptionDispatch+0xc2

    fffff880`02e17d08 : 00000000`00000000 fffff880`076c56b0 00000000`00000000 00000000`00000000 : nt!KiBreakpointTrap+0xf4

    fffff880`02e35614 : fffffa80`9c011570 00000000`0000000e 00000000`6349754d fffff800`021b34d3 : rdbss!RxFsdCommonDispatch+0xad8

    fffff880`08bcbcd7 : fffffa80`9c011570 fffffa80`6349754d fffffa80`21b0d620 00000000`00000000 : rdbss!RxFsdDispatch+0x224

    fffff880`019d5271 : fffffa80`1eb913d0 fffffa80`23651cc0 fffffa80`9c011570 00000000`00000000 : rdpdr!DrPeekDispatch+0x31f

    fffff880`019d3138 : fffff8a0`001878c0 fffffa80`23651cc0 00000000`00000001 00000000`00000000 : mup!MupiCallUncProvider+0x161

    fffff880`019d3b0d : fffffa80`9c011570 fffff880`019d1110 fffffa80`21b17de0 00000000`00000000 : mup!MupStateMachine+0x128

    fffff880`014de6af : fffffa80`1eb913d0 fffffa80`23651cc0 fffff880`076c5c00 fffffa80`9c011570 : mup!MupFsdIrpPassThrough+0x12d

    fffff880`0158b81a : fffffa80`1eb90010 00000000`00000003 fffffa80`1eb90010 fffff880`076c5c00 : fltmgr!FltpDispatch+0x9f

    fffff800`02397597 : fffffa80`21b17de0 fffff880`076c5ca0 fffffa80`21b17de0 fffffa80`9c011570 : mfehidk!DEVICEDISPATCH::DispatchPassThrough+0x13a

    fffff800`02397df6 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x607

    fffff800`0207e813 : fffffa80`20bfcb60 fffff880`076c5ca0 00000000`00000000 fffff800`02392200 : nt!NtDeviceIoControlFile+0x56

    00000000`76e6f72a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13

    00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x76e6f72a

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment