64-bit RPC traffic fails across ISA Sever 2006

64-bit RPC traffic fails across ISA Sever 2006

  • Comments 3
  • Likes

1. Introduction

 

This post describes an issue where two 64-bit Windows hosts are failing to communicate to each other using RPC . The hosts each operate in a network physically separated from each other by ISA Server 2006.  Figure 1 illustrates the basic scenario.

 

 

Figure 1 – Sample network diagram.

 

All other traffic allowed between these hosts functioned normally; only RPC calls were failing.

 

2. Identifying the problem

 

The traffic across the networks was working without problems for most of the protocols (ICMP, SMB, DNS, HTTP, etc). Only RPC Calls were failing and the actual RPC error was exposed by using the repadmin /showreps command when run from one DC. We got the error message below:  

 

The replication generated an error (1727): The remote procedure call failed and did not execute.

 

We used KB911799 (method 1) approach to understand where was failing and if ISA Server 2006 was really causing that. The tests showed that the RPC communication was failing only for communications between 64 bit platforms (Windows Server 2008 64 with Windows Server 2008 64, Windows Vista 64 with Windows Server 2008 64bit).

 

Looking to the network monitor trace that was taken from the DC (10.30.30.10) it was possible to see the moment of the failure when the DC from one side is trying to bind to the RPC in the other side:

 

TCP 3 Way Handshake for RPC:

10.30.30.10 10.40.40.16 TCP   TCP:Flags=......S., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102417964, Ack=0, Win=8192 (scale factor 8) = 2097152

 

10.40.40.16 10.30.30.10 TCP   TCP:Flags=...A..S., SrcPort=DCE endpoint resolution(135), DstPort=36457, PayloadLen=0, Seq=2217804385, Ack=102417965, Win=16384 (scale factor 0) = 16384

 

10.30.30.10 10.40.40.16 TCP   TCP:Flags=...A...., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102417965, Ack=2217804386, Win=257 (scale factor 8) = 65792

 

RPC Bind Request for the End Point Mapper (End Point Mapper's UUID is E1AF8308-5D1F-11C9-91A4-08002B14A0FA):

10.30.30.10 10.40.40.16 MSRPC MSRPC:c/o Bind:  UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT  Call=0x1  Assoc Grp=0x0  Xmit=0x16D0  Recv=0x16D0

- RPC: c/o Bind:  UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT  Call=0x1  Assoc Grp=0x0  Xmit=0x16D0  Recv=0x16D0

  - Bind: {E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT

     RpcVers: 5 (0x5)

     RpcVersMinor: 0 (0x0)

     PType: 0x0B - Bind

   + PfcFlags: 3 (0x3)

   + PackedDrep: 0x10

     FragLength: 160 (0xA0)

     AuthLength: 0 (0x0)

     CallId: 1 (0x1)

     MaxXmitFrag: 5840 (0x16D0)

     MaxRecvFrag: 5840 (0x16D0)

     AssocGroupId: 0 (0x0)

   - PContextElem:

      NContextElem: 3 (0x3)

      Reserved: 0 (0x0)

      Reserved2: 0 (0x0)

    + PContElem: 0x1

    - PContElem: 0x1

       PContId: 1 (0x1)

       NTransferSyn: 1 (0x1)

       Reserved: 0 (0x0)

     + AbstractSyntax: {E1AF8308-5D1F-11C9-91A4-08002B14A0FA} EPT

     + TransferSyntaxes: {71710533-BEBA-4937-8319-B5DBEF9CCC36} NDR64

    + PContElem: 0x1

 

 

RPC Bind Response:

10.40.40.16 10.30.30.10 MSRPC MSRPC:c/o Bind Ack:  Call=0x1  Assoc Grp=0x87F3 

- RPC: c/o Bind Ack:  Call=0x1  Assoc Grp=0x87F3  Xmit=0x16D0  Recv=0x16D0

  - BindAck:

     RpcVers: 5 (0x5)

     RpcVersMinor: 0 (0x0)

     PType: 0x0C - Bind Ack

   + PfcFlags: 3 (0x3)

   + PackedDrep: 0x10

     FragLength: 108 (0x6C)

     AuthLength: 0 (0x0)

     CallId: 1 (0x1)

     MaxXmitFrag: 5840 (0x16D0)

     MaxRecvFrag: 5840 (0x16D0)

     AssocGroupId: 34803 (0x87F3)

   + SecAddr: 135

   + Pad2: 0x1

   - PResultList:

      NResults: 3 (0x3)

      Reserved: 0 (0x0)

      Reserved2: 0 (0x0)

    - PResults: Provider rejection, Reason=Proposed transfer syntaxes not supported

       Result: Provider rejection

       Reason: Proposed transfer syntaxes not supported

     + TransferSyntax: {00000000-0000-0000-0000-000000000000} unknown

    + PResults: Acceptance, Reason=n/a

    + PResults: Negotiate Ack, Security Context Multiplexing Supported

 

The DC (10.30.30.10) sends an endpoint request for NTFRS Service UUID: f5cc59b4-4264-101a-8c59-08002b2f8426…

10.30.30.10 10.40.40.16 EPM   EPM:Request: ept_map: NDR, FrsRpc {F5CC59B4-4264-101A-8C59-08002B2F8426} v1.1, RPC v5, 0.0.0.0:135 (0x87) [DCE endpoint resolution(135)]

 

..but the DC 10.40.40.16 closes the connection

10.40.40.16 10.30.30.10 TCP   TCP:Flags=...A...F, SrcPort=DCE endpoint resolution(135), DstPort=36457, PayloadLen=0, Seq=2217804494, Ack=102418293, Win=65207 (scale factor 0) = 65207

 

10.30.30.10 10.40.40.16 TCP   TCP:Flags=...A...., SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102418293, Ack=2217804495, Win=257 (scale factor 8) = 65792

 

10.30.30.10 10.40.40.16 TCP   TCP:Flags=...A...F, SrcPort=36457, DstPort=DCE endpoint resolution(135), PayloadLen=0, Seq=102418293, Ack=2217804495, Win=257 (scale factor 8) = 65792

 

The key element in the above trace is the Provider Results, that appears as Provider rejection, Reason=Proposed transfer syntaxes not supported. Briefly this means that the DC 10.40.40.16 appeared to reject the bind request from DC 10.30.30.10 because the proposed transfer syntax is no supported. But this was not actually the DC 10.40.40.16 that sent that, that was ISA Server, let's understand why.

 

Note1: for a complete list of the meaning of each rejection reasons review Section 2 of the ISO 8823 standard.

 

3. RPC Filter

 

If you review the RPC Architecture you will notice that there is a component that belongs to the rpcrt4.dll called Marshalling Engine.  This component is responsible for providing a common RPC interface between RPC clients and servers through NDR (Network Data Representation). There are two transfer syntaxes variations for the NDR:

 

·         NDR20 – Used in 32-bit Architecture.

·         NDR64 – Used in 64-bit Architecture.

 

When two 64bit Windows Operating System are communicating using RPC they will negotiate the marshalling engine to use.  Since they both prefer NDR64, this is likely to be the format used. Natively, ISA Server 2006 RPC Filter doesn’t support NDR64; therefore the RPC Filter will reject any RPC communication which uses NDR64.

 

4. Resolution

 

To fix the problem you should install ISA Server 2006 SP1.

 

 

Author

Yuri Diogenes

Security Support Engineer – Microsoft CSS Forefront (ISA/TMG) Team

 

Technical Reviewers

Jim Harrison

Microsoft Forefront (ISA/TMG) Sustained Engineering Team

 

Doron Juster

Microsoft Forefront (ISA/TMG) Sustained Engineering Team

 

Comments
  • Hmm. Is it my deja vu or this bug was silently migrated to TMG 2010 RTM?

    1. Site A has ISA 2006 SP1 server as its gateway.

    2. Site B has ISA 2006 SP1 server as its gateway.

    3. Computers A1 and A2 are located at site A.

    4. Computers B1 and B2 are located at site B.

    5. Computers A1 and B1 are running 32-bit editions of Windows.

    6. Computers A2 and B2 are running 64-bit editions of Windows.

    7. I create Site-to-Site VPN tunnel between Site A and Site B.

    8. I try to establish RPC connection between computers A1 and B1. (How do I test? At computer A1 run eventvwr.msc /computer:"B1"). It works.

    9. I try to establish RPC connection betwwen computers A2 and B2. (How do I test? At computer A2 run eventvwr.msc /computer:"B2"). It works.

    10. I replace one of the gateways with TMG 2010 RTM and re-establish VPN tunnel.

    11. I try to establish RPC connection between computers A1 and B1. It works.

    12. I try to establish RPC connection betwwen computers A2 and B2. It does not work.

    13. I disable RPC Filter at TMG 2010 Server. (Note: a couple of times when I disabled and re-enabled the Filter my TMG machine went into BSoD. But eventually it worked as expected).

    8. I try to establish RPC connection between computers A1 and B1. It works.

    9. I try to establish RPC connection betwwen computers A2 and B2. It works.

    So... What's wrong with TMG RPC filter?

  • Sorry for broken numbering in my previous comment. The last two points should obviously be marked as ##14 and 15 respectively.

    I should also mention that my observations are inderectively confirmed by the following Forum threads.

    http://social.technet.microsoft.com/Forums/en-US/Forefrontedgegeneral/thread/f3a9d5ab-d80c-46af-b582-d7856b8c53bf

    http://social.technet.microsoft.com/Forums/en-US/Forefrontedgesetup/thread/041900b0-3150-42d8-a6d6-b78e90e15be2

    http://social.technet.microsoft.com/Forums/en-US/Forefrontedgesetup/thread/5fc378cf-c782-4e44-8f34-c911b0bcf794

    All of these Forum threads suppose disabling RPC Filter as a workaround for an issue when RPC traffic is being erroneously blocked.

    But disabling the Filter has its obvious drawbacks. The most significant one is that acces rules that allow RPC Protocols only (e.g. remote management rules from System Policy) stop working. The only workaround is to replace these rules with allowing all outbound traffic, which is not nice at all.

    So disabling RPC Filter could hardly be considered as an acceptable workaround.

  • More than 20 hours for troubleshooting problem of DC replications... Microsoft, I Hate you.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment