RPC to Go v.3 – Named Pipes

In this blog I’d like to give some information on what Named Pipes are, what a Named Pipes connection looks like, and some common things to look for if there is an issue. I do hope you have read RPC to Go v.1 and v.2. It makes this much easier to understand. Either way, I’ll try to keep this blog from getting complicated. I considered making it ‘less technical,’ but if you’re reading about RPC, you’re likely technical to a degree.

A customer case I worked on inspired this blog. I went onsite as they had a problem that was approaching its first birthday. The resolution was two-fold. The initial problem was resolved using Process Explorer to find handle leaks (tool available at www.systinternals.com ). The other issue could only be found by someone having a watchful eye on the flow of traffic.

This particular issue involved one client machine (XP SP2), a web server (Server 2003 with IIS), and a database server (Server 2003 with SQL 2005). The users accessed a web-based application in which they performed multiple tasks in one window. During certain tasks, they received varying errors. Most of the errors pointed to the file not existing (even though it did) or the response for query timing out.

After the initial issue was resolved, I crossed my fingers and had the customer start testing. I focused on three key areas: the process explorer (or task manager), the number of concurrent connections to both servers (using “netstat –ano” from a command prompt at the servers), and a network capture. As the number of connections to the server grew I noticed that the response time from the server to the clients increased. I checked process explorer. System resources were minimal on the application server. The same was true for the SQL server.

I filtered my capture for web (TCP port 80) and SQL (TCP port 1433 natively) traffic. These connections opened and closed with no issues. Becoming more perplexed, I removed the filter. I noticed an alarming number of connections on port 445 between the web and SQL servers. The SQL server sent an overwhelming number of its responses with STATUS_PIPE_DISCONNECTED. As it turned out, in the application code, the “call socket” function for the SQL query procedure was written to use Named Pipes to connect to the SQL server. We configured SQL to listen for Named Pipes. Issues resolved. This must be configured.

WHAT ARE NAMED PIPES?

Named Pipes, like RPC, allow inter-process communications (IPC). According to MSDN, named pipes are used to transfer data between processes that are not related processes and between processes on different computers. The differences between inter-process communications protocols include their capabilities and the way the socket connections are made. For a complete list of differences, visit the following link.

http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_pipes_for_ipc

I just want to focus on a Named Pipes connection establishment. Unlike RPC, there is no end point mapper (EPM) connection. Inter-process communications between separate hosts must accomplish several things:

• A connection must be established on a pre-determined port number.
• A level of authentication must be agreed upon and then met.
• The remote process must be identified (even though the listening port is specified)
  • RPC uses a Universally Unique Identifier (UUID) for process identification
  • Named Pipes uses \\MachineName\IPC$

In order to accomplish all three, Named Pipes uses the Microsoft implementation of CIFS – the Server Message Block protocol, or SMB (blog coming within a couple months for clarity – promise. In the interim, here’s a great link: http://msdn.microsoft.com/en-us/library/aa365233.aspx )

Once there’s a connection to the IPC$ tree, subsequent RPC binds and calls are encapsulated in SMB and sent over the wire using TCP port 139 or 445 natively. When binding to a new process, an SMB Create Andx Request is sent. The request for the new process is \\machinename\IPC$\ServiceName i.e. “wkssvc,” which is the workstation service.

The RPC portion does have typical RPC information. There is still a UUID, a transfer syntax and OpNum. There is also an ‘AssociationGroupID.’ The latter isn’t covered in previous blogs and is overkill for this blog.

WHAT TO LOOK FOR

• Does my traffic hit the wire? If no:
  • Make sure the name is resolvable.
  • Make sure there is a route the resolved IP address
• Hits the wire, but no connection is made to the remote host:
  • Make sure the remote host is listening on port 139 or 445. The Server service must start.
  • Check for firewall issues at the client, server and intermediate network devices.
• Makes a connection but is reset during the SMB portion:
  • Make sure the authenticating user has rights to the IPC$ share on the remote host.
  • Check your auth type. The source host must have a route to the DC for Kerberos auth.
• SMB works fine, but RPC call gets a BIND_NAK response.
  • Verify the remote service is running. (Check the UUID in the RPC portion of the frame.) Refer to previous “RPC to Go” versions for more troubleshooting info.
• Looking at a network capture? IPC$ is good about sending error messages back. There aren’t many blogs for these errors. Carefully examine where, in transmission, the errors occurred.

NETWORK CAPTURES

This is a concise example of a Named Pipes connection to a server named Fabfile-1 using remote registry. I’ll show the SMB tree connection to IPC$, the connection to the remote procedure, and the relevant RPC info.

1. In the Tree Connect Request, notice the file structure:

2. In the Create Andx Request, notice WinReg is requested as a file in the IPC$ share

67 08:43:34.725799 192.168.3.100         192.168.3.5           SMB      NT Create AndX Request, FID: 0x4000, Path: \winreg
SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        [Response in: 68]
        SMB Command: NT Create AndX (0xa2)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x18
        Flags2: 0xc807
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
       Tree ID: 2048  (\\FABFILE-1\IPC$)
        Process ID: 792
        User ID: 2048
        Multiplex ID: 192
    NT Create AndX Request (0xa2)
      [FID: 0x4000 (\winreg)]

3. In the encapsulated RPC bind, notice the UUID for WINREG and the’ x86’ transfer syntax. This is presented to the remote process when a procedure call is made.

Ctx Item[1]: ID:0
        Context ID: 0
        Num Trans Items: 1
        Abstract Syntax: WINREG V1.0
            Interface: WINREG UUID: 338cd001-2244-31f1-aaaa-900038001003
            Interface Ver: 1
            Interface Ver Minor: 0
       Transfer Syntax[1]: 8a885d04-1ceb-11c9-9fe8-08002b104860 V2
            Transfer Syntax: 8a885d04-1ceb-11c9-9fe8-08002b104860  ver: 2

Hopefully you’ve gained some insight on Named Pipes, its connection establishment and some “gotcha” to keep an eye on. I do hope the “RPC to Go” saves a couple of support calls in the near future.

-Rich Chambers