Configuring the RPC filter to support DCOM traffic is a particular pain point in ISA Server configuration. This entry provides a quick overview of the filter, the implications of the "Enable strict RPC compliance" setting, and some information on common issues with DCOM traffic.

RPC Filter

ISA Server's RPC filter monitors RPC traffic between hosts, and sets up secondary connections as required for RPC traffic. For outbound RPC requests, ISA Server inspects the traffic flowing between the source and destination. For incoming requests to published RPC servers, ISA Server inspects the traffic flowing between the source and destination, and dynamically opens and closes ports on the external published listener based on the protocols used by the RPC client and server. The RPC filter cannot be applied to traffic tunneled over another protocol, such as RPC over HTTP. When a rule references a protocol that is bound to the RPC filter, then the filter is applied to traffic matching the rule. By default, ISA Server provides three predefined RPC protocols for use by inbound and outbound RPC traffic:

• The Exchange RPC Server protocol has a list of UUIDs used for publishing Exchange.
• The RPC Server (all interfaces) protocol is used for publishing other RPC servers.
• The RPC (all interfaces) protocol is used in access rules for outbound RPC access.

By default the predefined RPC protocols are bound to the RPC filter. You can also create custom protocol RPC definitions using the New RPC Protocol Definition Wizard. When you create a custom RPC protocol using the wizard, the following defaults are applied:

• Port TCP 135 is enabled for the custom protocol
• The custom protocol is bound to the RPC filter

Note that traffic defined as "outbound" is not handled by ISA Server based on specific UUIDs, so it isn't possible to set up a custom protocol definition for specific UUIDs. For traffic defined as "incoming", you can create a custom protocol with specific UUIDs, either by selecting them from the endpoint mapper list, or by manually creating them.

Enable strict RPC compliance

For publishing rules ISA Server blocks DCOM traffic, and this setting cannot be modified. For access rules, a default "Enable strict RPC compliance" setting is configured on each RPC rule. With this setting in place, DCOM traffic is blocked. More specifically, any traffic (such as DCOM) that does not start an RPC exchange by communicating with the endpoint mapper is blocked. Turning off the "Enable strict RPC compliance" setting does not specifically allow DCOM traffic. It simply disables filtering for this traffic after the endpoint mapper requirements have been met. To allow DCOM traffic through an RPC access rule, either of the following is required:

•  An access rule that allows all protocols between the specified source and destination.
• Alternatively, you can do the following:
• Create a custom outbound protocol using a port that is not associated with any other application.
• Configure the RPC application or DCOM endpoint to use the custom protocol port as a static port.
• Create an access rule to allow the protocol between the required source and destination.

Hints for Troubleshooting RPC Server Publishing

• Ensure that you are using a recognized protocol definition in the rule. Either use a predefined protocol, or ensure that the custom protocol is defined correctly. Custom protocol definitions should be inbound, TCP port 135, with the correct UUID interfaces.
• Check that the publishing rule is enabled.
• Verify that the RPC filter is enabled for the protocol - to handle the secondary connection, and inspect the traffic.
• For Exchange server publishing use the New Mail Server Publishing Wizard. To publish other RPC servers, create a server publishing rule for the internal server, using the IP address of the adapter associated with the ISA Server External network.
• Check that the external IP address of the ISA Server and the IP address of the internal server are specified correctly in the rule.
• Ensure that a network rule between the source and destination networks exists, and that the rule relationship is appropriate to the traffic flow.
• Check that the RPC client/server application is working without ISA Server in the middle, by using a client in the Internal network to communicate with the published RPC server.

 Common Issues

• Problem: You cannot use DCOM between a computer in the Remote Management Computers sets and the ISA Server computer
• Workaround: In the system policy rule, there is no option to configure remote management to allow non-strict RPC traffic, so all DCOM traffic between the Remote Management Computers set and the Local Host network (the ISA Server) is dropped. As a workaround, remove the computer from the set, and create an additional access rule for the same traffic. Then clear the "Enforce strict RPC compliance setting" on the rule.

• Problem: When you request a certificate using the Certificate MMC snap-in, the request fails. This occurs even if the CA is started and you have sufficient permissions to request a certificate.
• Workaround: This issue occurs because DCOM is required to acquire a certificate (this issue also occurs if you are using CA Web enrollment).
• If ISA Server is requesting the certificate, disable the "Enforce strict RPC compliance setting" on the system policy rule. To do this, on the Firewall Policy tab of ISA Server Management, click Edit System Policy on the Tasks tab. Select the Active Directory group in the Configuration Groups list. On the General tab, clear the Enforce strict RPC compliance checkbox.
• If an internal host is requesting the certificate from another network through ISA Server, do the following: in the Firewall Policy tab of ISA Server Management, right-click the access rule allowing the traffic, and then click Configure RPC protocol. On the Protocol tab, clear Enforce strict RPC compliance.
• NOTE: In either case, after clearing the setting you need a rule to allow all traffic, or a rule for a custom protocol as described in the blog section "Enable strict RPC compliance". For more information on CA configuration in a firewall environment, the following document is a useful reference: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/webenroll.mspx.

• Problem: Issue with deploying the System Center Data Protection Manager 2006 agent on a computer running ISA Server 2004 or ISA Server 2006.
• Workaround: See KB http://support.microsoft.com/default.aspx/kb/929102

• Problem: Issue with installing MOM agent on a computer running ISA Server 2004 or ISA Server 2006.
• Workaround: See KB http://support.microsoft.com/default.aspx/kb/933989

• Problem: WMI scripts fail to run from remote systems
• Workaround: Create an access rule that allows RPC (All Interfaces) from the Internal network to the Local Host network. After creating the rule, right-click it and select Configure RPC Protocol. On the Protocol tab, clear Enable strict RPC compliance. After clearing the setting, you need arule to allow or traffic, or a custom protocol as described in the blog section "Enable strict RPC compliance".

Rayne Wiselman

ISA Server Product Team