===========================================================
***SEE UPDATE***
With the enhancements in Windows 2008 to allow for multi-subnet clustering it is becoming more common to see this utilized with Exchange 2007 SP1 installations.
When implementing a clustered solution, it is a requirement that there be a minimum of two interfaces on each node, and that each node can maintain communications across those interfaces. I see administrators implement this requirement in two different fashions with multi-subnet clusters:
If you are the second bullet, you’ll want to continue reading this blog. (If you are the first bullet you’ll probably want to read it anyway since you’ve made it this far…)
For users that have a configuration where both network interfaces are in different subnets this will generally require routing between those two subnets. A common mis-configuration that I see in this design is the use of default gateways on both of these network interfaces.
When a user attempts to configure two network interfaces each with a default gateway, the following error is noted from the operating system:
The text in this message is specifically important as it highlights at this time that this configuration will not produce the desired results.
The most likely cluster configuration where Exchange is used, with this type of clustering, is cluster continuous replication (CCR). When multiple default gateways are defined, users may see inconsistent results in the performance and ability to replicate logs between the nodes. The replication issues between nodes are also exacerbated when continuous replication hostnames are used utilizing the secondary networks with the default gateway assigned. These issues are secondary to any issues that the cluster service many have maintaining communications between the nodes and any communications issues clients may have connecting to the nodes.
If the default gateways are removed from the “private” adapters, reliable routed communications can only occur over the “public” interface. So…if two default gateways cannot be used, how should we ensure proper communications over both the “public” interface and “private” interface where both reside in different routed subnets.
The first part of this solution is to ensure that the binding order of the network interfaces is set correctly in the operating system. To confirm the binding order:
The second part of the solution is to maintain the default gateway on the “public” interface.
The third part of the solution is to enable persistent static routes on the “private” interfaces. In terms of the routes we simple need to configure routes to other “private” networks using gateway addresses that have the ability to route between those “private” networks. All other traffic not matching this route should be handled by the default gateway of the “public” adapter.
Let’s take a look at an example.
I desire to have a two node Exchange 2007 SP1 CCR cluster on Windows 2008 with each node residing in a different subnet.
NodeA:
Public
Private
NodeB:
Public:
(Note that gateway on network is not the default gateway setting but is the gateway on the private interface network that can route packets to the private network on the other nodes.)
In this case I would want to establish the necessary persistent static routes on each node. In order to accomplish this, I can use the route add command. The structure of the route command:
NodeA: Route add 10.0.1.0 mask 255.255.255.0 10.0.0.254 –p
NodeB: Route add 10.0.0.0 mask 255.255.255.0 10.0.1.254 –p
The –p switch will ensure that the routes are persistent lasting after a reboot. Failure to use the –p will result in the routes being removed post a reboot operation.
You can verify that the routes are correct by running route print and reviewing the persistent route information.
By utilizing only a default gateway on the “public” adapter, and static routes on the “private” adapters, you can ensure safe routed paths for client communications, cluster communications, and replication service log shipping.
========================================================
Update – 1-18-2010
With Windows 2008 and Windows 2008 R2 the recommendation to manage static routes has changed. Although route add should work the management of routes has technically been replaced with functionality in netsh. Therefore, it is a recommendation that the netsh commands be utilized to implement and manage static routes.
I will leave the previous information un-edited in the blog since many people have used it.
The first step in implementing static routes with the netsh command is to determine the interface names. The interface name is the logical name assigned to the network connection – for example Local Area Connection 1. It is recommended that these networks be named into something more logical, for example LAN-Replication-A. The logical network names may be the same on all nodes.
You can also determine that adapter name from an ipconfig /all. (Note the name listed below in RED)
Windows IP Configuration
Host Name . . . . . . . . . . . . : DAG-1 Primary Dns Suffix . . . . . . . : exchange.msft Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : exchange.msft
Ethernet adapter LAN:
Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter Physical Address. . . . . . . . . : 00-15-5D-00-02-07 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::dd27:d7f6:549f:6b9b%11(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.0.1(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 IPv4 Address. . . . . . . . . . . : 192.168.0.2(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 192.168.0.254 DHCPv6 IAID . . . . . . . . . . . : 234886493 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-45-7C-F8-00-15-5D-00-02-07 DNS Servers . . . . . . . . . . . : 192.168.0.253 192.168.0.252 192.168.0.251 Primary WINS Server . . . . . . . : 192.168.0.253 Secondary WINS Server . . . . . . : 192.168.0.252 192.168.0.251 NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter LAN-Replication-A:
Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter #2 Physical Address. . . . . . . . . : 00-15-5D-00-02-08 DHCP Enabled. . . . . . . . . . . : No Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 10.0.0.1(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : NetBIOS over Tcpip. . . . . . . . : Disabled
The netsh command format to add static routes looks like:
netsh interface ipv4 add route <IP/Mask> “InterfaceName” Gateway
Using the information from the above example, the following netsh commands would be utilized in place of route add:
NodeA: netsh interface ipv4 add route 10.0.1.0/24 “LAN-Replication-A” 10.0.0.254
NodeB: netsh interface ipv4 add route 10.0.0.0/24 “LAN-Replication-A” 10.0.1.254
The netsh command automatically assumes – unless otherwise specified in the command – that the route added is persistent.
If the command completes successfully the route addition can be verified by running:
netsh interface ipv4 show route
The following is sample output with the added route in RED (output truncated to show sample line including prefix and gateway):
C:\>netsh interface ip show route
Prefix Idx Gateway/Interface Name
------------------------ --- ------------------------
10.0.1.0/24 11 LAN-Replication-A
This is how the netsh command can be used to accomplish what would have previously been done with route add.
Update 9/18/2012:
Updated the netsh verification command to show correct syntax.