We have received calls about adding static routes on Windows Server 2008 Failover Clustering nodes and wanted to pass along some important information regarding this. The issue is that when you add a static persistent route to a network adapter that is on a Windows Server 2008 Failover Cluster and take a Clustered IP Address offline (or move it to another node), the “Active” route is removed and no connections can be made using this route even though it still shows as persistent. Once you bring the Clustered IP Address back online, the active route is returned.

I want to mention that the networking architecture in Windows Server 2008 Failover Clustering has been rewritten from the ground up and I will not into the specifics of it here. You can read more about it in the blog, “What is a Microsoft Failover Cluster Adapter anyway?”. 

On to the problem and resolution. As a little setup, here is the configuration that I want to discuss.

ClusterNode1
Physical IP Address: 10.44.60.4
Physical Subnet Mask: 255.255.0.0
Default Gateway: 10.44.60.1

ClusterNode2
Physical IP Address: 10.44.60.3
Physical Subnet Mask: 255.255.0.0
Default Gateway: 10.44.60.1

Failover Cluster Virtual IP Address
IP Address: 10.44.60.6

I also have a backup server that I use to create backups using an IP Address of 10.51.0.1 and subnet mask 255.255.0.0 that will use the same default gateway above. Most Network Administrators would use the following ROUTE.EXE command to add a persistent static route to the local tables so that a connection can be made.

route -p add 10.51.0.0 mask 255.255.0.0 10.44.60.1

So with everything online (including the Failover Cluster Virtual IP Address) on ClusterNode1, I can do a ROUTE PRINT command to display my IP Address version 4 table and see this. As a side note, I am just pulling the necessary information from the Route Table.

C:\>route print -4
IPv4 Route Table
==================================================================
Active Routes:
Network Destination  Netmask          Gateway     Interface  
10.44.0.0            255.255.0.0      On-link     10.44.60.4 
<<---
10.44.60.4           255.255.255.255  On-link     10.44.60.4  <<--- Phys. Node IP Address
10.44.60.6           255.255.255.255  On-link     10.44.60.4  <<--- Clustered IP Address
10.44.255.255        255.255.255.255  On-link     10.44.60.4  <<---
10.51.0.0            255.255.0.0      10.44.60.1  10.44.60.4  <<--- Static Route added
224.0.0.0            240.0.0.0        On-link     10.44.60.4 
<<---
255.255.255.255      255.255.255.255  On-link     10.44.60.4  <<---
===================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1     <<--- Persistent Route
===================================================================

As long as the Clustered IP Address of 10.44.60.6 is online on this node, all is well. However, if I were to take the 10.44.60.6 IP Address offline, things change.

C:\>cluster res "IP Address 10.44.60.6" /offline
Taking resource ''IP Address 10.44.60.6'' offline...
Resource                 Group         Node             Status
--------------------     ----------    ---------------  ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Offline

C:\>route print -4
IPv4 Route Table
==================================================================
Active Routes:
Network Destination  Netmask          Gateway     Interface  
10.44.0.0            255.255.0.0      On-link     10.44.60.4  <<---
10.44.60.4           255.255.255.255  On-link     10.44.60.4  <<--- Phys. Node IP Address
10.44.255.255        255.255.255.255  On-link     10.44.60.4  <<---
224.0.0.0            240.0.0.0        On-link     10.44.60.4  <<---
255.255.255.255      255.255.255.255  On-link     10.44.60.4  <<---
===================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1     <<--- Persistent Route
===================================================================

Notice here that the Clustered IP Address 10.44.60.6 as well as the 10.51.0.1 “Active” route is removed. Because the 10.51.0.0 route is removed, connectivity to the backup server is lost. If you bring the Clustered IP Address 10.44.60.6 online again, the “Active” routes are re-populated again and connectivity to the backup server is restored.

C:\>cluster res "IP Address 10.44.60.6" /online
Bringing resource ''IP Address 10.44.60.6'' online...
Resource                 Group         Node             Status
--------------------     ----------    ---------------  ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Online

C:\>route print -4
IPv4 Route Table
==================================================================
Active Routes:
Network Destination  Netmask          Gateway     Interface  
10.44.0.0            255.255.0.0      On-link     10.44.60.4  <<---
10.44.60.4           255.255.255.255  On-link     10.44.60.4  <<--- Phys. Node IP Address
10.44.60.6           255.255.255.255  On-link     10.44.60.4  <<--- Clustered IP Address
10.44.255.255        255.255.255.255  On-link     10.44.60.4  <<---
10.51.0.0            255.255.0.0      10.44.60.1  10.44.60.4  <<--- Static Route added
224.0.0.0            240.0.0.0        On-link     10.44.60.4  <<---
255.255.255.255      255.255.255.255  On-link     10.44.60.4  <<---
===================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1     <<--- Persistent Route
===================================================================

According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified. Specifying this entry will be the support stance moving forward from a networking perspective when you are talking about adding static persistent routes on any machine (Cluster or not).

The ROUTE.EXE command has additional parameters of METRIC and INTERFACE that you would need to specify that will bind the route to the card itself.

C:\>route /?

Manipulates network routing tables.

ROUTE [-f] [-p] [command [destination]
                  [MASK netmask]  [gateway] [METRIC metric]  [IF interface]

  interface    the interface number for the specified route.
  METRIC       specifies the metric, ie. cost for the destination.

So what you need to do first is determine what the interface is so that we can bind the route to it.  When doing the ROUTE PRINT or NETSH command, it will give you the interfaces at the top first.  Something similar to this:

C:\>route print
IPv4 Route Table
============================================================
Interface List
23 ...00 15 5d 4a ac 06 ...... Local Gigabit Controller
19 ...00 15 5d 4a ac 01 ...... Local Gigabit Controller #2
18 ...00 15 5d 4a ac 00 ...... Local Gigabit Controller #3
============================================================

-or-

C:\>netsh int ipv4 show int
Idx  Met   MTU       State        Name
---  ---  -----      -----------  -------------------
18   50 4294967295  connected    Local Gigabit Controller #3
19    5   1500      connected    Local Gigabit Controller #2
23    5   1500      connected    Local Gigabit Controller

I can go into the Network and Sharing Center if I have to to see which card is on this network. In my particular case, the “Local Gigabit Controller #3” is the one I want to use. So to get my persistent route to stay even though the Clustered IP Address goes offline, my command would be this. Note that the METRIC value is not needed.

route -p add 10.51.0.0 mask 255.255.0.0 0.0.0.0 metric 276 if 18

Now, the “Active” route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline. If you are adding a static persistent route, start specifying the 0.0.0.0 and the interface as this is the proper supported commands going forward from a networking perspective. This will result in the proper functioning no matter if Failover Clustering is configured or not. This will result in the proper functioning no matter if Failover Clustering is configured or not.

- John Marlin