Active Route Gets Removed on Windows 2008 Failover Cluster IP Address Offline

Active Route Gets Removed on Windows 2008 Failover Cluster IP Address Offline

  • Comments 2
  • Likes

We have received calls adding static routes on Windows Server 2008 Failover Clustering nodes and wanted to pass along some important information regarding this.  The issue is that when you add a static persistent route to a network adapter that is on a Windows Server 2008 Failover Cluster and take a Clustered IP Address offline (or move it to another node), the “Active” route is removed and no connections can be made using this route even though it still shows as persistent.  Once you bring the Clustered IP Address back online, the active route is returned.

First, I want to mention that the networking architecture in 2008 Failover Clustering has been rewritten from the ground up and now we have a  our own internal route listings as well as our own adapter (Microsoft Failover Cluster Virtual Adapter).  I will not into the specifics of it here, but you can read more about it in the blog “What is a Microsoft Failover Cluster Adapter anyway?”. 

On to the problem and resolution.  As a little setup, here is the configuration that I want to discuss.

ClusterNode1

Physical IP Address: 10.44.60.4

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

 

ClusterNode2

Physical IP Address: 10.44.60.3

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

Failover Cluster Virtual IP Address

IP Address: 10.44.60.6

I also have a backup server that I use to create backups using an IP Address of 10.51.0.1 and subnet mask 255.255.0.0 that will use the same default gateway above.  Most Network Administrators would use the following ROUTE.EXE command to add a persistent static route to the local tables so that a connection can be made.

route -p add 10.51.0.0 mask 255.255.0.0 10.44.60.1

So with everything online (including the Failover Cluster Virtual IP Address) on ClusterNode1, I can do a ROUTE PRINT command to display my IP Address version 4 table and see this.  As a side note, I am just pulling the necessary information from the Route Table.

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination  Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276   
<<---
10.44.60.4           255.255.255.255  On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.60.6           255.255.255.255  On-link       10.44.60.4    276   
<<--- Clustered IP Address
10.44.255.255        255.255.255.255  On-link       10.44.60.4    276   <<---
10.51.0.0            255.255.0.0      10.44.60.1    10.44.60.4     21    <<--- Static Route added
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255  On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                
<<--- Persistent Route added
===========================================================================

As long as the Clustered IP Address of 10.44.60.6 is online on this node, all is well.  However, if I were to take the 10.44.60.6 IP Address offline, things change.

C:\>cluster res "IP Address 10.44.60.6" /offline
Taking resource ''IP Address 10.44.60.6'' offline...
Resource                 Group         Node             Status
--------------------     ----------    ---------------  ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Offline

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination  Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276   
<<---
10.44.60.4           255.255.255.255  On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.255.255        255.255.255.255  On-link       10.44.60.4    276  
<<---
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255  On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                
<<--- Persistent Route added
===========================================================================

Notice here that the Clustered IP Address 10.44.60.6 as well as the 10.51.0.1 “Active” route is removed.  Because the 10.51.0.0 route is removed, connectivity to the backup server is lost.  If you bring the Clustered IP Address 10.44.60.6 online again, the “Active” routes are re-populated again and connectivity to the backup server is restored.

C:\>cluster res "IP Address 10.44.60.6" /online
Bringing resource ''IP Address 10.44.60.6'' online...
Resource                 Group         Node             Status
--------------------     ----------    ---------------  ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Online

 

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination  Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276   
<<---
10.44.60.4           255.255.255.255  On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.60.6           255.255.255.255  On-link       10.44.60.4    276   
<<--- Clustered IP Address
10.44.255.255        255.255.255.255  On-link       10.44.60.4    276   <<---
10.51.0.0            255.255.0.0      10.44.60.1    10.44.60.4     21    <<--- Static Route added
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255  On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                
<<--- Persistent Route added
===========================================================================

According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.

The ROUTE.EXE command has additional parameters of METRIC and INTERFACE that you would need to specify that will bind the route to the card itself.

C:\>route /?

Manipulates network routing tables.

ROUTE [-f] [-p] [command [destination]

                  [MASK netmask]  [gateway] [METRIC metric]  [IF interface]

  interface    the interface number for the specified route.

  METRIC       specifies the metric, ie. cost for the destination.

So what you need to do first is determine what the interface is so that we can bind the route to it.  When doing the ROUTE PRINT or NETSH command, it will give you the interfaces at the top first.  Something similar to this:

C:\>route print

IPv4 Route Table

===========================================================================

Interface List

23 ...00 15 5d 4a ac 06 ...... Local Gigabit Controller

19 ...00 15 5d 4a ac 01 ...... Local Gigabit Controller #2

18 ...00 15 5d 4a ac 00 ...... Local Gigabit Controller #3

===========================================================================

-or-

C:\>netsh int ipv4 show int

Idx  Met   MTU       State        Name

---  ---  -----      -----------  -------------------

18   50 4294967295  connected    Local Gigabit Controller #3

19    5   1500      connected    Local Gigabit Controller #2

23    5   1500      connected    Local Gigabit Controller

I can go into the Network and Sharing Center if I have to to see which card is on this network.  In my particular case, the “Local Gigabit Controller #3” is the one I want to use.  So to get my persistent route to stay even though the Clustered IP Address goes offline, my command would be below.  Please note that the METRIC is not needed as a requirement of the command.

route -p add 10.51.0.0 mask 255.255.0.0 0.0.0.0 metric 276 if 18

Now, the “Active” route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline.  A good rule of thumb, moving forward, would be that if you are adding a static persistent route, start specifying the 0.0.0.0 and the interface as this is the proper supported commands from a networking perspective.  This will result in the proper functioning no matter if Failover Clustering is configured or not.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Thanks for the info. We encountered the same problem on a Windows Server 2008 R2 (without Failover Clustering though) and this article provided us with the necessary clues.

    Persistent routes were disappearing from the list of active routes when the DeleteIPAddress function was used (we're dynamically adding and removing an extra IP address at runtime).

    One concern I have is that the interface ids apparently might change after a reboot, and I wonder what will happen with the persistent route in such a scenario.

    Will the persistent route still be bound to the correct interface after the interface ids get renumbered for some reason?

  • John Marlin,

    These days I keep hearing about switching over NETSH rather using ROUTE ADD.

    Internally technically been replaced with NETSH?

    blogs.technet.com/.../windows-2008-multi-subnet-clusters-and-using-static-routes.aspx