Hi! My name is Sai Balaji and I work with Microsoft Networking Support in Bangalore. I primarily help Premier Customers resolve their real time Windows Networking issues. The ‘push’ that I got to write this first post of mine goes back to my own experiences with this simple and elegant component called ARP (Address Resolution Protocol). Cutting out the allusive and clichéd introductory part, let us get ourselves aligned with the agenda.
Things that you can look forward to read on this post:
DAD (Duplicate Address Detection), gratuitous ARP and Windows machines:
A case study of Windows Vista/7’s gratuitous ARP Receive and Send behaviors:
Before diving into these pointers, you might want to understand about how DAD and gratuitous ARP works on a surface level.
You can make use of this neatly written TechNet blog that talks about this: http://blogs.technet.com/b/networking/archive/2009/03/30/tcp-ip-networking-from-the-wire-up.aspx
Here is a snippet of that blog to give you an overview:
Address conflict detection (Gratuitous ARP)
ARP is also used to detect IP address conflicts. Address conflict detection is used to insure that a system that is brought up on the network or that is assigned a new IP address does not have an address that conflicts with a system already on the network.
In address conflict detection, we use what is known as a Gratuitous ARP. When a system is configured with an IP address either manually or by DHCP it will send a Gratuitous ARP to insure that another node on the network is not already configured with this IP address. In the case of a conflict the two nodes are defined as follows. The Offending Node is the node that is sending the gratuitous ARP, and the Defending Node is a system already configured with the IP Address in question. The contents of this request and how this affects the ARP cache on other systems on the network differs depending on the OS.
First let me introduce some of the terms you might see further ahead. I just want to make sure that you get acquainted to them beforehand. The important ones, I suppose, are SHA and SPA.
In an ARP Request the fields are filled in as follows:
Gratuitous ARP Send behaviors:
When a new IP address is added to an Interface, this is how the various clients send gratuitous ARP packets for DAD.
Windows XP – Sends six gratuitous ARP packets at intervals of 1s with the IP address that it is trying to add in the SPA field and its MAC address in the SHA field.
Windows Vista/7 – Sends three gratuitous ARP packets at intervals of 1s with ‘0.0.0.0’ in SPA field and its MAC address in the SHA field. (I will explain why this change was made in a later section)
Windows 8 – Sends three gratuitous ARP packets at intervals of 1s with ‘0.0.0.0’ in SPA field and its MAC address in the SHA field, waits for 1 more second and then sends another gratuitous ARP packet with the ‘IP address’ that it is trying to add in the SPA field and its MAC address in the SHA Field. (Again, I will explain why this change was made in a later section)
Gratuitous ARP Receive behaviors:
When a gratuitous ARP packet, which contains ‘0.0.0.0’ in SPA, is received by a Windows Client, nobody is going to make any change to the ARP/Neighbor Cache entry because it is 0.0.0.0. The whole purpose of sending out such a gratuitous ARP packet was to prevent ARP poisoning and this technique fights it tooth and nail and emerges successful.
When a gratuitous ARP packet, which contains an ‘IP address’ in SPA, is received by a Windows Client, this is how the Clients react to it.
Windows XP – If the SPA matches with an IP in its ARP cache table entries, it will go ahead and replace the old entry with the new one i.e. the old MAC will be replaced by the new MAC that is present in the SHA field of the gratuitous ARP packet.
Windows Vista/7 – If the SPA matches with an IP in its Neighbor cache table entries, it will ‘not’ go ahead and replace the old entry with the new one but changes the state of the “Neighbor Cache entry” to ‘Stale’.
Windows 8 – If the SPA matches with an IP in its Neighbor cache table entries, it will go ahead and replace the old MAC with the new MAC.
All three of these behaviors hold valid for their Server equivalents as well – Windows Server 2003, 2008/R2 and Server 2012 respectively.
Keep reading for answers..
The reason for the Windows XP to Vista/7 Change:
The reason why gratuitous ARP Send and Receive behaviors were changed in Windows Vista/7 was to eliminate the shortcomings of the previous Windows XP gratuitous ARP behavior which encourages ARP poisoning.
With the new ‘0.0.0.0’ SPA technique, gratuitous ARP works pretty well for a normal DAD scenario as the Client sends a gratuitous ARP broadcast that does not poison the ARP tables of the other machines or the ARP table of the Router interface in its same subnet. Now, if there is a defending node claiming to own this IP already, it can directly respond to this offending/problem node by sending a Unicast ARP response.
This means that there is no need for the defending node to send a new gratuitous ARP broadcast over the Network to update the clients’ ARP tables, unlike XP, as the 0.0.0.0 SPA technique would not let that happen (sending side prevention) and if the receivers are Windows Vista/7, they do not replace the old MAC with the new MAC even if the SPA contains an ‘IP address’ (Receiving side prevention).
The reason for Windows Vista/7 to Windows 8 Change:
When I got a chance to observe the new gratuitous ARP behavior in Windows 8 /2012, I realized that it was not actually a ‘new behavior’ as something told me that I have seen this same behavior somewhere else in my past experience with other Microsoft products. That is when I realized that this new gratuitous ARP behavior was introduced way before Windows 8/2012.
For comprehending what I am going to say further here in this post, it is important for all of us to understand that DAD /gratuitous ARP is not something that is used only while adding an IP address to an interface from the Adapter properties or using net shell or after getting an IP address from DHCP.
There is one more component that uses DAD/gratuitous ARP. This component is very much used in almost all production environments nowadays, mainly, for making resources highly available. It is called ‘Failover Clustering’. Note, I am not just referring to ‘Windows Failover Clusters’ but also all other products in the market that offer the same ‘resource failover’ features.
A ‘failover cluster’ case study for Windows Vista/7’s gratuitous ARP Receive behavior:
I am not a Cluster expert but with my limited understanding about how a ‘failover’ operation works and on a very surface-level explanation, this is what happens:
When an IP resource fails over from one node (Node 1) to another (Node2), it is important that the Clients and the Routers on that subnet realize that the MAC address for that IP resource has changed from MAC1 (Node1’s MAC) to MAC2 (Node2’s MAC). Any delay in this process will add to the overall failover delay.
Total Failover time = Time taken when (the IP resource unbinds itself from Node1’s network stack + actual resource failover time + IP resource binds itself to Node 2’s Network stack +Clients ARP/Neighbor Cache tables replace MAC1 with MAC2+ Time taken by the Client Application to connect back in case the application time-out period has already elapsed).
This operation is, more or less, like adding a new IP to an interface and for all that we know, the same DAD process should kick in and it obviously does.
Windows 2008/2008 R2 Windows Failover Cluster follows a different gratuitous ARP approach for DAD as opposed to a Windows Vista/7 and Windows Server 2008/2008 R2 hosts that are ‘not’ Cluster nodes.
Moving further, on a failover incident from node 1 to node2, we can get a better understanding of gratuitous ARP behavior by elucidating it with an example and assuming a few key parameters:
Let us get into the core part of this scenario – The File Share resource fails and is failed over from Node1 to Node2. (Boom!)
Gratuitous ARP Send behavior in a 2008/R2 Cluster failover scenario:
Anecdotally, it is very clear that this new behavior has taken up a new synergistic approach by combining the Windows Vista/7’s 0.0.0.0 SPA technique and the previous Windows XP’s ‘IP address in SPA’ technique.
Now, looking through this new behavior from a normal DAD’s perspective, it is a clear indication that the Windows 8/2012 (non-Cluster hosts) follows this same technique to achieve both DAD and MAC address replacement (For a Cluster failover scenario) intelligently.
Wow! Justice has been served by this new gratuitous ARP behavior for both DAD and ‘Cluster failover’ scenarios! Wait a minute, I think now comes the tricky part. What about the gratuitous ARP Receive behavior?
This new 4th gratuitous ARP packet technique would
Windows Vista/7/2008/2008R2’s gratuitous ARP Receive behavior and the hotfix:
Windows Vista/7/2008/2008R2 ‘s TCP/IP stack is designed not to update the Neighbor Cache table when it receives the 4th gratuitous ARP packet as it would contain an ‘IP address’ against the SHA field. What it does is it marks the old entry as ‘Stale’. Therefore, it would wait for 5 more seconds (Worst case) to probe the old MAC by sending out Unicast ARP requests and later discover the new MAC by sending out Broadcast ARP requests for that IP (Cluster resource IP) which would apparently contribute to the overall failover delay. An important thing to know about this ‘probe’ behavior is that it happens only when the Application layer is constantly trying to use that old MAC address and failing.
More detailed information about different states of a neighbor cache entry would be overkill for this post which you might as well have thought from the beginning.
Okay, if you are still with me and with your head is up, you would probably be thinking, “How on earth would these ‘5 seconds’ actually pose a major threat whatsoever?”
Let us look through it. Consider this situation where we have 2 Windows Server 2008 R2 nodes that are a part of a Windows Failover Cluster. These nodes are connected to a Disk Array (Storage Cluster). Now, when the Storage controller fails over from one node to the other, it sends out gratuitous ARP packet broadcasts like any other failover operation. When the Cluster nodes receive this gratuitous ARP packet, after already going through a delay due to the failover operation from the Storage controller end, the 2008 R2 nodes would further delay by waiting for 5 more seconds (roughly). In this scenario, we are actually talking about ‘High available Cluster nodes’ acting as Clients. This small delay would apparently contribute to the whole threat posed against the ‘High availability’ feature of a Cluster.
This gratuitous ARP receive problem/issue in Windows Vista/7/Server 2008/2008 R2 has already been addressed and fixed by Microsoft with a hotfix described in the following article: http://support.microsoft.com/kb/2582281
After installing this hotfix and rebooting the Vista/7 Client, it would be able to replace the old MAC with the new MAC whenever it receives the gratuitous ARP broadcast packet on the fly.
Lab setup for ‘Cluster Failover’ for understanding Windows Vista/7 gratuitous ARP Receive behavior
Windows Server 2008 or Windows Server 2008 R2 Cluster (2 Nodes) :
We are going to simulate a failover from Node 1 to Node2 and check when the ARP/neighbor cache entries are getting changed for the resource IP 10.0.0.6 from 00-15-5D-50-AD-09 to 00-15-5D-50-AD-0A
For doing this on a finer detail, we can use a PowerShell script to continuously monitor the Clients’ ARP and Neighbor cache tables every second. In this way, we will be able to see the exact point in time when the entry gets changed from the old MAC to the new one and then compare it with the exact time when the gratuitous ARP packets hit those Clients.
For Windows 7 and Windows 8 Clients:
For the Windows XP Client:
Analyzing the captures and the outputs:
When the Cluster resource fails over from Node1 to Node2, it will release the IP from Node 1’s network stack and adds it to Node 2’s network stack.
When it binds the new IP to node 2’s stack, it sends out gratuitous ARP broadcast packets as an implementation of DAD and new MAC announcement. From the network capture, we see that from 11:01:30 to 11:01:32, there are 3 gratuitous ARP packets that reach all the Clients at the same time. The 4th gratuitous ARP broadcast packet which has the SHA as 10.0.0.6 reaches the Clients at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows XP:
The gratuitous ARP packet with SPA field set to 10.0.0.6 reaches the Client at 11:01:33.
The ARP entry is changed to the MAC address of node 2 exactly at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows 7 (without KB 2582281)
The 4th gratuitous ARP packet reaches the Windows 7 Client at 11:01:33
The Neighbor cache entry does not change even after receiving the 4th gratuitous ARP packet at 11:01:33 and still contains Node 1’ MAC. It is just marked ‘Stale’. Since we did not have any application running in the background and using this Neighbor Cache entry, the entry turned to a ‘Stale’ state. If you like to see the state change from ‘reachable’ to ‘stale’, you need to follow this same procedure with a continuous ping in the background from this Windows 7 Client.
Illustration of gratuitous ARP Receive behavior of Windows 7 (with KB 2582281)
The Neighbor cache entry changes from node1’s MAC to node 2’s MAC exactly at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows 8 / 2012
The 4th gratuitous ARP packet reaches the Windows 7 Client at 11:01:33.
The Neighbor cache entry changes from node1’s MAC to node 2’s MAC exactly at 11:01:
Windows Vista/7/2008/2008 R2’s gratuitous ARP Send behavior
The new Windows 8/2012 gratuitous ARP behavior and the Windows Vista/7/2008/2008R2 hotfix for gratuitous ARP Receive behavior tackle the DAD and the ‘Cluster failover’ scenarios efficiently.
Pushing further, let us take a completely different scenario highlighting the problem with “Windows Vista/7’s gratuitous ARP Send behavior”.
This is the setup:
I have a machine A in 10.x.x.x subnet which has a subnet mask of 255.0.0.0. There is another machine X in 11.x.x.x subnet which has a subnet mask of 255.0.0.0. There is a Gateway which acts an internetwork for 2 subnets – 10.x.x.x and 11.x.x.x
IP address of A – 10.1.1.1
Gateway’s IP for 10.x.x.x interface – 10.10.10.10
Gateway’s IP for 11.x.x.x interface – 18.104.22.168
Machine X pings Machine A for the first time:
Machine X would know that 10.1.1.1 does not belong to its subnet range and therefore would send the ICMP Echo Request packet to its Gateway. The Gateway would then send an ARP broadcast for that IP (if it does not have its MAC in its ARP table already) and when it gets a response from Machine A, it would cache that MAC (MAC1) for that 10.1.1.1 in its ARP table.
Then, the Gateway would send the ICMP Request packet to Machine A’s MAC (MAC1) which would respond by sending out an ICMP Reply packet. The Gateway forwards that packet to Machine X’s MAC and thus, ping is successful here.
Removing 10.1.1.1 from Machine A and assigning it to Machine B on the same subnet:
Now, if I remove this IP (10.1.1.1) from machine A’s interface and assign it to Machine B on the same 10.x.x.x subnet, the Windows Vista/7’s gratuitous ARP mechanism would not let its Neighbors, including the Router’s 10.x.x.x interface, know about the fact that 10.1.1.1 is now an IP address that should point to MAC2 and not MAC1.
Windows Clients within the same subnet would be able to recover from this state and contact the new MAC after sometime depending upon the ARP cache timeout in Windows XP Clients (2-10 minutes) and the ‘5 second Stale-Probe-New ARP broadcast’ phase in Windows Vista/7 Clients.
However, in most of the Customer environments, the Gateways (Routers) would be configured with an ARP cache time out specified in ‘hours’ (The maximum that I have seen is 2 hours).
The 0.0.0.0 SPA technique for gratuitous ARP/DAD that is adopted by Vista/7/2008/2008 R2 would not let the Router know about this change. This impacts cross –subnet clients as they would not be able to contact 10.1.1.1 for at least 2 hours unless we delete that old ARP entry from the Router.
If you are assigning an IP address that was used by a different machine to a Windows Vista/7/2008/R2 machine, you need to make sure that you delete the old ARP entry manually from the Gateway/Router.
Thanks for reading patiently! I hope that this post has answered some of your questions related to differences in gratuitous ARP/DAD behavior between Windows XP, Vista/7 and Windows 8 and other ‘failover delay’ woes. Happy gratuitous ARPing!
- Sai Balaji
Great Article :)