We often have customers who want to better understand the resolution to their networking support issue and why what we did fixed their issue. Depending on the issue, this explanation may be quite involved and complex. This would be like asking the pilot on the way off the airplane to quickly and briefly explain the aerodynamics, electronics, hydraulics etc that went into landing the plane and how they all work together. Since there is no brief explanation of the in-depth workings of how computer networks do all the things they do I will be covering this topic in a series of blog posts in which I hope give at least a basic understanding of network protocols and architecture. My intention here is to lay a foundation from which a deeper understanding of Windows networking can be gained. We will discuss IPv4 over Ethernet only, and will not be delving into IPv6 or other physical network topologies.
In order to continue we need to at least mention the 7 Layers of the International Standards Organization/Open System Interconnect (OSI) model. This is the model where we get the reference for Layer 2 and Layer 3 when referring to types of switches and routers, for example.
This model consists of the following seven layers:
Layer 1 is the Physical layer and this consists of the Network Interface Card (NIC) and other components that allow a system to physically and logically connect to a network. This is as deep as we need to go into Layer 1 for this discussion.
We will focus more on Layer 2 in this blog post, specifically starting with Layer 2 routing, and then get into Layer 3 routing in the next blog entry.
For more information on the OSI model and Window Network Architecture see the following:
Windows Network Architecture and the OSI Model http://msdn.microsoft.com/en-us/library/aa938287.aspx
TCP/IP Architecture http://technet.microsoft.com/en-us/library/cc751234.aspx
The first thing we will need in order to communicate on a computer network is some method for structuring what is being sent. This will be important not only for the computer itself but also allows other devices on the network such as routers and switches to be able to properly handle network traffic. The two standards we are using to accomplish this for TCP/IP, are Ethernet II and IEEE 802.3, and they are found in Layer 2 of the OSI model. These standards define what is included when data is "framed" to be sent so, data sent on a network will often be referred to as frames. For more in-depth discussion on how this is structured and the inner workings of Windows networking the following books are excellent references:
"Microsoft® Windows® Server 2003 TCP/IP Protocols and Services Technical Reference" http://www.microsoft.com/learning/en/us/books/5030.aspx
"Windows Server® 2008 TCP/IP Protocols and Services" http://www.microsoft.com/learning/en/us/Books/11630.aspx
Now that we have a standard for constructing a frame to put on the wire, we will need a way to determine where we are going to send the frame. In order to communicate with other computers on the network a system or "Node" must have a way of identifying itself and other systems within the local subnet or "Broadcast Domain", a Broadcast Domain being the network that is reachable by broadcast. This identification is done using the MAC address of the network adapter. A MAC address may also be referred to as an Ethernet Address or Physical Address. This address is assigned by the manufacturer at the time the network adapter, also known as the Network Interface Controller (NIC), is created. It is possible to find NICs that allow the MAC address to be changed manually but care should be taken in doing this as this could cause addressing problems on the local subnet.
A MAC address consists of 6 Bytes, the first 3 of which are used for the Organizationally Unique Identifier (OUI) which is unique to the manufacturer of the NIC.
You can determine the manufacturer of your NIC by running an IPConfig /all on your system from a command prompt. Next take the first 3 bytes of the Physical Address and plug them into the "Search For" under "Search the public OUI listing" at the following link:
The last 3 bytes of the MAC are specific to the NIC. Together they provide a unique local address.
As I mentioned, this gives a system a way to identify itself and other systems on the local network. However, for this to be useful we need a way to discover what the MAC address of other systems are, as well as to allow that address to be discovered by other systems. To this end, Address Resolution Protocol (ARP) was developed, and is described in RFC 826.
So that sounds pretty good, you might say. I have my MAC address. I'm all set to talk on the network. You would be mostly correct. From a layer 2 stand point you do have everything you need to communicate, however, most operating systems, including the Windows operating systems, allow communication to take place with systems beyond the Broadcast Domain. For this reason we have Layer 3, the Internet Protocol Layer, where we have Internet Protocol (IP) addresses. This is significant because the system cannot just always assume that traffic will be on the local subnet. The concept I want you to grasp here is that MAC addresses are for "local" routing and IP addresses are for "global" routing. This is a bit over simplified but let's run with this for now and it will all get clearer as we get farther into IP routing. For now, we just need to know that each system will have both an IP address as well as a MAC address. In order to communicate with other systems, we will need a way to match the IP address of the system we want to communicate with to its MAC address. In addition, the source system will want to share its own MAC and IP address so that the target node will know how to communicate back. In order to accomplish this matching of MAC to IP addresses we use Address Resolution Protocol (ARP).
You will notice as we discuss ARP that the requests are structured a certain way. This is because we are conforming to standards such as RFC 826 which defines ARP. ARP is used to resolve the next hop IP address of a node to its corresponding MAC address. This is significant because the next hop IP address is not necessarily the destination IP address. Remember the concept I mentioned earlier that the IP address can allow for global routing.
When looking at the ARP in a network capture you will see that there are four fields used to identify the source and target IP and MAC address.
In the ARP Request the fields are filled in as follows:
When the broadcast ARP request is received on Node 2, Node 2 updates its ARP cache with the information it received in the request. In the ARP reply, notice that the SHA and SPA are updated to match the correct information for the sending system. This is the information used by Node 1 to update its ARP cache. Once this is done, both systems will have the MAC and IP information it needs to communicate with the other node.
Once an address gets put in the ARP cache it is maintained for a set amount of time. The default behavior is a two minute timer that is reset every time the destination MAC is used for a total of 10 minutes. After 10 minutes of use the destination MAC address is discarded and must be resolved again with a new ARP request. If after two minutes the destination MAC has not been used it is discarded.
It is important to remember that in order for ARP to work, the requester must already have a destination IP address that it will request the MAC address for. The IP address for the destination may be entered manually or may be discovered through name resolution.
Note that starting with Windows Vista we no longer refer the cache as ARP cache, we will discuss this further, later in the this blog post.
ARP is also used to detect IP address conflicts. Address conflict detection is used to insure that a system that is brought up on the network or that is assigned a new IP address does not have an address that conflicts with a system already on the network.
In address conflict detection, we use what is known as a Gratuitous ARP. When a system is configured with an IP address either manually or by DHCP it will send a Gratuitous ARP to insure that another node on the network is not already configured with this IP address. In the case of a conflict the two nodes are defined as follows. The Offending Node is the node that is sending the gratuitous ARP, and the Defending Node is a system already configured with the IP Address in question. The contents of this request and how this affects the ARP cache on other systems on the network differs depending on the OS.
In Windows XP and Windows Server 2003 the Gratuitous ARP request is sent with the Senders MAC filled in with the MAC of the sending system and the Target MAC set to 0's, but the Senders and Target IP address are both set to the address of the sending system. If a conflict is detected then the defending system replies with its IP and MAC address.
The problem with this method is that all the nodes that receive this broadcast and have an ARP cache entry for this IP address will update their ARP cache with invalid data. So the defending node will now need to send its own Gratuitous ARP to correct the cache on the other systems on the network. Because of this, starting with Windows Vista the Gratuitous ARP is handled differently.
In Windows Vista and Windows Server 2008, ARP Cache is now known as Neighbor Cache. The ARP -a command will still display the legacy ARP Cache and we can still add static ARP entries.
The contents of the neighbor cache can be displayed with the following netsh command.
netsh interface ipv4 show neighbors
When this command is run you will notice that we have different states for neighbors. The following states are possible:
In Windows Vista and Windows Server 2008 there are some built in protections that reduce the chance of the Neighbor cache getting updated with incorrect information. This also helps keep the requesting system from incorrectly updating other systems.
First, a Windows Vista or Windows Server 2008 will not update the Neighbor cache if an ARP broadcast is received unless it is part of a broadcast ARP request for the receiver. What this means is that when a gratuitous ARP is sent on a network with Windows Vista and Widows Server 2008, these systems will not update their cache with incorrect information if there is an IP address conflict.
Additionally, when a gratuitous ARP is sent by a Windows Vista or Windows Server 2008, the following change has been made – the SPA field in the initial request is set to 0.0.0.0. This way the ARP or neighbor caches of systems receiving this request are not updated. So, if there is a duplicate IP address, the receivers do not need to have their cache corrected.
There will be times when a system needs to resolve the MAC address of a system that is not reachable within the Broadcast Domain. When this happens, we can use another device on the network to answer the ARP request, this is known as Proxy ARP. Proxy ARP is the answering of ARP Requests on behalf of another system. One example of this is when a Remote client connects to Windows Routing and Remote Access (RRAS) server. When the client connects to a RAS server it is assigned an IP address from the server and the server keeps track of which client was assigned the IP address. When clients on the internal network and remote clients attempt to communicate with each other the RAS server will use Proxy ARP to reply with its own MAC address. As far as the client sending the ARP request is concerned it has successfully resolved the IP to the MAC of the remote client. In the example, the LAN client is sending an ARP request for the IP of the Remote Access Client. Notice that the ARP reply comes from the RAS server using its own MAC.
Next time we will discuss IP routing and get deeper into IP addresses.
- Clark Satter
PingBack from http://itknowledgeexchange.techtarget.com/it-trenches/did-you-see-this-tcpip-networking-from-the-wire-up/
So is there any way of stopping Vista/2008 from ignoring GARPs?
That Microsoft has decided to not respond the proper way to a Gracious ARP packet on the network probably breaks a lot of network environments where GARP packets are being used to force a new mac address of a router / HA cluster etc into the MAC table of all clients on the network.
I'm really wandering how long it will take until Microsoft sees that they really make a big mistake here.
To name a few, Citrix Netscales use GARP during failover to make sure all traffic is going to the new primary system. Keepalived on Linux environments use it during failover. And these are 2 I know of right now, there are probebly a lot more that do exactly the same.
OK fine, the default GARP behaviour on W2K8 has change, but is there any programmatic way to explicitly send or force a GARP?
We use the IP Helper API to associated a virtual IP address with an adapter. This allows us to move a "floating" IP between several computers. On XP and W2K3, adding the virtual IP via the helper API automatically sent a GARP; but on W2K8 it doesn't.
Our problem is that we have a router between two domains; it used to update its ARP table in response to the GARP. With no GARP being sent automatically, it takes hours before clients on the other side of the router can see that the IP has moved.
What we are trying to do is exactly the same thing as what MS SQL Server clustering does. Except that - even on W2K8 - the clustering s/w *always* GARPs when failing the cluster IP over. Arg! How are you doing that?
They are beautiful, thank you expressions that you share with us for such things
In my experience while the receiving Windows Vista+ host will not use the information in a received gratuitous ARP (or any received unsolicited ARP) to update its ARP / neighbor cache (IPv6 semantics / terminology is used in Vista+) it will change the neighbor cache entry from 'REACHABLE' to 'STALE' so the next time the Windows host wants to communicate with the IP that has moved it will re-ARP for it. I have seen this not be the case, but I think an update may have enabled this I am not certain. This behavior seems a decent middle ground between ARP spoofing protection and utilizing GARP updates, but I have not seen it documented anywhere I have only seen it happen.
"What this means is that when a gratuitous ARP is sent on a network with Windows Vista and Widows Server 2008, these systems will not update their cache with incorrect information if there is an IP address conflict."
is disingenuous as it also (and arguably more importantly) prevents legitimate GARPs from achieving their purpose. As backed up by other comments in this post, the fact that this is so poorly documented and that it apparently cannot be configured puts Windows clients at a disadvantage when dealing with a HA resource that uses GARPs to move IPs around as the Windows host will initially try to update a STALE ARP / neighbor cache entry with a directed ARP (directed at the old / stale MAC) which if it isn't down will still probably not respond as the IP has likely moved. Windows will send 3 of these directed ARPs by default at approximately one second intervals before broadcasting the request. So while Mac/Linux/Pre Vista clients will automatically become aware of the HA resources new IP-MAC mapping with the GARP, Vista+ will take ~3 seconds plus a broadcast RTT.
Is the diagram under the Duplicate Address Detection section of this post incorrect?
Shouldn't the TPA for the ARP request of the Offending node be 10.10.0.10?
Would be good to see a Microsoft opinion how to update the caches with *CORRECT* information. A lot of people have expressed why that is necessary. So far I've heard only of two rather awkward workarounds: use a third machine on the subnet, clear its ARP cache and re-query (using ping, for example).
Or the weird suggestion to have everyone write its own NDIS drivers to send out the correct GARP at an IP address move. I can imagine how this works when many HA solutions run on the same box - either a dozen of these drivers jam the system or the next firewall cuts it off because of ARP spamming ;-)
Honestly, what would be needed is a tool that sends out a GARP with the new SPA.