John Howard - Senior Program Manager in the Hyper-V team at Microsoft

Senior Program Manager, Hyper-V team, Windows Core Operating System Division.

Hyper-V: MAC Address allocation and apparent network issues MAC collisions can cause

Hyper-V: MAC Address allocation and apparent network issues MAC collisions can cause

  • Comments 30
  • Likes

In a physical only world, you don’t usually have to worry about MAC addresses that much as each NIC vendor carves off a MAC address from their ranges which have been allocated to them. However, in a virtual environment, you have to be a little more careful, particularly if you are using dynamic MAC address assignment. This post looks at how Hyper-V allocates dynamic MAC addresses and some potential problems you can face. So often it can be the last thing people think to check, but can be the root cause of otherwise unexplained network oddities. 

Here’s a screenshot of a typical MAC collision problem – pings sometimes work, sometimes fail – and this is all on a local isolated network.

1 

To start the walkthrough, I have a base install of Windows Server 2008 on a server  with a single physical NIC – against best practice, but it serves fine for demonstration.  I have already installed the RTM update (KB950050) to the server, but have not yet added the Hyper-V role. Let’s look at an output of “ipconfig /all”. You can see that the MAC address of the physical NIC is 00-13-20-F5-F8-7D and I’m obtaining an IP address from a DHCP server on the private test network I’m using.

2

Now let’s use Server Manager to enable the Hyper-V role. Note that Server Manager allows you to create an external virtual network switch during role enabling, but I am choosing not to do this. Let’s see what has happened in the registry after the Hyper-V role is enabled. Specifically, I’m looking at two keys which have been created under HKLM\Software\Microsoft\Windows\NT\CurrentVersion\Virtualization, as-yet unpopulated: MinimumMacAddress and MaximumMacAddress, plus another key in the worker node, CurrentMacAddress – again as-yet unpopulated. (The astute walking through this in front of a machine will notice that CurrentMacAddress also appears in the Virtualization node. That key is not used though.)

3

4

Next, I’m going to create my first virtual machine. As I haven’t created any virtual network switches yet, I’ll leave the network disconnected. I don’t need a hard disk. Also, I’m deliberately choosing not to start it.  Let’s see what’s happened in the registry. MinimumMacAddress and MaximumMacAddress have been populated with 00-15-5d-c8-6a-00 and 00-15-5d-c8-6a-ff respectively – a range of 256 possible MAC addresses.

5

So where did this range come from? The first three bytes are the Microsoft IEEE Organizationally Unique Identifier, 00-15-5D which we use in Hyper-V. The next two bytes, C8-6A are derived from the lowest two octects of an IPv4 address on the server (the first IP address as NICs are enumerated). If you look at the second screenshot in this post, the IPv4 address on the only NIC on this server was 192.168.200.106. In Hex, this is “C0.A8.C8.6A”. The last two octets or bytes are C8 and 6A. The last byte of the address range is automatically generated with a minimum 00 and maximum FF.

You can probably now realize, that while this algorithm will work for many people, it may not necessarily be perfect and cause MAC address range clashes. To cope with multiple Hyper-V enabled servers, you would need to ensure address ranges are managed at a higher level across those servers, such as the use of SCVMM.

Let’s go back to the virtual machine I created. By default, when a virtual machine is created, it is allocated a dynamic MAC address. This can of course be changed in the settings for the virtual machine. Here’s the setting for the blank virtual machine. Notice that it’s set to Dynamic and the MAC address in the “Static” boxes show 00-00-00-00-00-00

6

Now I’m going to start the Virtual Machine and open the settings. Although some settings cannot be changed while a virtual machine is running (including changing static/dynamic MAC, or the static MAC itself), notice that the boxes under the static MAC address radio button are now populated with the first MAC address in the range defined in the registry: 00-15-5D-C8-6A-00.

7

Now for a bit of fun (and to make the walkthrough a bit simpler), let’s change the registry so that the maximum MAC address is 00-15-5D-C8-6A-02. (I’ll also do a reboot just to make sure the change takes effect) This change means that we are limited to three possible dynamically assigned MAC addresses, the last octet being 00 (in use by the “Blank” VM), 01 or 02.

8

Now, I’m going to create another virtual machine named 6A-01 and power it on, then create a third virtual machine named 6A-02 and power that on too.  Let’s look at the settings for each of these while all three virtual machines are running. As expected 6A-01 has a MAC address ending 6A-01 and 6A-02 has a MAC address ending 6A-02. That’s why we have the “CurrentMacAddress” registry key to track what MAC address to assign to VMs in turn.

 

9

10

Can you guess though at this point what would happen though if I create another virtual machine and power it on? I don’t have any MAC addresses left in my available range and all MAC addresses are currently in use.

11

Did you guess correctly? Let’s now power off the very first virtual machine (“Blank”) I created with MAC address 6A-00, and then try to run through the New Virtual Machine Wizard again with my “No MAC Addresses Available In Range” virtual machine. Try to guess what will happen at the end.

12

The virtual machine starts successfully and now has a duplicate MAC address to the first virtual machine I created, ‘Blank’:

13

Last quiz question: What would happen then if I tried to start “Blank” – will it start or not? After all, it has already been allocated a MAC address ending 6A-00.

14

Actually, we will detect this as you can see above and stop the virtual machine from powering on. So in some ways, on a single Hyper-V enabled server, we’re relatively immune to duplicate MAC addresses across virtual machines running on a single server. However, due to the algorithm for choosing the ranges of MAC addresses, while relatively safe, there is no guarantee of being unique across an entire network. And of course, chances are that you will want packets from or to virtual machines on a Hyper-V server to “hit” the physical network.

So hopefully that gives you a better idea why it is important to manage MAC addresses across multiple servers in a virtual machine environment. While the walkthrough above was specific to Hyper-V, the same types of issues could arise in Virtual Server.

Cheers,
John.

Comments
  • John, so if Hyper-V is immune to MAC duplication, how did you get the timeouts on MAC collision ping in the very first screenshot?

  • Polk - yes, good question and one which I didn't make clear above. I was using a second server running the Hyper-V role with a virtual machine on it configured with a duplicate MAC.

    Thanks,

    John.

  • So one question I cant seem to get answered- If we leave a Hyper-V guest network as dynamic MAC- will it ever change if that system is rebooted or shutdown, etc?

    I have had issues with two Hyper-V servers on the same LAN with several guests a piece and I have had broadcast storms and strange issues that have yet to be explained but when I run an arp -a command all the host and guest have different MACs.

    I was going to restart the hosts and see if these MACs remain the same or not.

  • Omar - no, the dynamic MAC address once allocated will not change, so if all VMs have unique MAC addresses, I wouldn't expect to see what you are seeing. Is it possible that there are other Hyper-V boxes elsewhere on the network which are conflicting?

    Thanks,

    John.

  • Here is any idea, MS design a product that doesn't have these limitations.  You don't have to worry about MAC duplication in a VMWare ESX environment.  This is just typical MS "features" that are only half baked, but the MS bigots can see no faults.

  • Craig

    Thank you for your comment. Unfortunately, without a higher management layer spanning multiple ESX servers (eg Virtual Center server), ESX suffers from exactly the same problem. Of course, feel free to correct me if you believe me to be incorrect. I am more than open to a discussion - if you want to take this offline, you can email me directly using the email link at the top of my blog.

    The reasons I believe you are incorrect are based on documentation from both VMWare, Cisco plus a thread on the VMWare communities forum:

    VMWare Infrastucture 3 Online library - ESX Server 3 Edition (http://pubs.vmware.com/vi35/wwhelp/wwhimpl/js/html/wwhelp.htm)

    - Expand nodes out through ESX Server 3 Configuration Guide

    - Advanced Networking

     - Setting up MAC Addresses

      - MAC Address Generation.

    The particular part is at the end of fourth paragraph: "The algorithm puts a limit on the number of running and suspended virtual machines at any one time on any given server. It also does not handle all cases when virtual machines on distinct physical machines share a subnet."

    The full section:

    MAC Addresses Generation

    Each virtual network adapter in a virtual machine is assigned its own unique MAC address. A MAC address is a six-byte number. Each network adapter manufacturer is assigned a unique three-byte prefix called an OUI (Organizationally Unique Identifier) that it can use to generate unique MAC addresses.

    VMware has the following OUIs:

    • One for generated MAC addresses.

    • One for manually set MAC addresses.

    • One that was previously used for legacy virtual machines, but is no longer used with ESX Server 3.

    The first three bytes of the MAC address that is generated for each virtual network adapter are comprised of the OUI. This MAC address-generation algorithm produces the other three bytes. The algorithm guarantees unique MAC addresses within a machine and attempts to provide unique MAC addresses across machines.

    The network adapters for each virtual machine on the same subnet should have unique MAC addresses. Otherwise, they can behave unpredictably. The algorithm puts a limit on the number of running and suspended virtual machines at any one time on any given server. It also does not handle all cases when virtual machines on distinct physical machines share a subnet.

    The VMware Universally Unique Identifier (UUID) generates MAC addresses that are checked for any conflicts. The generated MAC addresses are created by using three parts: the VMware OUI, the SMBIOS UUID for the physical ESX Server 3 machine, and a hash based on the name of the entity that the MAC address is being generated for.

    After the MAC address is generated, it does not change unless the virtual machine is moved to a different location, for example, to a different path on the same server. The MAC address in the configuration file of the virtual machine is saved. All MAC addresses that have been assigned to network adapters of running and suspended virtual machines on a given physical powered-off virtual machine is not checked against those of running or suspended virtual machines. It is possible but unlikely that when a virtual machine is powered on again, it can acquire a different MAC address. This acquisition is due to a conflict with a virtual machine that was powered on when this virtual machine was powered off.

    VMWare Infrastructure 3 in a Cisco Network Environment http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/vmware/VMware.html#wp696203

    Without quoting the whole section, the relevant part under vNIC MAC Addresses, Bootup, VMotion Migration states "The vNIC MAC addresses include the Organization Unique Identifiers (OUI) assigned by IEEE to VMware. The ESX host and the configuration filename information is used to create a vNIC MAC address. The OUIs used by VMware are 00-50-56 and 00-0c-29. The algorithm used to generate the MAC address reduces the chances of a MAC address collision, although the process cannot guarantee a MAC address is unique"

    Finally, on the VMWare communities site http://communities.vmware.com/thread/147630 this thread indicates how you configure a higher layer management server spanning multiple servers (Virtual Centre Server) using vpx generated MAC addresses.

    This is the reason I mentioned in my article why managing MAC addresses spanning multiple virtualization servers (eg using SCVMM 2007 for Hyper-V/Virtual Server and ESX, Virtual Centre Server for ESX) is important.

    Cheers,
    John.

  • So is it possible for the host machine to be a kind of "NAT" box, therefore the guest OS uses the physical NIC MAC address instead of it's own dynamically created one?

  • Bryan - No, this is is not possible.

    Thanks,

    John.

  • Thanks for the article John,  I now understand what has been happening with the MAC address generation on my sysprep cloned hyper-v servers.

    It seems that these registry entries were just copied from the template system, the MAC addresses as a result were all based on the template systems original IP address and were starting to come out with the same MAC addresses on different clones.

    It looks like hyper-v doesn't currently check dynamically whether these max and min entries still match the current machine address.

  • Chris - yes, that would be correct. We don't have a sysprep provider implemented in Hyper-V v1 (2008), so you would have clashing ranges in each of the clones.

    Thanks,

    John.

  • When connecting two physical machines with hyper-V for a demo (ie one network card shared for physical and virtual machine), ipconfig/all say that the two notebooks have the same MAC Address for physical computer.

    To be sure, we have made a reservation in DHCP. When the first boot, it takes the reserved IP address. When the second boot, it take the same ip address....

    Do you know this problem ? How can we fix it ?

  • Isambert - I would not expect two physical computers to have the same MAC address - very unlikely. I can think of one situation where it is possible due to a bug in Hyper-V for the virtual NIC in the parent partition to take on a MAC address from the internal pool MACs for VMs that we maintain.

    Can you confirm the MAC addresses which are clashing start 00-15-5D (the default range for our pool); that these are virtual NICs rather than physical nics in the parent (ipconfig /all will list the adapter as Microsoft Virtual Network Switch Adapter #n); the steps in Hyper-V Manager you went through to create the external virtual network switch (I suspect you created an internal virtual network, then changed it to external).

    Thanks,

    John.

  • I am experiencing this issue within our Hyper-V Cluster (4 servers) running about 25 VM's.  When we had the network down for maint, the cluster shut down and saved the VM's state.  When the network was restored, we experienced intermittent issues with access to VM's and even the Hyper-V Parent Partition.  We have MAC management set to automatic.  The VM's primarily use the same subnet.  

    It appears that based on the above comments, we should be manually managing the MACs (BUMMER! )

  • John-

    Thanks for the article.

    Also, can you let me know or point to some article which talk about where are these MAC address for VMs stored in WMI of Hyper V server.

  • Sachin - take a look at the WMI documentation for msvm_virtualswitchmanagementservice http://msdn.microsoft.com/en-us/library/cc136938(VS.85).aspx.

    Specifically the CreateInternalEthernetPort* methods.

    Thanks,

    John.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment