Welcome to the third installment of Network Virtualization in Virtual Machine Manager! This series will be an example-based walkthrough consisting of a few parts:
In this post, I’ll be introducing a network virtualization gateway. As previously discussed, VM networks that use network virtualization will be completely isolated and do not have external communication. We’ll be fixing that in this post. It is possible to create a VM running a software router and give it two NICs, attaching one to the VM network and one to the external network. However, this scenario is not supported by Microsoft, does not provide any integration with VMM, and can only be used by a single tenant.
First, we’ll need a cluster that sits on the edge of our network to house the VMs that will act as a gateway. I created a cluster named EDGECLUSTER running Windows Server 2012 R2 with a Server Core installation and it has two nodes named CON-HyperV8 and CON-HyperV9. The cluster has a witness disk and a sufficiently large CSV. Moving forward, I’ll assume you’ve done the same, as I won’t be discussing cluster creation in this series.
I’ve already added my EDGECLUSTER to VMM just as in the previous posts. In PowerShell:
Refer to my previous posts if you need GUI instructions.
Before beginning configuration, we should discuss how a network virtualization gateway functions. A network virtualization gateway is a specific VM that will act as a Layer3 device to allow traffic flow to an external network.
The gateway itself is controlled by VMM from the management network and will forward traffic from an NVGRE network to an external network. As such, the gateway requires connectivity to all three networks.
The gateway can function as a NAT device, a forwarding device, and can provide a site-to-site VPN tunnel. In this post, I’ll only be discussing deploying the gateway as a NAT device to provide external connectivity to tenants. When I discuss hybrid cloud connectivity in a future post, I’ll cover the other capabilities.
At this point, my physical networking topology looks like this:
Each host has two NICs that are assigned as trunk ports in the switch. I want my logical topology to look like this:
Management Network (Thin Solid Line)
Connects to all hosts and is used for VMM communication.
Tenant Network(s) (Dashed Line)
These are the networks we created in the previous post (we actually only created one) to isolate tenants using VLANs. These networks are not needed on the management cluster or the virtualization cluster.
Virtualization Network(s) (Dotted Line)
These networks exist as a logical entity through network virtualization. Each network is segregated but uses the same VLAN for transport. From a Layer-2 perspective, there is only a single network and the segregation is enforced higher in the network stack. In this post, we’ll be adding external connectivity to this network.
External Network (Thick Solid Line)
This network is the connection to another network for the virtualized traffic. Although I’m using an external network, this network could also be something else, like a corporate network.
In the diagram above, notice that the EDGECLUSTER has three networks. The network virtualization gateways will reside on these hosts and will connect to all three of these networks. But first, we’ll need to create the networking objects in VMM. The host clusters in my VMM server look like this:
All of my VMs that perform management (AD, File Cluster, VMM Cluster, and SQL Cluster) reside on my MGMTCLUSTER. My HOSTCLUSTER has tenant VMs whose networks are segregated using VLANs, my VIRTUALCLUSTER has tenant VMs segregated via network virtualization, and my EDGECLUSTER is currently not configured, as it’s just been added.
Of the three networks on my EDGECLUSTER, two already exist: the virtualization network and the management network. We’ll want to create the third network for external traffic and create a logical switch whose associated uplink port profile connect to all three networks. So, let’s begin.
First, the external network. In my environment, this is VLAN 801, so I’ll configure that. As before, start the Create Logical Network Wizard. Name the network External.
Next, create the network site and apply it to the Edge group, and provide the IP information.
Finish the wizard to create the Logical Network. In PowerShell:
I’ll need to create my Uplink Port Profile next, but I first need to update the host group for the Virtualization network. When we configured the Virtualization Logical Network, we only assigned its network site to the Virtualization host group. To add the Edge group, simply bring up the properties of the Virtualization network.
On the Network Site tab, select the Edge network and click OK.
Now the Virtualization Logical Network can be associated with a host in the Edge group via an Uplink Port Profile. We didn’t have to do this step for the Management network because its network site is set to the All Hosts group. In PowerShell:
Now to create the Uplink Port Profile. Start the Create Hyper-V Port Profile wizard as in previous posts and name it Edge, choosing Uplink port profile as the type.
For the Network Configuration, check the boxes next to the three networks that the EDGECLUSTER needs: External, Management, and Virtualization.
Finish the wizard to create the Uplink Port Profile. In PowerShell:
Now we create the Logical Switch called Edge and assign it the Edge Uplink Port Profile. This is the same as previous posts, so I won’t walk through the steps. In PowerShell:
Next, we’ll create the VM Network. There is nothing special about this in the GUI. Just create a new VM Network named External and assign it to the External Logical Network. In PowerShell:
Lastly, we need to add the Logical Switch to the hosts. I’ve covered the GUI for these steps before in previous posts. I’ll defer the PowerShell for a moment, as there is something else that needs to be done.
One last configuration step is to enable the hosts in my EDGECLUSTER to be dedicated as gateway hosts. To do this, bring up the properties of the host and find the Host Access tab. Here, you’ll find a checkbox for This host is a dedicated network virtualization gateway…. that you’ll need to check:
This will need to be done on all the hosts in EDGECLUSTER. In PowerShell, we’ll do this while adding the Logical Switch to the host:
Notice the Set-SCVMHost cmdlet includes the parameter –IsDedicatedToNetworkVirtualizationGateway that we set to true.
That’s all for host configuration. Now on to deploying the gateway.
The System Center Virtual Machine Manager product team has provided a service template for deploying a network virtualization gateway. You can download the ZIP here.
The ZIP contains a document that outlines the process, but I’ll be going through it here as well. First, you’ll need to have a sysprep’d disk image in your VMM Library containing Windows Server 2012 R2 (I won’t be covering this step). You’ll also need two empty disk images in your library for the cluster disks. Lastly, you’ll need to import the custom resource included with the template. Once you have that, you can import the template.
First, create the disks. This is simplest in PowerShell via a command like:
Next, you’ll need to import the resources. To import, in the Library pane, click the Import Physical Resource button on the Home ribbon.
In the dialog, click the Add custom resource… button to browse to the folder called VMClusterSetup.cr that was included with the ZIP. Do this again for each disk that was earlier created, this time clicking Add resource…. Next, click the Browse… button to identify the VMM Library location where the resource will be copied. Once that’s all populated, click Import.
In PowerShell, this would be something like (your paths will vary):
Note that, depending on your certificate configuration, you may have to check the Use unencrypted transfer checkbox. In PowerShell, this is the flag –AllowUnencryptedTransfer for each command. Once the resource is imported, you can import the template.
To import a template in VMM, in the Library pane, click the Import Template button on the Home ribbon.
In the wizard, click Browse… and find the extracted location. Select the Windows Server 2012 HA Gateway 3NIC.xml file and click Open and then Next. On the next page, you’ll identify the specific resource mappings. For any mapping, you can click the pencil icon to select the appropriate resource in the Library. Once finished, click Next.
Finish the wizard to import the template. The PowerShell equivalent of this can be a bit tricky. You’ll have to create a template package, create a package mapping, add each resource to the mapping, and then call the Import-SCTemplate on that package object and its mapping. The PowerShell below is a loose guidelines that you might have to tweak for your environment:
Once imported, we can deploy the template.
There are few requirements for deploying the gateway template. Each of these requirements reflect the supported configuration for the gateway that was tested by the product team. I’ll speak to possible workarounds for these requirements, but I won’t go through their configuration details.
First, the NIC that connects to the external network is configured to use a static IP. As such, you’ll need a static IP pool in VMM. If you wish to use DHCP, you’ll need to edit the template and change this.
Next, the template uses shared storage but is not configured as highly available (HA). Since VMs that are not HA cannot use CSV for shared storage, you’ll need to configure a Scale-Out File Server. The reasoning behind not making this template HA was likely to disallow automatic VM failover. As an alternative, you could mark this template as HA and configure the available nodes on the cluster role for the VM after deployment.
Lastly, the storage classification for the OS drive for the template is marked as Local Storage. If you don’t have any local volumes on your hosts that are available for placement, you might want to just clear the classification for that drive in the template.
To deploy a template in VMM, in the Library pane, expand Templates and click Service Templates, and then right-click the template we just imported and select Configure Deployment.
Supply the name Gateway and select the appropriate VM networks. The Infrastructure network is the network that the VMM server will use to communicate with this VM (in my environment, this is Management) and the External network is the outgoing connection for the virtualization traffic (in my environment, this is External). Click OK to proceed.
Once the deployment window appears, populate the required settings and click the Refresh Preview button to allow VMM to run placement and decide which hosts the VMs should be deployed to.
After running placement, you should be ready to deploy. Be sure that when choosing the name for the cluster, you choose a name that does not exist in DNS and that it is less than the NetBIOS restriction of 15 characters. Note that I did not include a static IP address for the cluster.
Click the Deploy Service button to begin the deployment.
In PowerShell, deploying a service is somewhat more involved. You’ll need to create a service configuration object, populate its service settings, run placement on the configuration object, and deploy the configuration object, which creates a service instance. Here is a PowerShell script meant to be a guide:
This script isn’t incredibly robust or well-tested; it’s just meant to give you an idea of the process so you can write your own script. Once deployment has completed, you’ll have a Gateway service running.
Next, we need to connect VMM to this gateway.
VMM will need to communicate with the gateway (hence the management connection) to configure it. To do this, we’ll add the gateway as a Network Service to VMM. To create a Network Service, in the Fabric pane, expand Networking, right-click Network Service, and select Add Network Service.
On the first page of the wizard, name the service NAT Gateway.
On the next page, select Microsoft Windows Server Gateway as the Model.
Provide the appropriate Run As Account on the next page. This account needs to either match the account you supplied for the template or another account with appropriate rights.
Continue to the Connection String page. When a provider is installed on the VMM server, the provider author provides a library for VMM to use that contains a set of callback functions. The connection string is provided as input to these functions, and these functions parse the string. As such, VMM does not dictate the format of the string; it’s provider-specific.
For the Gateway Provider, the connection string must follow the pattern Parameter1=Value1;Paramter2=Value2;…;ParameterN=ValueN. The possible parameters are:
In my environment, I’ll use:
You can supply this string in the field and click Next:
Continue to the Provider page and click the Test button to verify configuration.
Continue to the Host Group page and select Virtualization and Edge.
Finish the wizard to add the network service. In PowerShell, you might want to do this in two steps. First, test the configuration:
That should give you output like this:
Providing you get a True for all but MarkedForDeletion and IsViewOnly, you can proceed to add the gateway:
Now you should have a gateway:
The next step is configuring the networks to use this new gateway.
First, open the properties of the gateway and click on the Connectivity page. Select the appropriate adapters. The Frontend network will be the External connection and the Backend will be the Virtualization network. The template has already populated information based on the NIC names that were generated in the template scripts. In my environment, my External network is corporate domain corp.microsoft.com. I’ll select the NIC corresponding to that connection for my frontend network. For the backend network, I’ll select the NIC that the template scripts labeled as BackEnd. My settings look like this:
After clicking OK, VMM will configure the appropriate settings and connect the vNICs to the proper networks through the Management adapter. In PowerShell:
Now that we have all those components in place, we simply need to add the gateway to a VM Network using network virtualization for isolation.
First, bring up the properties of the VM Network using network virtualization. If you’ve been following my posts, you might recall I have Fabrikam2 and Fabrikam3 as isolated VM Networks on the Virtualization network. I’ll bring up the properties of Fabrikam2. On the Connectivity page, some options now appear. We’ll select Connect directly to an additional logical network and choose Network address translation (NAT) as the gateway type.
You can see some statistics around remaining capacity and information about the gateway device. Note that the gateway device lists its capabilities based on the provider capabilities, not the actual configuration. In a future post, I’ll get into configuring each capability.
You can click OK to make the changes. At this point, VMM will configure the gateway to connect its NIC to the virtualization network and create all the routing necessary to make connectivity happen. In PowerShell,
If you have existing VMs, they should now have external connectivity. In order to successfully resolve DNS, the IP pool assigned to the virtualized network will need DNS servers. You can either deploy a DNS server within the virtualized network, or you can simply supply external IP addresses for DNS servers.
In the next post, we’ll be cleaning up the environment and flattening the VLAN topology to get a converged fabric, making things easier to manage and making onboarding of new servers simple. I hope you’re finding this series helpful. Feel free to offer any requests for content. If you want to read about, someone else does, too.
If you’ve been following this series, you’ll notice I added a Part VI that will be related to technical details. I’ve had feedback from readers to go into more technical content, so I’ll be planning that post as a follow-up to this series. I’ll try to make that post as comprehensive as possible, but leave a comment if there are any specific components you want the details about.
Sorry for bothering you again. It will be nice to explain what are the cluster group resources created on the host and guest clusters. What kind of information is saved on the CSV guest cluster. What is the dependency between the guest and host cluster resources.
Also we were thinking how to backup WSG cluster. As it has CSV it must be within the VM but what will happen after the restore. If we have older point in time does VMM will re apply the configuration after that time.
You understand that there are too many unanswered questions when there is no documentation and high demand for the product.
In the deployment steps, I can see you deploy the Gateway VMs to the same domain in which your VMM, your hosts and other management roles (DC) are running. This is not a production-ready example.
Please advise how the Gateway VMs can be deployed in a secure manner. The Gateway VMs are running on dedicated hosts which are placed on the edge of your network.
So from a security perspective, do we need to deploy the gateway VMs and dedicated hosts to a DMZ domain? How do we enable communication between VMM and the Gateway VMs/hosts as this is required in order to sent out the virtualization policies?
It's no bother. Getting feedback on the lacking documentation is an important step in the process of getting things documented. My goal is to get all the necessary technical information to the field to help you. With end-to-end scenarios like the WSG, there were a lot of teams involved. When I start to document the VMM components, I won't discover that the RRAS components also need documentation unless I get that feedback. So thanks, and I'll be sure to include that info in the technical dive. My current plan is to go through the NVGRE setup one component at a time and stop at each step to show properties and settings that get configured to show how things work. It sounds like it would be best if I don't use a template, instead creating the gateway cluster by hand, so I'll do it that way.
That's a great point. I set up things this way because the template built by the feature team used that configuration. There are a few security approaches, depending on what you want to protect.
You could dedicate an isolated network for your entire environment. That is, an entire segregated domain could sit in the DMZ, which would include your VMM server and clusters. Or, you could put both the edge cluster and the cluster housing the VM workload in the DMZ. Or, you could put the hosts with the VM workload in your corporate network, place the management network in a DMZ, and place the edge cluster in a separate DMZ. There are lot of options, providing connectivity requirements are met.
Going back to your question, I could put together an example that follows these blog posts but places your Edge cluster outside your management network, but there are a few requirements.
First, you could not use the template provided by the feature team to deploy. You'd have to set up the guest cluster manually and configure its components.
Next, the DMZ must have a domain and the nodes in the Edge cluster must belong to it. It could be a different domain than your management network, but clustering requires AD, so there's no way around that. The same goes for the in-guest gateway cluster: it needs some AD to be present for the cluster.
The last thing is L3 connectivity. The virtualization PA space will need to be routable across the DMZ, and the VMM server must have L3 connectivity to the gateway VM and the edge cluster. In your firewall, you'd need to ensure that the policies to pass this traffic are in place.
I'll put together an example on this type of setup for the deep-dive, hopefully that will help you.
1. I think the Powershell Command for create VM Network for the External Network cannot by trought. You have assigned the VLAN 801 to the External Network and in the Powershell command you specific the IsolationMode -eq to NoIsolation, but it cannot by NoIsolation it must by "VLANNetwork".
2. You create a VM Network in the previews Post II for the Virtualization Network with NoIsolation and specific this in Bold. But on the 3 Posts you not use this VM Network "Virtualization" any more? Why you create a VM Networt for the PA Network?
To answer the first question, I chose NoIsolation when I created the External Network, so the VM Network must also be NoIsolation. In fact, I didn't even need to provide the IsolationMode argument to the cmdlet, as there is no other option.
On the second question, you're correct, and I should have been clearer on that point. I'm not using the Virtualization VM Network; I'm only using the Logical Network and it's dedicated for the PA space. I created the VM Network for completeness, and VMs could be deployed to this network (if you had any reason to do so) but I won't be using it in this series.
Thanks for the great series. I've been able to finally grasp network virtualization with VMM through your posts.
thanks for your answer. We have follow your exact guidance by creating the Network Virtualization including the Gateway be one of oure customer. Now it seems to by all ok, but we have the Problem that we can only ping to each other and to the outside World. But all other protocols will not work like RDP, SMB etc. All FW are disabled. Can you given us a tip what could be wrong?
Glad to help. Feel free to ask for clarification on anything.
So ICMP traffic is working between VMs and outbound, but application-layer traffic is not working? Does that mean your VMs can't connect via RDP to each other or does it mean you can't connect with RDP to an external address from a VM? Have you tried outbound HTTP traffic? It could be MTU issues. Check my comment on the previous post related to MTU questions YSDimov asked. In summary, you could try 'ping bing.com -f -l <size>' and see if you get a reply or timeout. Start with a size like 1430, which should work. Then increase the size. If you get timeouts, your MTUs are probably improperly set. If you get only responses below a certain size and only fragmentation messages above it, everything is as it should be and the problem is probably somewhere else.
thanks for the tip. It could be that the problem is the MTU size.
I testet the ping and all above 1430 are lost! Which adapter should we Change the MTU Size? Only the vNic adapter or all virtual and physical adapter on each host? And can this done by powershell?
Thanks for posting this blog. Really helpful. I have successfully implemented Hyper-V Network Virtualization in our test environment but find a feature missing.
I setup the gateway as a NAT Gateway which allow all the VMs in the virtual network to get out. Now I need to allow certain services from our physical network into the virtual network. Unfortunately, there is no way to define source IP addresses in the NAT GUI. For example, if I have 3 different VMs in the virtual environment and I want to setup RDP to each one of those VMs from my desktop, I can't do that. I can only do one Incoming Port and Destination IP/Port combination. So once I define RDP port 3389 for a destination virtual IP/Port I can not define a second 3389 for the second destination virtual IP/Port. You should allow more than one Incoming IP Addresses for such a scenario.
Let me know if I am missing something here.
Hi Steve disable Encapsulation Offload in the driver of the logicalswitch/Network it will fix the problem
When can we expect the next installment Kurt?
Kurt, will you continue the series?