Windows Server 2012 provided major improvements to the Hyper-V role, including increased consolidation of server workloads, Hyper-V Replica, Cluster Aware Updating (CAU), network virtualization and the Hyper-V extensible switch, just to name a few! Hyper-V 3.0, as some call it, helps organizations improve server utilization while reducing costs.
The following is a checklist I initially developed for Windows Server 2008 R2 SP1 (which can be found here: http://blogs.technet.com/b/askpfeplat/archive/2012/11/19/hyper-v-2008-r2-sp1-best-practices-in-easy-checklist-form.aspx) and overhauled with the latest release. Those of you who have used my previous checklist will notice quite a few items remaining; that’s because many of the best practices still apply to Hyper-V in Server 2012!
I find having a checklist can be a great tool to use not only when reviewing an existing Hyper-V implementation, but one which can be leveraged as part of pre-planning stages, to ensure best practices are implemented from the start.
It’s important to note this is not an exhaustive compilation, rather a grouping of features/options commonly used in businesses I’ve had the pleasure of assisting.
A special thanks to Ted Teknos, Ryan Zoeller and Rob Hefner for their input/suggestions/corrections as I put this together!
So, without further ado, here’s the newly updated Hyper-V 2012 Best Practice Checklist!
Disclaimer: As with all Best Practices, not every recommendation can – or should – be applied. Best Practices are general guidelines, not hard, fast rules that must be followed. As such, you should carefully review each item to determine if it makes sense in your environment. If implementing one (or more) of these Best Practices seems sensible, great; if it doesn't, simply ignore it. In other words, it's up to you to decide if you should apply these in your setting.
GENERAL (HOST):
⎕ Use Server Core, if possible, to reduce OS overhead, reduce potential attack surface, and to minimize reboots (due to fewer software updates).
⎕ Ensure hosts are up-to-date with recommended Microsoft updates, to ensure critical patches and updates – addressing security concerns or fixes to the core OS – are applied.
⎕ Ensure all applicable Hyper-V hotfixes and Cluster hotfixes (if applicable) have been applied. Review the following sites and compare it to your environment, since not all hotfixes will be applicable:
· Update List for Windows Server 2012 Hyper-V: http://social.technet.microsoft.com/wiki/contents/articles/15576.hyper-v-update-list-for-windows-server-2012.aspx
· List of Failover Cluster Hotfixes: http://social.technet.microsoft.com/wiki/contents/articles/15577.list-of-failover-cluster-hotfixes-for-windows-server-2012.aspx · Failover Cluster Management snap-in crashes after you install update 2750149 on a Windows Server 2012-based failover cluster: http://support.microsoft.com/kb/2803748
· List of Failover Cluster Hotfixes: http://social.technet.microsoft.com/wiki/contents/articles/15577.list-of-failover-cluster-hotfixes-for-windows-server-2012.aspx
· Failover Cluster Management snap-in crashes after you install update 2750149 on a Windows Server 2012-based failover cluster: http://support.microsoft.com/kb/2803748
⎕ Ensure hosts have the latest BIOS version, as well as other hardware devices (such as Synthetic Fibre Channel, NIC’s, etc.), to address any known issues/supportability
⎕ Host should be domain joined, unless security standards dictate otherwise. Doing so makes it possible to centralize the management of policies for identity, security, and auditing. Additionally, hosts must be domain joined before you can create a Hyper-V High-Availability Cluster.
· For more information: http://technet.microsoft.com/en-us/library/ee941123(v=WS.10).aspx
⎕ RDP Printer Mapping should be disabled on hosts, to remove any chance of a printer driver causing instability issues on the host machine.
⎕ Do not install any other Roles on a host besides the Hyper-V role and the Remote Desktop Services roles (if VDI will be used on the host).
⎕ The only Features that should be installed on the host are: Failover Cluster Manager (if host will become part of a cluster), Multipath I/O (if host will be connecting to an iSCSI SAN, Spaces and/or Fibre Channel), or Remote Desktop Services if VDI is being used. (See explanation above for reasons why installing additional features is not recommended.)
⎕ Anti-virus software should exclude Hyper-V specific files using the Hyper-V: Antivirus Exclusions for Hyper-V Hosts article, namely:
⎕ Default path for Virtual Hard Disks (VHD/VHDX) should be set to a non-system drive, due to this can cause disk latency as well as create the potential for the host running out of disk space.
⎕ If you choose to save the VM state as the Automatic Stop Action, the default virtual machine path should be set to a non-system drive, due to the creation of a .bin file is created that matches the size of memory reserved for the virtual machine. A .vsv file may also be created in the same location as the .bin file, adding to disk space used for each VM. (The default path is: C:\ProgramData\Microsoft\Windows\Hyper-V.)
⎕ If you are using iSCSI: In Windows Firewall with Advanced Security, enable iSCSI Service (TCP-In) for Inbound and iSCSI Service (TCP-Out) for outbound in Firewall settings on each host, to allow iSCSI traffic to pass to and from host and SAN device. Not enabling these rules will prevent iSCSI communication.
To set the iSCSI firewall rules via netsh, you can use the following command:
Netsh advfirewall firewall set rule group=”iSCSI Service” new enable=yes
⎕ Periodically run performance counters against the host, to ensure optimal performance.
GENERAL (VMs):
⎕ Ensure you are running only supported guests in your environment. For a complete listing, refer to the following list: http://blogs.technet.com/b/schadinio/archive/2012/06/26/windows-server-2012-hyper-v-list-of-supported-client-os.aspx
PHYSICAL NICs:
⎕ Ensure NICs have the latest firmware, which often address known issues with hardware.
⎕ Ensure latest NIC drivers have been installed on the host, which resolve known issues and/or increase performance.
⎕ NICs should not use APIPA (Automatic Private IP Addressing). APIPA is non-routable and not registered in DNS.
⎕ VMQ should be enabled on VMQ-capable physical network adapters bound to an external virtual switch.
⎕ TCP Chimney Offload is not supported with Server 2012 software-based NIC teaming, due to TCP Chimney has the entire networking stack offloaded to the NIC. If software-based NIC teaming is not used, however, you can leave it enabled.
⎕ Jumbo frames should be turned on and set for 9000 or 9014 (depending on your hardware) for CSV, iSCSI and Live Migration networks. This can significantly increase (6x increased throughput) throughput while also reducing CPU cycles.
⎕ NICs used for iSCSI communication should have all Networking protocols (on the Local Area Connection Properties) unchecked, with the exception of:
⎕ NIC Teaming should not be used on iSCSI NIC’s. MPIO is the best method. NIC teaming can be used on the Management, Production (VM traffic), CSV Heartbeat and Live Migration networks.
⎕ If you are using NIC teaming for Management, CSV Heartbeat and/or Live Migration, create the team(s) before you begin assigning Networks.
⎕ If using aggregate (switch-dependent) NIC teaming in a guest VM, only SR-IOV NICs should be used on guest.
⎕ If using NIC teaming inside a guest VM, follow this order:
METHOD #1:
METHOD #2:
⎕ When creating virtual switches, it is best practice to uncheck Allow management operating system to share this network adapter, in order to create a dedicated network for your VM(s) to communicate with other computers on the physical network. (If the management adapter is shared, do not modify protocols on the NIC.)
Please note: we fully support and even recommend (in some cases) using the virtual switch to separate networks for Management, Live Migration, CSV/Heartbeat and even iSCSI. For example two 10GB NIC’s that are split out using VLANs and QoS.
⎕ Recommended network configuration when clustering:
Min # of Networks on Host
Host Management
VM Network Access
CSV/Heartbeat
Live Migration
iSCSI
5
“Management”
“Production”
“CSV/Heartbeat”
“Live Migration”
“iSCSI”
** CSV/Heartbeat & Live Migration Networks can be crossover cables connecting the nodes, but only if you are building a two (2) node cluster. Anything above two (2) nodes requires a switch. **
⎕ Turn off cluster communication on the iSCSI network.
⎕ Redundant network paths are strongly encouraged (multiple switches) – especially for your Live Migration and iSCSI network – as it provides resiliency and quality of service (QoS).
VLANS:
⎕ If aggregate NIC Teaming is enabled for Management and/or Live Migration networks, the physical switch ports the host is connected to should be set to trunk (promiscuous) mode. The physical switch should pass all traffic to the host for filtering.
⎕ Turn off VLAN filters on teamed NICs. Let the teaming software or the Hyper-V switch (if present) do the filtering.
VIRTUAL NETWORK ADAPTERS (NICs):
⎕ Legacy Network Adapters (a.k.a. Emulated NIC drivers) should only be used for PXE booting a VM or when installing non-Hyper-V aware Guest operating systems. Hyper-V's synthetic NICs (the default NIC selection; a.k.a. Synthetic NIC drivers) are far more efficient, due to using a dedicated VMBus to communicate between the virtual NIC and the physical NIC; as a result, there are reduced CPU cycles, as well as much lower hypervisor/guest transitions per operation.
DISK:
⎕ New disks should use the VHDX format. Disks created in earlier Hyper-V iterations should be converted to VHDX, unless there is a need to move the VHD back to a 2008 Hyper-V host.
⎕ Disks should be fixed in a production environment, to increase disk throughput. Differencing and Dynamic disks are not recommended for production, due to increased disk read/write latency times (differencing/dynamic disks).
⎕ Use caution when using snapshots. If not properly managed, snapshots can cause disk space issues, as well as additional physical I/O overhead. Additionally, if you are hosting 2008 R2 (or earlier) Domain Controllers, reverting to an earlier snapshot can cause USN rollbacks. Windows Server 2012 has been updated to help better protect Domain Controllers from USN rollbacks; however, you should still limit usage.
⎕ The recommended minimum free space on CSV volumes containing Hyper-V virtual machine VHD and/or VHDX files:
⎕ It is not supported to create a storage pool using Fiber Channel or iSCSI LUNs.
⎕ Page file on Hyper-V Host should managed by the OS and not configured manually.
MEMORY:
⎕ Guest OS should be configured with (minimum) recommended memory
CLUSTER:
⎕ Set preferred network for CSV communication, to ensure the correct network is used for this traffic. (Note: This will only need to be run on one of your Hyper-V nodes.)
*** Set preferred network for Live Migration, to ensure the correct network(s) are used for this traffic:
⎕ The Cluster Shutdown Time (ShutdownTimeoutInMinutes registry entry) should be set to an acceptable number
⎕ Run the Cluster Validation periodically to remediate any issues
⎕ Consider enabling CSV Cache if you have VMs that are used primarily for read requests, and are less write intensive. Scenarios such as Pooled VDI VMs; also can be leveraged for reducing VM boot storms.
HYPER-V REPLICA:
⎕ If utilizing Hyper-V Replica, update inbound traffic on the firewall to allow TCP port ‘80’ and/or port ‘443’ traffic. (In Windows Firewall, enable “Hyper-V Replica HTTP Listener (TCP-In)” rule on each node of the cluster.
To enable HTTP (port 80) replica traffic, you can run the following from an elevated command-prompt:
netsh advfirewall firewall set rule group="Hyper-V Replica HTTP" new enable=yes
To enable HTTPS (port 443) replica traffic, you can run the following from an elevated command-prompt:
netsh advfirewall firewall set rule group="Hyper-V Replica HTTPS" new enable=yes
⎕ Compression is recommended for replication traffic, to reduce bandwidth requirements.
⎕ Configure guest operating systems for VSS-based backups to enable application-consistent snapshots for Hyper-V Replica.
⎕ Integration services must be installed before primary or Replica virtual machines can use an alternate IP address after a failover
⎕ Virtual hard disks with paging files should be excluded from replication, unless the page file is on the OS disk.
⎕ Test failovers should be performed monthly, at a minimum, to verify that failover will succeed and that virtual machine workloads will operate as expected after failover
⎕ Hyper-V Replica requires the Failover Clustering Hyper-V Replica Broker role be configured if either the primary or the replica server is part of a cluster.
⎕ Feature and performance optimization of Hyper-V Replica can be further tuned by using the registry keys mentioned in the article below:
CLUSTER-AWARE UPDATING:
⎕ Place all Cluster-Aware Updating (CAU) Run Profiles on a single File Share accessible to all potential CAU Update Coordinators. (Run Profiles are configuration settings that can be saved as an XML file called an Updating Run Profile and reused for later Updating Runs. http://technet.microsoft.com/en-us/library/jj134224.aspx
SMB 3.0 FILE SHARES:
⎕ An Active Directory infrastructure is required, so you can grant permissions to the computer account of the Hyper-V hosts.
⎕ Loopback configurations (where the computer that is running Hyper-V is used as the file server for virtual machine storage) are not supported. Similarly, running the file share in VM’s that are hosted on compute nodes that will serve other VM’s is not supported.
VITRUAL DOMAIN CONTROLLERS (DCs):
⎕ Domain Controller VMs should have "Shut down the guest operating system" in the Automatic Stop Action setting applied (in the virtual machine settings on the Hyper-V Host)
INTEGRATION SERVICES:
⎕ Ensure Integration Services (IS) have been installed on all VMs. IC's significantly improve interaction between the VM and the physical host.
⎕ Be certain you are running the latest version of integration services – the same version as the host(s) – in all guest operating systems, as some Microsoft updates make changes/improvements to the Integration Services software. (When a new Integration Services version is updated on the host(s) it does not automatically update the guest operating systems.)
OFFLOADED DATA TRANSFER (ODX) Usage:
⎕ If your SAN supports ODX (see this post for help; also check with your hardware vendor), you should strongly consider enabling ODX on your Hyper-V hosts, as well as any VMs that connect directly to SAN storage LUNs.
I sincerely hope you find this blog posting useful! If you do, please forward the link on to others who may benefit!
Until next time,
Roger Osborne, Microsoft PFE
Thanks for the post. Regarding the Guest OS and (minimum) recommended memory - the values should be in MBs, not in GBs so it's a typo, am I right? :)
@Miroslav - Yes, that was a typo. Thanks, I have corrected that on the blog post! :-)
This page has some weird facebook related warnings floating on it. Only showing when viewed w/ IE. Chrome etc. doesn't show them.
SECURITY WARNING: Please treat the URL above as you would your password and do not share it with anyone.
s-static.ak.facebook.com/.../xd_arbiter.php
@NA I see this as well we will look into this.
Roger, really a great resource! Two questions: (1) Dynamic VHDX: the "performance-tuning-guidelines-windows-server-2012.docx" from http://download.microsoft.com states on page 125 "When using the VHDX format, we recommend that you use the dynamic type because it offers resiliency guarantees in addition to space savings that are associated with allocating space only when there is a need to do so. ". You still recommend fixed size. Why? (2) Is it a bad / neutral / good suggestion to set *NdisDeviceType in the registry to value 1 (NDIS_DEVICE_TYPE_ENDPOINT) on dedicated iSCSI interfaces in order to turn off Windows Firewall on this device ("Public Network" problem)?
@Maurice,
Thank you for your comment!
Regarding your questions:
1.) I'm not saying never use it in production. :-) I just think you need to weigh the pros & cons. Our Hyper-V team did AMAZING work on the new VHDX format; however, when using dynamically expanding VHDX files, I see the possibility for file fragmentation as the VHDX file increases in size over time, which can lead to overall reduced speed when compared to a VHDX that has been fully allocated at creation.
The second reason for recommending fixed over dynamic is to prevent your VM storage volumes from running out of disk space, due to having dynamically expanding VHDX's (suddenly and unexpectedly) maxing out the volume they're sitting on (SAN, local or otherwise), leading to VMs suddenly pausing, and creating outages. Careful planning and oversight should be adhered to when you decide to use dynamic disks.
2.) I’m not sure I’m following your point regarding turning off the firewall on the iSCSI NICs via the registry entry, as there wouldn’t be any appreciable performance gains, in my opinion. If your goal is to ensure uninhibited iSCSI traffic, my recommendation is to enable (allow) the built-in iSCSI rule within Windows Firewall, and not change the registry.
Thanks for reading!
Is the free space recommendation for the CSV also applies for Windows 2008 R2 SP1 Hyper-V?
For the Hyper-V host OS recommendation, why not the recommendation to use Hyper-V server 2012 in case the server is dedicated as virtualization host?
With Hyper-V Server 2012 you will have better results in reducing OS overhead, reducing potential attack surface, and to minimize reboots (due to fewer software updates). Hyper-V Servers 2012 can also be used in combination with Windows Server Standard and Datacenter Licenses.
@ Mark - Yes, I would consider this a recommendation that would apply to Server 2008 R2.
@ Dennis - There are certainly times when using Hyper-V Server 2012 is desirable, such as a lab environment; however, I most often recommend using Windows 2012 Datacenter (or Enterprise), as these products almost always end up being the most cost-effective, due to licensing. Hyper-V Server 2012 does not come with any licenses. www.microsoft.com/.../buy.aspx
Very nice article, love the detail.
What software for backup vm in hyper-v ?!
Great article. Love it!
@ Teachmepls - I recommend looking at System Center Data Protection Manager (DPM) 2012. technet.microsoft.com/.../hh758173.aspx
I thank you for all of the information. I have two quick questions. I have a server that has the ability to do NIC teaming within the BIOS. Would that be recommended or should I stay with Windows to complete the task? Also I have 12 nics within each server how many should I put towards the ISCSI connection and how many for the virtual networks?
Thank you
Great article! Thank you for putting this together. Definitely adding this to my Hyper-V deployment bible.