Ask the Core Team : Cluster, Hyper-V

Creating a Virtual Machine to be stored on a Cluster Shared Volume

jeffhugh — Mon, 30 Nov 2009 15:39:24 GMT

Cluster Shared Volumes is a new feature of Windows Server 2008 R2 Failover Clustering intended to enhance the capabilities of Highly Available Virtual Machines. It is recommended to utilize Cluster Shared Volumes if you intend to use the Live Migration feature of Windows Server 2008 R2 Hyper-V. In this blog I would like to cover a simple yet common oversight when creating a Virtual Machine to be used with Cluster Shared Volumes.

How to enable Cluster Shared Volumes

To enable Cluster Shared Volumes, on which you will place your Virtual Machines, we can use Failover Cluster Manager or the following Failover Cluster PowerShell CmdLets.

Using Failover Cluster Manager, simply select the cluster and from the Configure pane select Enable Cluster Shared Volumes.

Figure 1: Enabling Cluster Shared Volumes

Using PowerShell, ensure that the Failover Cluster module is loaded and run the following command:

PS C:\> get-cluster “your cluster name” | %{$_.EnableSharedVolumes="Enabled"}

Read the exception returned by the above command and then enable CSV following the instructions.

We must stop here for the fine print: Cluster Shared Volumes Support for Hyper-V
In Windows Server 2008 R2, the Cluster Shared Volumes feature included in failover clustering is only supported for use with the Hyper-V server role. The creation, reproduction, and storage of files on Cluster Shared Volumes that were not created for the Hyper-V role, including any user or application data stored under the ClusterStorage directory of the system drive on every node, are not supported and may result in unpredictable behavior, including data corruption or data loss on these shared volumes. An example of a file type that is created for the Hyper-V role is a virtual hard disk (.vhd) file.

Once enabled, you can add one or more available storage clustered disks as a Cluster Shared Volume:

Figure 2: Adding disks to be used as Cluster Shared Volumes

Now that we enabled and added disks as Cluster Shared Volumes, we will have the following folder structure on the %SystemDrive%, which will typically be C:\

C:\ClusterStorage\Volume1
C:\ClusterStorage\Volume2
C:\ClusterStorage\Volume3

VolumeN above represents an entire clustered disk. These folders can be renamed for administrative purposes; however the C:\ClusterStorage folder must remain as it is.

For our example here I will rename C:\ClusterStorage\Volume1 to C:\ClusterStorage\AccntsDB-VMs.

For a more detailed walk-through of using Cluster Shared Volumes for virtual machine live migration see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

Now for the main purpose of this blog…

Creating the Virtual Machine on the Cluster Shared Volume:

For a Highly Available VM to function correctly all configuration and data files for the virtual machine must reside on a cluster shared disk that will be accessible to which ever node currently owns the VM. When using Cluster Shared Volumes this means all files must reside in the same folder, the folder where the Cluster Shared Volume disk is mounted, in our example C:\ClusterStorage\AccntsDB-VMs.

We create the Virtual Machine using Hyper-V Manager, as you would any VM. You can create a new VHD to install into or utilize an existing VHD. Below is an example of each. The critical point in this process is where to locate the entire Virtual Machine, not only the VHD file. I have seen many people copy the VHD file to the correct CSV folder but miss this step, resulting in all of the configuration files being placed into the default location, which most likely will not be available to all nodes in the cluster resulting in failures when attempting to make highly available or move the VMs between cluster nodes.

While creating a New Virtual Machine, choose a location:

Figure 3: Change the VM location to the desired CSV folder

Specifying a VHD file for the VM:

Figure 4: Specify the location and name for the VHD

Or use an existing VHD… (Of course we would need to have copied the file to this location previously)

Figure 5: Specify the location of an existing VHD

Once you have located the VM and VHD(s) onto the Cluster Shared Volume properly, proceed with creating the virtual machine as you normally would. In addition, we still need to create the Highly Available virtual machine resource using Failover Cluster Manager.

For a more detailed walk-through of creating virtual machines for live migration please see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

For more information please reference:

Understanding Cluster Shared Volumes in a Failover Cluster

Understanding Hyper-V and Virtual Machines in the Context of a Cluster

Errors you may encounter if not done properly include:

The operation has failed. None of the virtual machine configurations chosen was successfully made highly available.

Report Errors:

“Disk path 'C:\ProgramData\Microsoft\Windows\Hyper-V' is not a path to storage in the cluster or to storage that can be added to the cluster. You must ensure this storage is available to every node in the cluster to make this virtual machine highly available.”

There was a failure bringing the virtual machine configuration resource 'Virtual Machine Configuration New Virtual Machine' online.

Chris Allen
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Expected Snapshot Merge Behavior for a Highly Available VM

jeffhugh — Fri, 29 May 2009 15:22:03 GMT

Hello my name is Sean Dwyer, and I'm a Support Escalation Engineer working in the Windows CORE team here at Microsoft.

While working with snapshots that are attached to a highly available VM in a Cluster, you may notice after deleting a snapshot, it does not merge as expected. You'll notice the merge process almost immediately ends and the VM begins to restart.

Let's explore a scenario I ran myself into the other day and then I'll explain why you may see this behavior.

Fig. 1.

Here I have configured a HA-VM of XP SP2 in Fig 1. and it is Online and happy, within the Cluster.

Fig. 2.

Switching to the Hyper-V Management MMC in Fig. 2, I've decided that my testing is over with this VM, and I need to clean up the snapshots. The 'Now' state is where I want the VM to remain, so I'm going to issue a Delete Snapshot Subtree command.

Fig. 3.

I choose 'Yes', and the snapshots are removed from the VM in Fig. 3.
Next, I choose to shut down the VM in Fig. 4, as I want to commit the merge now, and not wait until later.

Fig. 4.

The Merge process starts, but immediately finishes and the VM is restarting as seen in Fig. 5 and Fig. 6.

Fig. 5.

Fig. 6.

The merge didn't complete! What just happened?

Failover Cluster is doing its job!

When deleting a snapshot for a VM that is configured as a Highly Available Resource, you're taking action against the VM outside of the Cluster Management UI, and therefore, the Cluster service considers any shutdown events, or reboots, a failure of the resource and takes appropriate action.

In my example, the default behavior is restarting the VM as shown in Fig.7.

Fig. 7.

The end result is that the snapshot merge process has NOT taken place and I've still got AVHDs out there shown in Fig. 8.

Fig. 8.

So, what do we do, to get the Snapshots merged successfully? Easy!

We need to configure the Clustered VM Resource to allow the Snapshot Merge process to complete without being interrupted.

Open up the Properties of the VM, and then select the Offline Actions Tab.

You'll note, in Fig. 9, the default behavior is to Save the VM.

Fig. 9.

In Fig. 10, let's change this default behavior temporarily to Shutdown.

Fig. 10.

Once the change has been made, you can then choose to shutdown the VM either through Hyper-V Manager or through the Clustering UI, and the Snapshot Merge process will begin.

Once the Merge completes, reset the behavior of the VM in the Offline Actions Tab for the VM back to Save as shown in Fig. 11.

Fig. 11.

Once you start the VM, you'll be back in business!

I hope this blog post will help you continue to use our Clustering and Virtualization products successfully!

Sean Dwyer
Support Escalation Engineer
Microsoft Enterprise Platforms Support

Top Issues for Microsoft Support for Windows Server 2008 Hyper-V (Q3)

jeffhugh — Wed, 27 May 2009 17:24:55 GMT

It is time to update everyone on the issues our support engineers have been seeing for Hyper-V for the past quarter. The issues are categorized below with the top issue(s) in each category listed with possible resolutions and additional comments as needed. I think you will notice that the issues for Q3 have not changed much from Q1\Q2. Hopefully, the more people read our updates, the fewer occurrences we will see for some of these and eventually they will disappear altogether. There will probably be one more blog for the Q4 results. Additionally, I would like to mention that we are highly recommending the installation of Windows Server 2008 Service Pack 2 on all servers running the Hyper-V Role.

Deployment\Planning

Issue #1

Customers looking for Hyper-V documentation.

Resolution: Information is provided on the Hyper-V TechNet Library which includes links to several Product Team blogs. Additionally, the Microsoft Virtualization site contains information that can be used to get a Hyper-V based solution up and running quickly.

Installation Issues

Issue #1

A customer was experiencing an issue on a pre-release version of Hyper-V.

Resolution: Upgrade to the release version (KB950050) of Hyper-V.

Issue #2

After the latest updates off Windows Update are installed or KB950050 is installed, virtual machines fail to start with one of the following error messages:

An error occurred while attempting to chance the state of the virtual machine vmname .
‘ vmname ’ failed to initialize.
Failed to read or update VM configuration.

An error occurred while attempting to change the state of virtual machine vmname .
" VMName " failed to initialize
An attempt to read or update the virtual machine configuration failed.
" VMName " failed to read or update the virtual machine configuration: Unspecified error (0x80040005).

Cause: This issue occurs because virtual machine configurations that were created in the beta version of the Hyper-V are incompatible with later versions of the Hyper-V.

Resolution: Perform the steps documented in KB949222.

Issue #3

After the Hyper-V role is installed, a customer creates a virtual machine but it fails to start with the following error:

The virtual machine could not be started because the hypervisor is not running

Cause: Hardware virtualization or DEP was disabled in the BIOS.

Resolution: Enable Hardware virtualization or DEP in the BIOS. In some cases, the server may need to be physically shutdown in order for the new BIOS settings to take effect.

Virtual Devices\Drivers

Issue #1

Synthetic NIC was listed as an unknown device in device manager.

Cause: Integration Components needed to be installed.

Resolution: Install Integration Components (IC) package in the VM.

Issue #2

Stop 0x00000050 on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: This issue can occur if a Hyper-V virtual machine is configured with a SCSI controller but no disks are attached.

Resolution: Perform the steps documented in KB969266.

Issue #3

Stop 0x0000001A on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: Vid.sys

Resolution: Install hotfix KB957967 to address this issue.

Snapshots

Issue #1

Snapshots fail to merge with error 0x80070070

Cause: Low disk space.

Resolution: Free up disk space to allow the merge to complete.

Issue #2

Snapshots were deleted

Cause: The most common cause is that a customer deleted the .avhd files to reclaim disk space (not realizing that the .avhd files were the snapshots).

Resolution: Restore data from backup.

For more information on Snapshots, please refer to the Snapshot FAQ: http://technet.microsoft.com/en-us/library/dd560637.aspx.

Issue #3

Snapshots were lost

Cause: Parent VHD was expanded (not supported). If snapshots are associated with a virtual hard disk, the parent vhd file should never be expanded. This is documented in the Edit Disk wizard:

Resolution: Restore data from backup.

Integration Components

Issue #1

A Windows 2000 (SP4) virtual machine with the Integration Components installed may shut down slowly.

Cause: This problem is caused by a bug in the Windows Software Trace Pre-Processor (WPP) tracing macro (outside of Hyper-V).

Resolution: KB959781 documents the workarounds for this issue on Server 2008.

Issue #2

Attempting to install the Integration Components on a Server 2003 virtual machine fails with the following error:

Unsupported Guest OS

An error has occurred: The specified program requires a newer version of Windows.

Cause: Service Pack 2 for Server 2003 wasn’t installed in the virtual machine.

Resolution: Install SP2 in the Server 2003 VM before installing the integration components.

Virtual machine State and Settings

Issue #1

You may experience one of the following issues on a Windows Server 2008 system with the Hyper-V role installed or Microsoft Hyper-V Server 2008:

When you attempt to create or start a virtual machine, you receive one of the following errors:

· The requested operation cannot be performed on a file with a user-mapped section open. ( 0x800704C8 )

· ‘VMName’ Microsoft Synthetic Ethernet Port (Instance ID

{7E0DA81A-A7B4-4DFD-869F-37002C36D816}): Failed to Power On with Error 'The specified network resource or device is no longer available.' (0x80070037).

· The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3)

Virtual machines disappear from the Hyper-V Management Console.

Cause: This issue can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files.

Resolution: Perform the steps documented in KB961804.

Issue #2

Creating or starting a virtual machine fails with the following error:

'General access denied error' (0x80070005).

Cause: This issue can be caused by the Intel IPMI driver.

Resolution: Perform the steps documented in KB969556.

Issue #3

Virtual machines have a state of "Paused-Critical"

Cause: Lack of free disk space on the volume hosting the .vhd or .avhd files.

Resolution: Free up disk space on the volume hosting the .vhd or .avhd files.

High Availability (Failover Clustering)

Issue #1

How to configure Hyper-V on a Failover Cluster.

Resolution: A step-by-step guide is now available which covers how to configure Hyper-V on a Failover Cluster.

Issue #2

Virtual machine settings that are changed on one node in a Failover Cluster are not present when the VM is moved to another node in the cluster.

Cause: The "Refresh virtual machine configuration" option was not used before attempting a failover.

Resolution: When virtual machine settings are changed on a VM that’s on a Failover Cluster, you must select the ‘Refresh virtual machine configuration’ option before the VM is moved to another node. There is a blog that discusses this.

Backup (Hyper-V VSS Writer)

Issue #1

You may experience one of the following symptoms if you try to backup a Hyper-V virtual machine:

· If you back up a Hyper-V virtual machine that has multiple volumes, the backup may fail. If you check the VMMS event log after the backup failure occurs, the following event is logged:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin

Source: Microsoft-Windows-Hyper-V-VMMS

Event ID: 10104

Level: Error

Description:

Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

· The Microsoft Hyper-V VSS Writer may enter an unstable state if a backup of the Hyper-V virtual machine fails. If you run the vssadmin list writers command, the Microsoft Hyper-V VSS Writer is not listed. To return the Microsoft Hyper-V VSS Writer to a stable state, the Hyper-V Virtual Machine Management service must be restarted.

Resolution: An update (KB959962) is now available to address issues with backing up and restoring Hyper-V virtual machines.

Issue #2

How to backup virtual machines using Windows Server Backup

Resolution: Perform the steps documented in KB958662.

Virtual Network Manager

Issue #1

Virtual machines are unable to access the external network.

Cause: The virtual network was configured to use the wrong physical NIC.

Resolution: Configure the external network to use the correct NIC.

Issue #2

Network connectivity issues

Cause: NIC teaming software

Resolution: Remove the NIC teaming software. Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Issue #3

Customers inquiring if Hyper-V supports NIC Teaming.

Resolution: Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Hyper-V Management Console

Issue #1

How to manage Hyper-V remotely.

Resolution: The steps to configure remote administration of Hyper-V are covered in a TechNet article. John Howard also has a very thorough blog on remote administration.

Import/Export

Issue #1

Importing a virtual machine may fail with the following error:

A Server error occurred while attempting to import the virtual machine. Failed to import the virtual machine from import directory <Directory Path>. Error: One or more arguments are invalid (0x80070057).

Resolution: Perform the steps documented in KB968968.

Miscellaneous

Issue #1

You may experience one of the following issues on a Windows Server 2003 virtual machine:

· An Event ID 1054 is logged to the Application Event log:

Event ID: 1054
Source: Userenv
Type: Error
Description:
Windows cannot obtain the domain controller name for your computer network. (The specified domain either does not exist or could not be contacted). Group Policy processing aborted.

· A negative ping time is displayed when you use the ping command.

· Perfmon shows high disk queue lengths

Cause: This problem occurs when the time-stamp counters (TSC) for different processor cores are not synchronized.

Resolution: Perform the steps documented in KB938448.

As always, we hope this has been informative for you.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Cluster Resource Type options for Hyper-V

tomac — Wed, 27 Aug 2008 22:39:25 GMT

Windows Server 2008 Clustering includes resource type specific options which are available by right clicking on the group under Applications and Services.

For Hyper-V, we have the following options.

Manage virtual machine. This option opens Hyper-V Manager with connections to all participating nodes in the cluster. **NOTE: The virtual machine configuration resource must be online for the VM to be registered with Hyper-V and viewable in Hyper-V Manager.
Refresh virtual machine configuration. This option is the most important and is the primary focus of this blog.

In order to better explain how the Refresh virtual machine configuration option is important, let’s run thru a quick breakdown of how Windows Server 2008 Clustering stores information about each virtual machine.

Once you make a virtual machine highly available, we create two resources per VM excluding any disk resource(s).

Virtual machine configuration: This resource is responsible for registering/unregistering the virtual machine with Hyper-V on each node.

Virtual machine: This resource is responsible for changing the state of the VM. IE: Turn on, turn off, shutdown.

The VM resource that we want to inspect is the virtual machine configuration resource. In the screenshot below, I have used cluster.exe to dump the private properties for the “Virtual Machine Configuration HA VM” resource using the following syntax:

Cluster res “Virtual Machine Configuration HA VM” /priv

Note that we store the following in private properties.

VmSwitchPorts: This references the GUID and friendly name for the virtual network(s) the VM’s adapters are connected to as well as any VLAN ID defined for the virtual adapter(s) in the VM. In the example above, my Virtual Network is identified by Switch=’External’.

VmPhysicalDisks: If the virtual machine is configured to use a physical disk as a pass-through disk, the name of the Cluster Disk resource used as the pass-through disk will be referenced here. In the example above, my pass-through disk is identified as Resource=’Cluster Disk’.

VmID: This represents the VM GUID for the Virtual Machine. This is dynamically generated when the virtual machine is created and will not change.

VMStoreRootPath: This is the root folder location of the virtual machines directory for the VM, where our configuration data is stored.

The private properties only contain information that was passed to the Cluster service when the resource was created. So what if networking or disk information changes?

This is where the Refresh virtual machine configuration option comes into play. Whenever you need to make changes to the virtual network, such as connecting a VM Adapter to another virtual network, defining a VLAN ID, or changing the VLAN ID, you must refresh the virtual machine configuration in order for the VM Resource to come online. Also, if you decide to add a physical disk to the VM or replace a physical disk, you must refresh the virtual machine configuration in the cluster.

After right clicking on the group and selecting the option to “Refresh virtual machine configuration”, you will be prompted to take the virtual machine resource offline if it is not already. Once complete, you will be provided with a report similar to the Validation wizard with the outcome of the refresh operation.

The following screenshots provide an example of the report output. In this example, no changes are made. It is important to review the output incase other configuration changes are required.

Author:
Rob Hefner
Support Escalation Engineer
Microsoft Corporation

Technorati Tags: Hyper-Virtual Machine