Ask the Core Team : Cluster

Creating a Virtual Machine to be stored on a Cluster Shared Volume

jeffhugh — Mon, 30 Nov 2009 15:39:24 GMT

Cluster Shared Volumes is a new feature of Windows Server 2008 R2 Failover Clustering intended to enhance the capabilities of Highly Available Virtual Machines. It is recommended to utilize Cluster Shared Volumes if you intend to use the Live Migration feature of Windows Server 2008 R2 Hyper-V. In this blog I would like to cover a simple yet common oversight when creating a Virtual Machine to be used with Cluster Shared Volumes.

How to enable Cluster Shared Volumes

To enable Cluster Shared Volumes, on which you will place your Virtual Machines, we can use Failover Cluster Manager or the following Failover Cluster PowerShell CmdLets.

Using Failover Cluster Manager, simply select the cluster and from the Configure pane select Enable Cluster Shared Volumes.

Figure 1: Enabling Cluster Shared Volumes

Using PowerShell, ensure that the Failover Cluster module is loaded and run the following command:

PS C:\> get-cluster “your cluster name” | %{$_.EnableSharedVolumes="Enabled"}

Read the exception returned by the above command and then enable CSV following the instructions.

We must stop here for the fine print: Cluster Shared Volumes Support for Hyper-V
In Windows Server 2008 R2, the Cluster Shared Volumes feature included in failover clustering is only supported for use with the Hyper-V server role. The creation, reproduction, and storage of files on Cluster Shared Volumes that were not created for the Hyper-V role, including any user or application data stored under the ClusterStorage directory of the system drive on every node, are not supported and may result in unpredictable behavior, including data corruption or data loss on these shared volumes. An example of a file type that is created for the Hyper-V role is a virtual hard disk (.vhd) file.

Once enabled, you can add one or more available storage clustered disks as a Cluster Shared Volume:

Figure 2: Adding disks to be used as Cluster Shared Volumes

Now that we enabled and added disks as Cluster Shared Volumes, we will have the following folder structure on the %SystemDrive%, which will typically be C:\

C:\ClusterStorage\Volume1
C:\ClusterStorage\Volume2
C:\ClusterStorage\Volume3

VolumeN above represents an entire clustered disk. These folders can be renamed for administrative purposes; however the C:\ClusterStorage folder must remain as it is.

For our example here I will rename C:\ClusterStorage\Volume1 to C:\ClusterStorage\AccntsDB-VMs.

For a more detailed walk-through of using Cluster Shared Volumes for virtual machine live migration see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

Now for the main purpose of this blog…

Creating the Virtual Machine on the Cluster Shared Volume:

For a Highly Available VM to function correctly all configuration and data files for the virtual machine must reside on a cluster shared disk that will be accessible to which ever node currently owns the VM. When using Cluster Shared Volumes this means all files must reside in the same folder, the folder where the Cluster Shared Volume disk is mounted, in our example C:\ClusterStorage\AccntsDB-VMs.

We create the Virtual Machine using Hyper-V Manager, as you would any VM. You can create a new VHD to install into or utilize an existing VHD. Below is an example of each. The critical point in this process is where to locate the entire Virtual Machine, not only the VHD file. I have seen many people copy the VHD file to the correct CSV folder but miss this step, resulting in all of the configuration files being placed into the default location, which most likely will not be available to all nodes in the cluster resulting in failures when attempting to make highly available or move the VMs between cluster nodes.

While creating a New Virtual Machine, choose a location:

Figure 3: Change the VM location to the desired CSV folder

Specifying a VHD file for the VM:

Figure 4: Specify the location and name for the VHD

Or use an existing VHD… (Of course we would need to have copied the file to this location previously)

Figure 5: Specify the location of an existing VHD

Once you have located the VM and VHD(s) onto the Cluster Shared Volume properly, proceed with creating the virtual machine as you normally would. In addition, we still need to create the Highly Available virtual machine resource using Failover Cluster Manager.

For a more detailed walk-through of creating virtual machines for live migration please see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

For more information please reference:

Understanding Cluster Shared Volumes in a Failover Cluster

Understanding Hyper-V and Virtual Machines in the Context of a Cluster

Errors you may encounter if not done properly include:

The operation has failed. None of the virtual machine configurations chosen was successfully made highly available.

Report Errors:

“Disk path 'C:\ProgramData\Microsoft\Windows\Hyper-V' is not a path to storage in the cluster or to storage that can be added to the cluster. You must ensure this storage is available to every node in the cluster to make this virtual machine highly available.”

There was a failure bringing the virtual machine configuration resource 'Virtual Machine Configuration New Virtual Machine' online.

Chris Allen
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Resource Hosting Subsystem (RHS) In Windows Server 2008 Failover Clusters

jeffhugh — Mon, 23 Nov 2009 12:39:43 GMT

In this blog, I would like to explore some of the inner-workings of the Resource Host Subsystem (RHS) which is responsible for monitoring the health of the various cluster resources being provided as part of highly available services in a Failover cluster. A Windows Server 2008 Failover Cluster is capable of providing high availability services using a variety of resources some of which are included as part of the Failover Cluster feature and others are as part of ’cluster-aware’ applications like SQL and Exchange. Resources are designed to work together and are typically organized in Resource Groups (Figure 1). For example, a group of resources supporting a highly available File Server may consist of one or more of the following types of resources – Client Access Point (IP Address(s) + Network Name resource), Physical Disk (Storage), and a File Server. A highly available SQL Instance could contain the following resources - Client Access Point (IP Address + Network Name resource), Physical Disk (Storage), SQL Server and SQL Server Agent. Cluster resources are supported by special ‘plugins’ or resource Data Link Libraries (DLLs) that include coding to allow them to properly integrate\interoperate with the cluster service.

Figure 1

A Windows Server 2008 Failover Cluster is capable of hosting an unlimited number of resources. The management of these resources is the responsibility of the Resource Control Manager (RCM) and the Resource Host Subsystem (RHS) which provide this functionality as part of the Cluster Service itself (Figure 2).

Figure 2

The Resource Control Manager (RCM) is part of the overall cluster architecture and is responsible for implementing failover mechanisms and policies for the cluster service as well as establishing and maintaining the dependency tree (Figure 3) for each resource (e.g. a File Server resource requires a dependency on a Client Access Point and a Storage resource).

Figure 3

The Resource Control Manager maintains the state for individual resources (Online, Offline, Failed, Online Pending, and Offline Pending) as well as for Resource Groups (Online, Offline, Partial Online, and Failed). The Resource Control Manager can execute the following actions on a group of resources – Move, Failover and Failback. Which action is executed depends on several factors including the current ‘health’ of resources in the group, administrative actions taken on the group (e.g. Move Group), or the current policies in effect for the group. Here is an example (Figure 4) of Failover and Failback Group Policies –

Figure 4

Individual resources have policies (Figure 5) that apply to them as well.

Figure 5

The Resource Hosting Subsystem (RHS) is responsible for initially hosting all resources that come Online in the cluster in one default process – rhs.exe (Resource Host Monitoring process) (Figure 6).

Figure 6

Note: The rhs.exe *32 process supports 32-bit resource DLLs running in the cluster.

In previous versions of Microsoft clustering, this was called the resource monitor process (resrcmon.exe) (Figure 7).

Figure 7

There is one exception to this rule which has been implemented in the Windows Server 2008 R2 Failover Clustering feature. In Windows Server 2008 R2, the Cluster Group which consists of the Cluster Network Name resource, one or more associated IP address resources and a ‘witness’ resource and the Available Storage group are considered to be ‘critical’ cluster resource groupings and are hosted in an rhs.exe process separate from all the other cluster resources.

The Resource Hosting Subsystem (RHS) conducts periodic health checks of all cluster resources to ensure they are functioning properly. This is accomplished by executing IsAlive and LooksAlive processes which are specific to the type of resource. Examples of these are documented in the following KB article –

KB 914458 - Behavior of the LooksAlive and IsAlive functions for the resources that are included in the Windows server Clustering component of Windows Server 2003.

How often health checks are conducted is determined by the specific resource DLL or by a policy set by the cluster administrator. An example of this policy is shown in Figure 5. Should a resource fail to respond to a low-level LooksAlive check, a more in-depth IsAlive check is conducted. If a resource fails an IsAlive check, additional policies are executed until such time it is determined that a resource cannot run on a particular node in the cluster. When that point has been reached, RHS notifies the Resource Control Manager which will report the resource as Failed to the cluster service and a Failover is executed to move the Resource Group to another node in the cluster provided the default policy (Figure 8) is in effect.

Figure 8

There are times when a cluster administrator will choose not to implement the default policy shown in Figure 8 for specific ‘non-critical’ resources. This reduces instability in the cluster which could adversely impact clients connected to highly available service(s).

The IsAlive and LooksAlive health monitoring function is but a small part of what can be done with cluster resources. Figure 9 shows a listing of additional Resource DLL Entry-Point functions.

Figure 9

Note: Information on the Failover Cluster APIs can be found on MSDN.

Failure of an IsAlive call into a resource is but one way resources can become unavailable in the cluster. Other ways include:

Deadlocks in a resource DLL
Crashes in a resource DLL
RHS process itself terminates in the cluster
Cluster service fails on the node
Operating system failures (e.g. resource exhaustion)

Most of us who have been working with clusters for a long period of time understand what happens if a resource fails a critical health check. I want to spend a little time discussing resource deadlocks.

What is a resource ‘deadlock’? Basically, there are two common reasons for instability within a resource DLL. The resource DLL itself crashes (e.g. access violation in the resource DLL) or the resource fails to respond to a command in a timely fashion. Every time a call is made into a resource, a timer is started. If a response is not received within a specific period of time (configurable), the resource is considered to be deadlocked and the RHS process hosting that resource will be terminated and the resource will be placed in a newly created RHS process thereby isolating it from all the other resources running in the default rhs.exe process. When a deadlock happens, the Failover Cluster service registers an event in the cluster log. Here is an example of a deadlock occurring in the ‘Cluster Name’ resource –

000008c8.00002528::2009/06/17-20:07:57.900 WARN [RCM] ResourceControl(GET_NETWORK_NAME) to Network Name (email) returned 5910.

00000f1c.00000f28::2009/06/17-20:07:58.009 ERR [RHS] RhsCall::DeadlockMonitor: Call LOOKSALIVE timed out for resource 'Cluster Name'.

00000f1c.00000f28::2009/06/17-20:07:58.009 ERR [RHS] Resource Cluster Name handling deadlock. Cleaning current operation and terminating RHS process.

000008c8.00001cc4::2009/06/17-20:07:58.009 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'Cluster Name', gen(0) result 4.

000008c8.00001cc4::2009/06/17-20:07:58.009 WARN [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'Cluster Name' has crashed or deadlocked; marking it to run in a separate monitor.

Figure 10

Entries are also made in the Windows System Event Log. Here is an example –

06/17/2009 04:07:58 PM Error Server1.contoso.com. 1230 Microsoft-Windows-FailoverCluste Resource Control NT AUTHORITY\SYSTEM Cluster resource 'Cluster Name' (resource type '', DLL 'clusres.dll') either crashed or deadlocked. The Resource Hosting Subsystem (RHS) process will now attempt to terminate, and the resource will be marked to run in a separate monitor.

06/17/2009 04:07:58 PM Critical Server1.contoso.com. 1146 Microsoft-Windows-FailoverCluste Resource Control NT AUTHORITY\SYSTEM The cluster resource host subsystem (RHS) stopped unexpectedly. An attempt will be made to restart it. This is usually due to a problem in a resource DLL. Please determine which resource DLL is causing the issue and report the problem to the resource vendor.

Figure 11

Information on these specific Failover Cluster error messages can be found on TechNet. The information for the two events shown in Figure 11 is shown in Figure 12.

Figure 12

In Windows Server 2008 R2, RHS events are registered with Windows Error Reporting. These events can be viewed in the Action Center under Control Panel. All RHS issues will be listed under the category ‘Failover Cluster Resource Host Subsystem.’

Examining the properties of a cluster resource highlights some of the information we have been discussing. Figure 13 points out some of the pertinent properties of a resource.

Figure 13

MonitorProcessID: Indicates the Process Identifier (PID) in task manger of the rhs.exe process associated with this resource. If multiple resources have been placed in their own RHS process, it can be difficult to discern which process is associated with which resource. Examining the properties of the specific resource can help.

Note: The Process ID is not displayed by default in Task Manager. You need to add the Column to the display by selecting View in the Menu Bar and from the drop down list select Select Columns. Check the box for PID (Process Identifier).

SeparateMonitor: Indicates if the resource has been placed in a separate monitor (0:No, 1:Yes).

IsAlivePoleInterval: Default is as shown indicating it is using the default setting for this specific resource type.

LooksAlivePollInterval: Default is as shown indicating it is using the default setting for this specific resource type.

DeadlockTimeout: Default setting indicating 5 minutes.

Resource deadlock detection was actually introduced in Windows Server 2003 clusters, however it was not turned on by default. Figure 14 illustrates this.

Figure 14

Deadlock detection is turned on by default in Windows Server 2008 (RTM + R2) and cannot be disabled.

So, what is the moral of this story? It is important to understand that cluster resource deadlocks are a symptom of a larger problem. The deadlock itself is not the problem….cluster is a victim of a problem that can exist either internal to the cluster node itself or somewhere external to the cluster. Applying a logical troubleshooting methodology can help understand where the problem may exist. But, to do that requires a couple of pieces of knowledge –

Identification of the specific resource that is deadlocked.
What is the entry point that is failing?
What is the entry point trying to do?

Using the example provided in Figures 10 and 11, we can see there was a deadlock in the cluster name resource during a LooksAlive entry point. Understanding what is being evaluated for a LooksAlive process for a Network Name resource may help identify the problem which could end up being local to the node or could perhaps involve connectivity to a DNS server on the network. Referring back to KB 914458, the cluster resource DLL (ClusRes.dll) is responsible for Network Name resource health checking (IsAlive\LooksAlive tests). Some of the tests that are conducted include:

· Determining if the Network Name (NetBIOS Name) is still registered on the network stack on the node. Opening a command prompt on a node and running an nbtstat –n command to view the local NetBIOS name table, will show the registrations for cluster Network Name resources. Here is an example of a Network Name supported a Client Access Point for a File Server –

Inspecting the Parameter data for the resource in the cluster registry hive, confirms the information –

Determine the result of a DNS registration attempt (dynamic DNS is required for this test).
If the Require DNS property is set and registration fails, then the IsAlive\LooksAlive test fails.

If all DNS registrations fail and the NetBIOS name is no longer registered locally on the node, the Network Name is no longer considered reachable and the resource is placed in a Failed state. Recovery processes are initiated by the cluster service on the local node first. If local recovery fails, the Group containing the Failed Network Name resource could be moved to another node in the cluster.

What are some things that can be done to help avoid, or at least mitigate, situations where a deadlock may occur? While not set in stone, here are some of my personal recommendations:

Make sure the operating system (OS) is running with the latest service pack plus any post-service pack updates that pertain to Failover Cluster, networking or storage connectivity.
If running highly available Microsoft applications like SQL or Exchange, ensure they are updated as well.
Consult with the storage vendor and ensure the shared storage is updated and configured correctly to work in a Microsoft Failover Cluster. Most storage vendors maintain a current support matrix.
Ensure there are reliable and redundant communications paths between all nodes in the cluster.
Ensure there is reliable connectivity between all nodes in the cluster and Active Directory.
Document all Third party products that are running in the cluster and ensure they are fully updated. Third party products that interact with storage or network connectivity are always potential suspects.
Use the cluster validation process to help troubleshoot issues seen in a cluster.
If you are a Cluster Administrator, you must be aware of all changes being implemented in the corporate infrastructure to determine potential impacts on highly available services.

Hopefully, you will find this information useful. Thanks again and please come back.

Additional References:

http://blogs.msdn.com/clustering/archive/2009/06/27/9806160.aspx

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Failover Cluster Validation Firewall Error in Windows Server 2008 R2

jeffhugh — Mon, 28 Sep 2009 18:02:58 GMT

An issue involving a firewall configuration error in the cluster validation process just surfaced here in Microsoft Support so I thought I would post a quick blog in an effort to not only inform our readership, but to ‘nip this in the bud’ before we start seeing more.

After running a Windows Server 2008 R2 Failover Cluster validation report, you may see the following error –

“An error occurred while executing the test. There was an error verifying the firewall configuration. An item with the same key has already been added”

The error, as is, does not provide a clear direction to take when trying to troubleshoot. Thanks to the efforts of Cluster Product Group, the source of the issue was identified and a quick data collection process can be executed to help determine the ‘root’ cause.

The firewall configuration error is reported if any of the network adapters across the cluster nodes being validated have the same Globally Unique Identified (GUID). This can be determined by running the following WMI query on each node in the cluster and comparing the results. I chose to run the query inside PowerShell to display sample data in a formatted list-

GetWMI Win32_NetworkAdapter | fl Name,GUID

The sample output above shows the information associated with the three physical network adapters that exist in one of the nodes in my cluster. After the data is gathered from each node in the cluster, you just need to compare it and identify the duplicate GUID information.

The next logical question is, “How does one find themselves in this predicament?” In the cases we have encountered thus far, the cluster nodes were being deployed in an unsupported manner. In each case an ‘image’ was being used to deploy the nodes. We discovered that the operating system image was not properly prepared before being deployed by, for example, running sysprep.

Hopefully this information will be useful and will help avoid further occurrences of this issue. Thanks again and please come back.

Additional References:

Failover Cluster Step-by-Step Guide: Validating hardware for a Failover Cluster

KB 943984: The Microsoft Policy for Windows Server 2008 Failover Clusters

Deployment Tools Technical Reference

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Expected Snapshot Merge Behavior for a Highly Available VM

jeffhugh — Fri, 29 May 2009 15:22:03 GMT

Hello my name is Sean Dwyer, and I'm a Support Escalation Engineer working in the Windows CORE team here at Microsoft.

While working with snapshots that are attached to a highly available VM in a Cluster, you may notice after deleting a snapshot, it does not merge as expected. You'll notice the merge process almost immediately ends and the VM begins to restart.

Let's explore a scenario I ran myself into the other day and then I'll explain why you may see this behavior.

Fig. 1.

Here I have configured a HA-VM of XP SP2 in Fig 1. and it is Online and happy, within the Cluster.

Fig. 2.

Switching to the Hyper-V Management MMC in Fig. 2, I've decided that my testing is over with this VM, and I need to clean up the snapshots. The 'Now' state is where I want the VM to remain, so I'm going to issue a Delete Snapshot Subtree command.

Fig. 3.

I choose 'Yes', and the snapshots are removed from the VM in Fig. 3.
Next, I choose to shut down the VM in Fig. 4, as I want to commit the merge now, and not wait until later.

Fig. 4.

The Merge process starts, but immediately finishes and the VM is restarting as seen in Fig. 5 and Fig. 6.

Fig. 5.

Fig. 6.

The merge didn't complete! What just happened?

Failover Cluster is doing its job!

When deleting a snapshot for a VM that is configured as a Highly Available Resource, you're taking action against the VM outside of the Cluster Management UI, and therefore, the Cluster service considers any shutdown events, or reboots, a failure of the resource and takes appropriate action.

In my example, the default behavior is restarting the VM as shown in Fig.7.

Fig. 7.

The end result is that the snapshot merge process has NOT taken place and I've still got AVHDs out there shown in Fig. 8.

Fig. 8.

So, what do we do, to get the Snapshots merged successfully? Easy!

We need to configure the Clustered VM Resource to allow the Snapshot Merge process to complete without being interrupted.

Open up the Properties of the VM, and then select the Offline Actions Tab.

You'll note, in Fig. 9, the default behavior is to Save the VM.

Fig. 9.

In Fig. 10, let's change this default behavior temporarily to Shutdown.

Fig. 10.

Once the change has been made, you can then choose to shutdown the VM either through Hyper-V Manager or through the Clustering UI, and the Snapshot Merge process will begin.

Once the Merge completes, reset the behavior of the VM in the Offline Actions Tab for the VM back to Save as shown in Fig. 11.

Fig. 11.

Once you start the VM, you'll be back in business!

I hope this blog post will help you continue to use our Clustering and Virtualization products successfully!

Sean Dwyer
Support Escalation Engineer
Microsoft Enterprise Platforms Support

Top Issues for Microsoft Support for Windows Server 2008 Hyper-V (Q3)

jeffhugh — Wed, 27 May 2009 17:24:55 GMT

It is time to update everyone on the issues our support engineers have been seeing for Hyper-V for the past quarter. The issues are categorized below with the top issue(s) in each category listed with possible resolutions and additional comments as needed. I think you will notice that the issues for Q3 have not changed much from Q1\Q2. Hopefully, the more people read our updates, the fewer occurrences we will see for some of these and eventually they will disappear altogether. There will probably be one more blog for the Q4 results. Additionally, I would like to mention that we are highly recommending the installation of Windows Server 2008 Service Pack 2 on all servers running the Hyper-V Role.

Deployment\Planning

Issue #1

Customers looking for Hyper-V documentation.

Resolution: Information is provided on the Hyper-V TechNet Library which includes links to several Product Team blogs. Additionally, the Microsoft Virtualization site contains information that can be used to get a Hyper-V based solution up and running quickly.

Installation Issues

Issue #1

A customer was experiencing an issue on a pre-release version of Hyper-V.

Resolution: Upgrade to the release version (KB950050) of Hyper-V.

Issue #2

After the latest updates off Windows Update are installed or KB950050 is installed, virtual machines fail to start with one of the following error messages:

An error occurred while attempting to chance the state of the virtual machine vmname .
‘ vmname ’ failed to initialize.
Failed to read or update VM configuration.

An error occurred while attempting to change the state of virtual machine vmname .
" VMName " failed to initialize
An attempt to read or update the virtual machine configuration failed.
" VMName " failed to read or update the virtual machine configuration: Unspecified error (0x80040005).

Cause: This issue occurs because virtual machine configurations that were created in the beta version of the Hyper-V are incompatible with later versions of the Hyper-V.

Resolution: Perform the steps documented in KB949222.

Issue #3

After the Hyper-V role is installed, a customer creates a virtual machine but it fails to start with the following error:

The virtual machine could not be started because the hypervisor is not running

Cause: Hardware virtualization or DEP was disabled in the BIOS.

Resolution: Enable Hardware virtualization or DEP in the BIOS. In some cases, the server may need to be physically shutdown in order for the new BIOS settings to take effect.

Virtual Devices\Drivers

Issue #1

Synthetic NIC was listed as an unknown device in device manager.

Cause: Integration Components needed to be installed.

Resolution: Install Integration Components (IC) package in the VM.

Issue #2

Stop 0x00000050 on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: This issue can occur if a Hyper-V virtual machine is configured with a SCSI controller but no disks are attached.

Resolution: Perform the steps documented in KB969266.

Issue #3

Stop 0x0000001A on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: Vid.sys

Resolution: Install hotfix KB957967 to address this issue.

Snapshots

Issue #1

Snapshots fail to merge with error 0x80070070

Cause: Low disk space.

Resolution: Free up disk space to allow the merge to complete.

Issue #2

Snapshots were deleted

Cause: The most common cause is that a customer deleted the .avhd files to reclaim disk space (not realizing that the .avhd files were the snapshots).

Resolution: Restore data from backup.

For more information on Snapshots, please refer to the Snapshot FAQ: http://technet.microsoft.com/en-us/library/dd560637.aspx.

Issue #3

Snapshots were lost

Cause: Parent VHD was expanded (not supported). If snapshots are associated with a virtual hard disk, the parent vhd file should never be expanded. This is documented in the Edit Disk wizard:

Resolution: Restore data from backup.

Integration Components

Issue #1

A Windows 2000 (SP4) virtual machine with the Integration Components installed may shut down slowly.

Cause: This problem is caused by a bug in the Windows Software Trace Pre-Processor (WPP) tracing macro (outside of Hyper-V).

Resolution: KB959781 documents the workarounds for this issue on Server 2008.

Issue #2

Attempting to install the Integration Components on a Server 2003 virtual machine fails with the following error:

Unsupported Guest OS

An error has occurred: The specified program requires a newer version of Windows.

Cause: Service Pack 2 for Server 2003 wasn’t installed in the virtual machine.

Resolution: Install SP2 in the Server 2003 VM before installing the integration components.

Virtual machine State and Settings

Issue #1

You may experience one of the following issues on a Windows Server 2008 system with the Hyper-V role installed or Microsoft Hyper-V Server 2008:

When you attempt to create or start a virtual machine, you receive one of the following errors:

· The requested operation cannot be performed on a file with a user-mapped section open. ( 0x800704C8 )

· ‘VMName’ Microsoft Synthetic Ethernet Port (Instance ID

{7E0DA81A-A7B4-4DFD-869F-37002C36D816}): Failed to Power On with Error 'The specified network resource or device is no longer available.' (0x80070037).

· The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3)

Virtual machines disappear from the Hyper-V Management Console.

Cause: This issue can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files.

Resolution: Perform the steps documented in KB961804.

Issue #2

Creating or starting a virtual machine fails with the following error:

'General access denied error' (0x80070005).

Cause: This issue can be caused by the Intel IPMI driver.

Resolution: Perform the steps documented in KB969556.

Issue #3

Virtual machines have a state of "Paused-Critical"

Cause: Lack of free disk space on the volume hosting the .vhd or .avhd files.

Resolution: Free up disk space on the volume hosting the .vhd or .avhd files.

High Availability (Failover Clustering)

Issue #1

How to configure Hyper-V on a Failover Cluster.

Resolution: A step-by-step guide is now available which covers how to configure Hyper-V on a Failover Cluster.

Issue #2

Virtual machine settings that are changed on one node in a Failover Cluster are not present when the VM is moved to another node in the cluster.

Cause: The "Refresh virtual machine configuration" option was not used before attempting a failover.

Resolution: When virtual machine settings are changed on a VM that’s on a Failover Cluster, you must select the ‘Refresh virtual machine configuration’ option before the VM is moved to another node. There is a blog that discusses this.

Backup (Hyper-V VSS Writer)

Issue #1

You may experience one of the following symptoms if you try to backup a Hyper-V virtual machine:

· If you back up a Hyper-V virtual machine that has multiple volumes, the backup may fail. If you check the VMMS event log after the backup failure occurs, the following event is logged:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin

Source: Microsoft-Windows-Hyper-V-VMMS

Event ID: 10104

Level: Error

Description:

Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

· The Microsoft Hyper-V VSS Writer may enter an unstable state if a backup of the Hyper-V virtual machine fails. If you run the vssadmin list writers command, the Microsoft Hyper-V VSS Writer is not listed. To return the Microsoft Hyper-V VSS Writer to a stable state, the Hyper-V Virtual Machine Management service must be restarted.

Resolution: An update (KB959962) is now available to address issues with backing up and restoring Hyper-V virtual machines.

Issue #2

How to backup virtual machines using Windows Server Backup

Resolution: Perform the steps documented in KB958662.

Virtual Network Manager

Issue #1

Virtual machines are unable to access the external network.

Cause: The virtual network was configured to use the wrong physical NIC.

Resolution: Configure the external network to use the correct NIC.

Issue #2

Network connectivity issues

Cause: NIC teaming software

Resolution: Remove the NIC teaming software. Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Issue #3

Customers inquiring if Hyper-V supports NIC Teaming.

Resolution: Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Hyper-V Management Console

Issue #1

How to manage Hyper-V remotely.

Resolution: The steps to configure remote administration of Hyper-V are covered in a TechNet article. John Howard also has a very thorough blog on remote administration.

Import/Export

Issue #1

Importing a virtual machine may fail with the following error:

A Server error occurred while attempting to import the virtual machine. Failed to import the virtual machine from import directory <Directory Path>. Error: One or more arguments are invalid (0x80070057).

Resolution: Perform the steps documented in KB968968.

Miscellaneous

Issue #1

You may experience one of the following issues on a Windows Server 2003 virtual machine:

· An Event ID 1054 is logged to the Application Event log:

Event ID: 1054
Source: Userenv
Type: Error
Description:
Windows cannot obtain the domain controller name for your computer network. (The specified domain either does not exist or could not be contacted). Group Policy processing aborted.

· A negative ping time is displayed when you use the ping command.

· Perfmon shows high disk queue lengths

Cause: This problem occurs when the time-stamp counters (TSC) for different processor cores are not synchronized.

Resolution: Perform the steps documented in KB938448.

As always, we hope this has been informative for you.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Active Route Gets Removed on Windows 2008 Failover Cluster IP Address Offline

jeffhugh — Tue, 26 May 2009 21:31:50 GMT

We have received calls adding static routes on Windows Server 2008 Failover Clustering nodes and wanted to pass along some important information regarding this. The issue is that when you add a static persistent route to a network adapter that is on a Windows Server 2008 Failover Cluster and take a Clustered IP Address offline (or move it to another node), the “Active” route is removed and no connections can be made using this route even though it still shows as persistent. Once you bring the Clustered IP Address back online, the active route is returned.

First, I want to mention that the networking architecture in 2008 Failover Clustering has been rewritten from the ground up and now we have a our own internal route listings as well as our own adapter (Microsoft Failover Cluster Virtual Adapter). I will not into the specifics of it here, but you can read more about it in the blog “What is a Microsoft Failover Cluster Adapter anyway?”.

On to the problem and resolution. As a little setup, here is the configuration that I want to discuss.

ClusterNode1

Physical IP Address: 10.44.60.4

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

ClusterNode2

Physical IP Address: 10.44.60.3

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

Failover Cluster Virtual IP Address

IP Address: 10.44.60.6

I also have a backup server that I use to create backups using an IP Address of 10.51.0.1 and subnet mask 255.255.0.0 that will use the same default gateway above. Most Network Administrators would use the following ROUTE.EXE command to add a persistent static route to the local tables so that a connection can be made.

route -p add 10.51.0.0 mask 255.255.0.0 10.44.60.1

So with everything online (including the Failover Cluster Virtual IP Address) on ClusterNode1, I can do a ROUTE PRINT command to display my IP Address version 4 table and see this. As a side note, I am just pulling the necessary information from the Route Table.

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276    <<---
10.44.60.4           255.255.255.255 On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.60.6           255.255.255.255 On-link       10.44.60.4    276    <<--- Clustered IP Address
10.44.255.255        255.255.255.255 On-link       10.44.60.4    276   <<---
10.51.0.0            255.255.0.0      10.44.60.1    10.44.60.4     21    <<--- Static Route added
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255 On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                 <<--- Persistent Route added
===========================================================================

As long as the Clustered IP Address of 10.44.60.6 is online on this node, all is well. However, if I were to take the 10.44.60.6 IP Address offline, things change.

C:\>cluster res "IP Address 10.44.60.6" /offline
Taking resource ''IP Address 10.44.60.6'' offline...
Resource                 Group         Node             Status
--------------------     ----------    --------------- ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Offline

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276    <<---
10.44.60.4           255.255.255.255 On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.255.255        255.255.255.255 On-link       10.44.60.4    276   <<---
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255 On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                 <<--- Persistent Route added
===========================================================================

Notice here that the Clustered IP Address 10.44.60.6 as well as the 10.51.0.1 “Active” route is removed. Because the 10.51.0.0 route is removed, connectivity to the backup server is lost. If you bring the Clustered IP Address 10.44.60.6 online again, the “Active” routes are re-populated again and connectivity to the backup server is restored.

C:\>cluster res "IP Address 10.44.60.6" /online
Bringing resource ''IP Address 10.44.60.6'' online...
Resource                 Group         Node             Status
--------------------     ----------    --------------- ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Online

According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.

The ROUTE.EXE command has additional parameters of METRIC and INTERFACE that you would need to specify that will bind the route to the card itself.

C:\>route /?

Manipulates network routing tables.

ROUTE [-f] [-p] [command [destination]

[MASK netmask] [gateway] [METRIC metric] [IF interface]

interface the interface number for the specified route.

METRIC specifies the metric, ie. cost for the destination.

So what you need to do first is determine what the interface is so that we can bind the route to it. When doing the ROUTE PRINT or NETSH command, it will give you the interfaces at the top first. Something similar to this:

C:\>route print

IPv4 Route Table

===========================================================================

Interface List

23 ...00 15 5d 4a ac 06 ...... Local Gigabit Controller

19 ...00 15 5d 4a ac 01 ...... Local Gigabit Controller #2

18 ...00 15 5d 4a ac 00 ...... Local Gigabit Controller #3

===========================================================================

-or-

C:\>netsh int ipv4 show int

Idx Met MTU State Name

--- --- ----- ----------- -------------------

18 50 4294967295 connected Local Gigabit Controller #3

19 5 1500 connected Local Gigabit Controller #2

23 5 1500 connected Local Gigabit Controller

I can go into the Network and Sharing Center if I have to to see which card is on this network. In my particular case, the “Local Gigabit Controller #3” is the one I want to use. So to get my persistent route to stay even though the Clustered IP Address goes offline, my command would be below. Please note that the METRIC is not needed as a requirement of the command.

route -p add 10.51.0.0 mask 255.255.0.0 0.0.0.0 metric 276 if 18

Now, the “Active” route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline. A good rule of thumb, moving forward, would be that if you are adding a static persistent route, start specifying the 0.0.0.0 and the interface as this is the proper supported commands from a networking perspective. This will result in the proper functioning no matter if Failover Clustering is configured or not.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Recovering a Deleted Cluster Name Object (CNO) in a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 27 Apr 2009 14:32:49 GMT

Greetings once again from the support trenches here on the CORE team. I want to talk a bit about a Windows Server 2008 Failover Cluster issue that appears to be on the rise. What we are seeing is the Computer Object for the Cluster Name (a.k.a. Cluster Name Object (CNO) being removed from Active Directory resulting in the Cluster Name no longer being able to function properly. This does not happen automatically. It requires some sort of human interaction either by consciously going into AD and deleting the object or running some script (process) that deletes it. However this is being done, it appears to us that the implications are not fully understood and there is no quick recovery from this. In this blog, I hope to provide information that will help avoid this scenario from happening within your organization. Along the way, I want to provide some 'value-add' information by discussing how the cluster computer objects relate to each other.

The first step to preventing this from happening in your organization is to be sure there is a clear understanding of the cluster security model in Windows Server 2008. Rather than spend a whole lot of time and space here rehashing what is already publicly available, I refer you to the following:

KB 947049: Description of the Failover Cluster Security Model in Windows Server 2008.

Failover Cluster Step-by-Step Guide: Configuring Accounts in Active directory

After reviewing the materials, you should have an understanding of how security works in Windows Server 2008 Failover Clusters and an appreciation for the importance of not removing (or disabling) the Computer Objects created in Active Directory by the cluster. By default, the Computer Objects created by the cluster are all placed in the Computers container. These can be relocated to another OU, or even pre-staged in an OU before the cluster is created. If pre-staging, be sure to review the requirements in the Step-by-step Guide already mentioned. As an example (Figure 1), I created a Cluster OU and moved the cluster nodes and their associated objects into the OU.

Figure 1

You may want to consider implementing a similar practice in your organization as it groups the cluster objects together thereby reinforcing the idea that this grouping of objects is 'special' in some way.

Before moving forward and discussing the actual recovery process, I want to spend a little time reviewing the cluster 'family tree' to help you gain an understanding of how cluster objects are related. To illustrate, I will use a cluster named W2K8-CLUS (Figure 2) in the CONTOSO domain.

Figure 2

This cluster is located in the Cluster OU shown in Figure 1. Using Regedit.exe, I open the cluster registry hive and inspect the properties for the cluster. I can see the name of the cluster and the resource GUID for the Cluster Name.

Figure 3

Expanding the Resource GUID corresponding to the Cluster Name, I inspect additional properties for the resource. Selecting the Parameters entry displays the ObjectGUID for the cluster Computer Object in Active directory (Figure 4).

Figure 4

In Figure 5, we see the attribute in Active directory (must enable Advanced Features before the Attribute Editor tab is visible). You can also use ADSIEdit to view the same information.

Figure 5

The Cluster Name Object (CNO) functions as the primary security context for the cluster. The CNO is responsible for creating any additional Computer Objects (Virtual Computer Objects (VCO)) associated with the cluster. These Computer Objects represent Network Name resources in a cluster. A Network Name resource is created as part of a Client Access Point (CAP). Each Computer Object created by a cluster CNO contains an Access Control Entry (ACE) for the CNO on the Access Control List (ACL) for the object. The CNO is also responsible for synchronizing the password for each VCO in the domain. The VCOs associated with a particular CNO can be determined either by manually inspecting the ACL for each VCO in AD, or the information can be obtained in the cluster registry.

Opening the cluster registry hive and inspecting the properties of the Cluster Name resource, we can see an entry called ObjectGUIDS. This is a listing for each Computer Object created by the CNO in Active directory. In Figure 6, I have four Computer Objects in Active Directory associated with this cluster.

Figure 6

One of them is a Computer Object (VCO) associated with the CAP representing a highly available Print Server (CONTOSO-PS1) in this cluster (Figure 7).

Figure 7

Well, there you have it…the cluster family tree.

So, what happens if the Cluster Name Object is deleted from Active Directory? A few important things –

· The Cluster Name, if Online, will stay Online but will fail to come Online again if the resource is cycled (it will be placed in a Failed state). This will prevent being able to connect to the cluster remotely when trying to administer the cluster.

· The security context for the cluster is lost. This prevents the passwords for all associated VCOs from being synchronized within the domain. Also, any user, service or other process needing permission to access cluster objects will fail to be authenticated.

· No more CAPs can be created in the cluster.

Besides the items listed above, there are other indications of problems. The Cluster Name resource in the Cluster Core Resources group will be in a Failed state. Attempts to bring the resource Online will generate a pop-up error (Figure 8)

Figure 8

A FailoverClustering error (Event ID 1207) will be registered in the System Log (Figure 9).

Figure 9

The cluster log will report a failure to locate the CNO Computer Object in Active Directory (Figure 10)

Figure 10

It is, therefore, very important the CNOs Computer Object in the domain not be deleted.

How does one recover from this? The supported way(s) to recover an Active Directory object that has been accidentally, or intentionally, deleted are described in the following articles and will not be covered in detail here–

KB840001: How to restore deleted user accounts and their group memberships in Active Directory

TechNet Content - Recovering Active Directory Domain Services

Additionally, there are 3^rd party solutions that can be used to protect Active Directory objects and\or recover them if deleted. Finally, as a last ditch effort, and when there is no other alternative, there is a free utility called ADRestore (32-bit only) that can be used to recover the Computer Object associated with the CNO. Please review the following information before deciding to use this utility –

Microsoft Supportability Newsletter – Using ADRestore tool to restore deleted objects

Either of these methods can be used, but they may end up being time consuming, expensive or both.

Once the Computer Object has been recovered from Active Directory, the Repair Active Directory object action can be used to restore functionality in the cluster (Figure 11).

Figure 11

Note: The logged on user that will perform the Repair action must have rights to administer the cluster and must have the right to Reset Passwords in the domain.

I personally believe ‘an ounce of prevention is worth a pound of cure.’ To that end, my top recommendation is to implement the steps outlined in the section Preventing unwanted deletions in the TechNet Content already mentioned above. Beginning with Windows Server 2008, objects in Active Directory, such as the Computer Object shown here (Figure 12), can be protected from accidental deletion by simply checking a box – Protect object from accidental deletion.

Figure 12

With this ‘guard’ in place, when an object is selected for deletion, the first pop-up is presented (Figure 13)

Figure 13

If Yes is selected, the next error is presented to the user (Figure 14) thus preventing deletion.

Figure 14

If this isn’t enough, there is more help coming in Windows Server 2008 R2. Domain Services in Windows Server 2008 R2 will include an optional feature called Active Directory Recycle Bin. This feature is not enabled by default and must be added. Details about the feature can be found on TechNet

TechNet Content – Active Directory Recycle Bin Step-by-Step Guide

That about wraps it up for this installment. As usual, we hope this information is useful. Come back and visit.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Windows 2008 Failover Cluster Validation Fails on ‘Validate SCSI-3 Persistent Reservation’

jeffhugh — Wed, 15 Apr 2009 15:02:42 GMT

We’ve been seeing a lot of calls lately from customers who are running the validation that’s required prior to installing and configuring failover clustering, and the validation fails in the ‘Storage’ portion of the tests. The specific error seen in the validation report is:

If you click on the ‘Validate SCSI-3 Persistent Reservation’ link in the report. It will take you to the detail section.

Validate SCSI-3 Persistent Reservation

Validate that storage supports the SCSI-3 Persistent Reservation commands.

Validating Cluster Disk 0 for Persistent Reservation support

Registering PR key for cluster disk 0 from node node1.cluster.com

Failed to Register PR key for cluster disk 0 from node node1.cluster.com status 1

Cluster Disk 0 does not support Persistent Reservation

If you dig a little deeper, you can also look at the ValidateStorage.txt file that’s located in the Windows\Cluster\Reports directory.

00000fd4.00000fd8::15:56:45.857 CprepDiskPRUnRegister: Enter CprepDiskPRUnRegister: ulSignature 0xd0426bb2
00000fd4.00000fd8::15:56:45.857 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:49.977 CprepDiskPRUnRegister: Failed to unregister PR key, status 1117
00000fd4.00000fd8::15:56:49.977 CprepDiskPRUnRegister: Exit CprepDiskPRUnRegister: hr 0x8007045d
00000fd4.00000fd8::15:56:54.097 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:54.097 CprepDiskIsPRPresent: Failed to read PR reservations, status 0
00000fd4.00000fd8::15:56:54.097 CprepDiskIsPRPresent: Exit CprepDiskIsPRPresent hr 0x0, Present 0
00000fd4.00000fd8::15:56:54.097 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:54.097 DoIoctlAndAlloc: ControlCode 0x70050, retCode 1, status 122
00000fd4.00000fd8::15:56:54.097 CprepDiskGetArbSectors: Exit CprepDiskGetArbSectors: hr 0x0, SectorX 11 SectorY 12

So what is a “Persistent Reservation” (PR) and why should you care? A PR is a SCSI command, which clustering uses to protect LUN’s. When a LUN is reserved, no other computers on the SAN can access the disk, except the ones cluster controls. This is important to protect other machines from accessing the disk and corrupting the data on the disk.

Validate is a functional test tool that verifies that your storage supports all the necessary SCSI commands that clustering requires. It is critical that Validate tests pass, for your cluster to work correctly. The Storage tests are by far the most important, they should not be dismissed!

If you are reading this blog, then the bad news is that Validate has probably identified that your storage does not support Persistent Reservations, and is not compatible with Windows Server 2008 Failover Clustering. The good news is that it most likely will work, you just have to do a few things! All storage vendors and almost all current shipping models support Win2008 Failover Clustering, but many require firmware updates or configuration settings. Microsoft has been working closely with partners such as HP, EMC, IBM, NetApp, HDS, Fujitsu, Lefthand, Equallogic, Xiotech, NEC, LSI, Infortrend, 3PAR, Intransa, FalconStor, Nexsan, and even more… and they all work!

First things first, call your storage vendor and ask them if your storage is compatible AND configured for use with Windows Server 2008 Failover Clustering.

There are two things to verify:

Correct firmware version
Correct configuration settings

The storage vendor is really the right person to tell you how to correctly configure their arrays to work with Failover Clustering, so they are the right source. We can’t post the specific steps for each vendor but as we become aware of publicly available documentation from the SAN vendors, we’ll add them to this post as they start being published.

HP has a publicly available document detailing the steps needed to get SCSI-3 PR’s to work specific to their hardware (pages16-18)

Implementing Microsoft® Windows® Server 2008 Service Pack 2 beta on HP ProLiant servers

With any of these vendor links, although they may contain steps to resolve the PR problem, we still strongly recommend being directly engaged with the vendor to verify with them that these storage configuration changes are current, appropriate for your environment and hardware, and non destructive to your data. Microsoft makes no guarantees on any of the 3rd party links we are providing. They are solely intended to have information on hand to discuss with your particular vendor.

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Maximizing Limited Storage Resources in a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 09 Mar 2009 16:10:53 GMT

We have encountered scenarios where customers are implementing Windows Server 2008 Failover Clusters and they want to make quite a few services and applications highly available but, they are not able to purchase additional Storage to facilitate this. Or, it may be that another business within their organization has higher priority for existing storage assets and their 'lower' priority cluster will just have to make do with what storage has already been allocated. What is a cluster administrator to do under such circumstances?

In this blog, as an example, I will show you how to configure a highly available File Server Group so that it can also be used to support highly available Print Services. I am using File and Print Services because that is the most common scenario we see in customer environments. To implement this configuration we will be using the concepts explained in KB947050: Advanced resource configuration in Windows Server failover clusters.

Note: Even though these procedures are fully supported, the preference is to use dedicated storage for each highly available service or application. This allows the built-in wizard-based processes to be used thus ensuring the correct configuration. It is recommended that customers review their current storage utilization within each cluster and discuss with their local Storage Team to see if it would be possible recover any excess storage space on current LUNs which could then be used to create new LUNs.

The starting point will be an already configured File Server application (CONTOSO-FS1) providing some shared folders for a couple of business groups within an organization.

We will be using the storage in this file Server group to also host the files needed for a highly available Print Server. The procedures we will use will be executed outside of the normal wizard-based process for configuring a highly available Print Server. When using the normal wizard-based process, a highly available Print Server application looks something like this:

The resources in the group include a Client Access Point (CAP) (CONTOSO-PS1), a Print Spooler resource and a piece of storage for storing print jobs and printer drivers. Additionally, since this was configured using the wizard-based process, we also have the correct group type configured [101] so we will have access to the Print Management interface via the Failover Cluster Management snap-in:

The process we will use will not provide direct access to the Print Management snap-in from inside Failover Cluster Manager and, we will have to take that into account. Let's get started....

Since we are outside of the normal wizard-based process, the cluster will not be able to verify the installation of any prerequisite Roles and\or Features, so the first thing is to ensure we install the Print Server Role on all nodes in the cluster.

Once the Print Server role is installed, the Print Management snap-in is listed under Administrative Tools.

The Print Management snap-in provides management capabilities for print services running either in the context of the local node or a highly available Print Server using a configured Client Access Point (CAP).

With the Print Server role installed, we can move forward with the manual configuration of the resources we will need to place in the highly available File Server group.

The Print Spooler resource requires a dependency on a Network Name and Physical disk resource. Both of these resources already exist and are Online in the group so we could use the existing resources. However, we will choose to create a new Client Access Point (CAP) (CONTOSO-PS2) so the users can connect using another Network Name. In the Actions pane to the right, select Add a resource action and select a Client Access Point.

This starts a wizard where we create a NetBIOS name and IP Address (IP address information would not be requested if using DHCP).

Once the resource is created, bring it Online.

Next, create a Print Spooler resource.

The Print Spooler resource is created and placed in an Offline state. The resource is Offline because additional configuration is required before it can be brought Online. If the resource were brought Online at this point, it would fail and would take the whole group down because a failover would be initiated. This would disrupt any user connections to shared folders (more on this later).

To complete the configuration, Right-click on the resource and select Properties. On the General tab you can change the display name for the resource (optional) but you must enter a path to a folder on the storage where spooled print jobs and printer drivers can be stored. I created a Spooler directory on the storage so I enter the path information.

Next, select the Dependencies tab and add dependencies for the storage and the CAP.

Verify the default setting for the Policies and Advanced Policies tabs and then click OK. Bring the Print Spooler resource Online.

Test failover to other nodes in the cluster to be sure you have high availability.

Earlier, I mentioned we would not be able to manage highly available printers in the Failover Cluster Management snap-in when we configure a Print Spooler resource using the method above. If we were to open the Print Management snap-in, accessible in Administrative Tools, we would only see the local cluster node listed.

We have to manually add the new CAP we created for the Print Spooler resource.

Once we complete this action, both the local and the highly available Print Server will be visible in the management interface.

Looking at the final result where we have a single grouping of resources which consist of both File Server and Print Server resources, we need to consider what happens in case of any failure?

This is a critical question that needs to be answered because if the default behavior is implemented which is "if restart is unsuccessful, fail over all resources in this service or application", then all the resources in the application group will be taken Offline and moved to another node in the cluster and brought back Online.

In this scenario then, if the Print Spooler resource were to fail and could not be restarted on the node that owns it, the entire group, which includes the File Server resource and its associated shared folders, would be taken Offline and moved to another node in the cluster. This will interrupt all client connections to the shared folders until they are brought back Online on another node and the connections re-established. A decision may be required, in this situation, where perhaps it is more important that access to the File Server be maintained while the Print Spooler is in a Failed state. To accomplish this, a cluster administrator would have to modify the setting by Unchecking the box on both the Print Spooler resource and the associated Client Access Point so that failures of either of these will not result in a failover. You obviously would not do this for the disk resource because that is the single common resource being used by both services and would want that to failover.

To ensure the modified settings result in the desired behavior, we can simulate failures on the modified resources and observe the results. Here, I am simulating a failure on the Client Access Point for the Print Server.

After a single successful restart of the resource, execute another simulated failure and the resource will go into a Failed state but will not force a failover of the group.

So there you have it. A method for maximizing existing resources in a cluster. Before we wrap up, I want to emphasize again that the preferred method would be to have dedicated storage for each highly available application or service and not try to multi-purpose current storage.

I hope this information proves to be useful to someone and keep the cards and letters coming.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

How to Configure Multiple Instances of Distributed Transaction Coordinator (DTC) on a Windows Server Failover Cluster 2008

tomac — Wed, 18 Feb 2009 03:45:35 GMT

What we will be going over in this blog is one of the improvements of Distributed Transaction Coordinator (DTC) on a Windows Server Failover Cluster 2008. The Distributed Transaction Coordinator is a transaction manager which permits client applications to include several different sources of data in one transaction and which then coordinates committing the distributed transaction across all the servers that are enlisted in the transaction.

In versions of Failover Clusters from previous operating systems such as Windows Advanced Server 2000 and Windows Server 2003, it only allowed you to configure a single instance of DTC on a Cluster. If the DTC was in a group that was active on Node B, it would act as the transaction manager for the whole Cluster. Then the application and resource manager on Node A whenever needed would call the DTC proxy. The DTC proxy on Node A would then forward all MS DTC calls to the DTC on Node B.

DTC Architecture in Windows Advanced Server 2000 and Windows Server 2003 Failover Clusters

If Node B fails, then the DTC on Node A would take over. It reads the DTC log file on the shared disk, performs recovery, and then serves as the transaction manager for the entire Cluster.

Unlike the previous operating systems with the improvements made on a Windows Server Failover Cluster 2008, you are now allowed to have multiple instances of DTC resources all active at the same time referred to as Active-Active DTC. This same principal can be applied to services or applications such as SQL Server, Exchange, MSMQ or any application that needs to make a distributed transaction API call.

As you can see, we have 3 mechanisms of associating a DTC resource with a service or application:

· 1^st – Is a programmatic way, by using a function calls such as DtcGetTransactionManagerEx.

· 2^nd – Command line configuration, by using MSDTC.exe from a command line interface (CLI).

· 3^rd – Administration User Interface (UI), by using DCOMCnfg.exe.

By default, each system uses its local Distributed Transaction Coordinator (DTC) transaction manager which is installed by the operating system for initiating and coordinating transactions. However, you can configure your computer to use the DTC transaction manager of another system as the default transaction coordinator. The DTC transaction manager on the specified system is used as the transaction coordinator whenever a client on the local system begins a DTC transaction and that client does not explicitly specify a transaction coordinator.

DTC Architecture in Windows Server 2008 Failover Cluster

The default transaction coordinator coordinates all transactions that are initiated by COM+ or any other client that uses transactions. The default transaction coordinator also serves as the transaction coordinator for all resource managers on the local system that enlist in any DTC transaction.

So in theory, you can have an application which requires DTC that can be Clustered or non-Clustered. Some applications use a DtcGetTransactionManagerEx API function call as listed in the DTC Programmer's Reference documentation by passing in the Cluster resourceId. You would need to configure the application to specifically map to a particular DTC instance; otherwise if it has to fallback it would use the following rules:

If there is a DTC resource instance that is part of that Cluster resource, it will be used.
If not, and there is a Cluster default DTC instance available, it will be used.
If there are no Cluster instances available, then a local DTC instance will be used with a notification of a potential loss of availability.

In some cases, you may have a legacy application which doesn’t use the new parameters available in DtcGetTransactionManagerEx in which case,

The application will pick the Cluster default if present.
If Cluster default is not present it will use the local DTC.

To enable legacy applications to optimally pick the DTC instance and application that is in the same Cluster group, you can use the command line to configure the application to pick a particular DTC instance using the syntax below.

msdtc.exe -tmMappingSet -name
[-exe <pathname> | -service <full service name> | -complusappID <app ID>]
[-local|-ClusterResourceName <resource name>]

Then the legacy DtcGetTransactionManagerEx call will pick the configured DTC instance in the same group as the application instead of Cluster default. See some examples listed below.

As in previous operating systems, it is not necessary to reset the DTC log file to reclaim disk space within the file because the DTC does this automatically. When a transaction has successfully been committed or aborted, the DTC notes that the log records are no longer needed and it eventually reuses the space for other transactions.

Keep in mind that resetting the MSDTC log will cause all knowledge of currently processed transactions to be lost, and your RMs (Resource Manager) will have to be manually resolved. Resetting the log isn't required under any normal operation, unless they were corrupted or lost. Using the CLI msdtc -resetlog still works in Windows Server 2008, but it will only reset the local MSDTC log file.

Now with the improvements of Active-Active DTC from the command prompt, you should be using syntax below for resetting each individual Clustered DTC resource instance whenever needed:

msdtc -resetclustertmlog {guid}

- or -

msdtc -resetclustertmlog <resource_name>

Where {guid} is the GUID of the DTC resource (this is the name of the folder that that Cluster log resides in on a shared disk), and <resource_name> is the name of the MSDTC resource listed in the Failover Clustering interface.

For further information on some tools used for troubleshooting DTC you can visit any of the following links:

o DtcTrace, TraceFmt, MSDTCVTr, DtcPing, DtcTester and WinRM.

The steps listed below are for creating MSDTC in any group such as SQL Server, Exchange, MSMQ or any application that needs to make a distributed transaction API call.

Clustered MSMQ Group in Failover Cluster Management snap-in

One way of adding a DTC resource to a group, is by simply running through the High Availability Wizard to setup DTC to be highly available. Then just right-click on DTC and move it to whatever group you want to be in as seen in Figure 1.0 (below).

NOTE: Doing it this way means, all resources tied to DTC will move as well. So changes on the Dependencies tab for DTC would need to be made, if you wish to eliminate duplicate resources (such as Network Name, IP Address and Disk) in the particular service or application DTC moved into. You also need to make sure you go into Windows Firewall in Control Panel and Change setting by enabling the resources necessary on the Exceptions tab.

Figure 1.0

Another way of adding a DTC resource to a group, is by using an advanced procedure which is executed outside of any of the wizards that Cluster configuration normally uses.

1. To accomplish this, highlight the particular application group in the left-hand pane and in the right-hand Actions pane, select Add a resource, select the More resources…, then select Add Distributed Transaction Coordinator as seen in Figure 1.1:

Figure 1.1: Adding a DTC Resource in Failover Cluster Management snap-in Using Advanced Procedures

2. This will creates a DTC resource in the particular application group chosen and it will be in an Offline state. It is Offline as seen below, because it requires additional configuration by the user as seen in Figure 1.2:

Figure 1.2: DTC Resource Created in MSMQ Resource Application Group

3. To configure the DTC resource, right-click on the resource and select Properties from the drop down list as seen in Figure 1.3:

Figure 1.3: Selecting Properties Page for DTC Resource

4. Select the Dependencies tab and configure a dependency for a Physical Disk and Network Name resource as seen in Figure 1.4:

Figure 1.4: DTC Resource Dependency Information

5. After applying the changes and selecting OK, and bring the DTC resource Online as seen in Figure 1.5:

Figure 1.5: Bringing the DTC Resource Online in the Cluster

6. The resource should come Online and then failover can be tested as seen in Figure 1.6:

Figure 1.6: DTC Resource Online in Failover Cluster Management snap-in

7. If you use Windows Explorer to browse the disk, you will notice an MSDTC folder was created to support the DTC Log file for this instance.

Figure 1.7: MSDTC Folder on Shared Disk

Next, would be to configure the application instance in this case MSMQ to use the DTC resource in its own application group.

In Windows Server 2008, the first Clustered DTC resource becomes the default Clustered coordinator. So all services/applications unless otherwise specified will use the default Clustered coordinator by default. Let’s take the following situation where the Failover Cluster is hosting both MSMQ1 and MSMQ2, by default all MSMQ instances would use the default Clustered coordinator.

8. To avoid any confusion, we could explicitly map each DTC resource instance to each MSMQ instance with the msdtc.exe command as per the Map Clustered MS DTC Resources reference.

So let’s say that the MSMQ Service names are MSMQ-MSMQ-1 and MSMQ-MSMQ-2, then the DTC resource names are DTC-1 and DTC-2.

To map MSMQ-MSMQ-1 to DTC resource in the same service or application group, run the following command:

Msdtc –tmMappingSet –name Mapping1 –service MSMQ-MSMQ-1 –ClusterResourceName MSDTC-MSMQ-1

To map MSMQ-MSMQ-2 to DTC resource in its own DTC service or application group, run the following command:

Msdtc –tmMappingSet –name Mapping2 –service MSMQ-MSMQ-2 –ClusterResourceName MSDTC-DTC-2

NOTE: Syntax is important, if the first hyphen is missed on tmMappingSet you may not get any dialog box. Otherwise, if any hyphens from subsequent parameters are missed, resource names are misspelled or don't exist then you will get some form of dialog box for further troubleshooting.

Then you can run the following to see all the mappings:

Msdtc –tmmappingview *

Or open Registry editor and look at the following key:

HKEY_LOCAL_MACHINE\Cluster\MSDTC\\TMMapping\Service\

o Under Service you should see Mapping1 and Mapping2 which is what links the service or application resource back to a specific DTC instance.

Another thing to take notice, is in Services each MSDTC Clustered instance is identified by a unique Cluster resource {GUID} while services or applications such as SQL Server, Exchange, MSMQ or any applications are identified by the actual Cluster resource name given.

By using the Administration UI through DCOMCnfg.exe, you can manage all the Clustered DTC resources including the Local DTC. Some additional tasks that are available through the Administration UI that people may find very useful are things such as Tracing, Logging options, Security setting and Transaction Manager Communication settings.

By launching the Component Services Administration UI using DCOMCnfg.exe on each node, is where the Cluster Default Coordinator resource.

On a stand-alone server, when you go to the Properties of My Computer, then on the MSDTC tab you would notice a Default Coordinator and checkbox to Use local coordinator or you can uncheck the box and point it to a Remote coordinator host name of another server on the domain.

Once several servers form a Failover Cluster, when you go to the Properties of My Computer, then on the MSDTC tab of any node of the Cluster you would notice a Cluster Default Coordinator and the dropdown list is empty until the very first Clustered DTC resource is created. If multiple DTC resource instances are created to take advantage of Active-Active DTC, the from the dropdown list on each node you can choose which Cluster default coordinator resource name that node will use.

RELATED RESOURCE REFERENCE(S):

Understanding MS DTC Resources in Windows Server 2003 Failover Clusters

Understanding MS DTC Resources in Windows Server 2008 Failover Clusters

Map Clustered MS DTC Resources

DTC Administration Guide

DTC Developers Guide

DTC Programmers Reference

MSDTC changes in WS2008, Part 1: Introduction

MSDTC changes in WS2008, Part 2: Cluster requirements

MSDTC changes in WS2008, Part 3: Cluster features

MSDTC changes in WS2008, Part 4: Miscellaneous features

Mike Rosado
Senior Support Engineer
Microsoft Enterprise Platforms Support

Technorati Tags: DTC,MSDTC,Distributed Transaction Coordinator

CCS v1 - How to setup network adapters and bindings on Compute nodes ?

tomac — Wed, 18 Feb 2009 02:26:23 GMT

QUESTIONS:
Is it important to configure the binding order on the "Adapters and Bindings" tab like in Microsoft Failover Clusters?

This is located in Network Connections from Control Panel, by going to the Advanced drop down menu then selecting Advanced Setting. Especially when you have Infiniband (IB) network which probably should appear last on the list.

So how can this order be controlled remotely from the head node (HN) and on each compute node (HN)? Any script to do this with clusrun?

ANSWER:

The order of the network binding is insignificant with CCS / MSMPI.
You should define the "Private" and MPI" networks from the TODO list on the HN.

Moreover, CCS sets the hosts files on each CN with the cluster node's NetBIOS name to Private network IP address mapping. So the default NetBIOS name of a CN is resolved to the Private network address.

What does matter, is the following:

If CCS is setup to only use Public for HN and CN, then go through the Manage Windows Firewall Setting (Wizard) under the Networking section of the To Do List and disable ICS/NAT.

If CCS is setup to only use Public/Private for HN and CN, then go through the Manage Windows Firewall Setting (Wizard) under the Networking section of the To Do List and enable ICS/NAT.

NOTE: When enabling ICS/NAT it act as a mini DHCP server to the CN, hence is the reason why a full blown DHCP Server does not need to be installed on the HN.

The MPI net mask is passed for every executed task as an environment variable CCP_MPI_NETMASK, which mpiexec picks and uses for mpi traffic.

For additional information regarding Microsoft Compute Cluster Server, please visit our Windows HPC (High Performance Computing) Community forums

Author: Mike Rosado
Senior Support Engineer
Microsoft Enterprise Platforms Support

Technorati Tags: CCS

CCS v1 - How does MPI network actually work without name resolution?

tomac — Wed, 18 Feb 2009 02:23:10 GMT

Technorati Tags: Compute Cluster Server,CCS

As you may already know, with the release of Windows Compute Cluster Server 2003 (CCS) we included Microsoft Message Passing Interface (MS‑MPI) implementation which is fully compatible with the reference MPICH2. This allows integration with Active Directory and enables role based security for administrators and users, and the use of Microsoft Management Console (MMC) which provides a familiar administrative and scheduling interface.

The Microsoft CCS can use GbE, InfiniBand (IB), Myrinet, Quadrics, or legacy high-speed fabrics as interconnects for high performance computing. The majority of high performance computing clustered systems use GbE, but more and more customers these days prefer the high speed and low latency of interconnects such as InfiniBand or legacy specialty hardware. Our implementation of CCS supports all WSD-compatible fabrics.

This is one of those things that you wake up some days wondering “How does this thing actually work?” Which seems to be a simple question, but then after couple of discussions with the developer you realize that “Hmm, actually it is not very clear or you say some magic is happening somewhere!” If you are trying to find out answers for the following questions, then listen up…

What magic happens during MPI initialization?
What are business cards, and how do MPI apps get these for other nodes?
How MPI network works without name resolution?

What’s more interesting that when we checked the test clusters with IB cards, we found that DNS and Default Gateway settings are not configured on IB network interface cards (NICs). There was no name resolution mechanism, on the MPI network at all. So how we force the MPI traffic using mpich subnet mask without name resolution…..

After thoroughly discussing this with Mr MPI , here is a brief summary on how magic happens…

Myth: Subnet manager running on IB switch does name resolution on MPI network?
Wrong !

User submits an mpi job

Job Scheduler allocates number of nodes (or processors) requested for mpi job.

First allocated node runs the mpiexec with all required parameters that are passed by job scheduler (ccp_nodes; ccp_mpi_network …etc)

mpiexec kicks off and forms a tree by talking first to the msmpi service running on the same node, which spawns the smpd manager talking to msmpi services running on other (allocated) compute nodes that’s where we need name resolution. Because smpd manager on the first node needs to talk to other msmpi service/smpds on allocated nodes

Each mpi application starts up and queries all the LOCAL addresses for that node. Then they register this information in a “business card” in a shared database inside the smpd tree business card has all available interfaces on the node.

When MPI app rank x running on node X needs to connect to MPI app rank y running on node Y, it get y’s bizcard from the smpd tree and connects directly to x, using the address list in the business card.

MPI app x filters y addresses using MPIHC_NETMASK environment variable. This environment variable is set by mpiexec by reading CCP_MPI_NETMASK; which in turn is set as a cluster variable by ccp management services. This var is set when you select the networks in the ToDoList; it set to the MPI network if selected or to the Private network if MPI network is not present.

So bottom line is we do not need to have a name resolution on MPI network as long as node can resolve their names through private or public network.

For additional information regarding Microsoft Compute Cluster Server, please visit our Windows HPC (High Performance Computing) Community forums

RELATED RESOURCE REFERENCE(S):

· Message Passing Interface (MPI) Documentation

· Using Microsoft Message Passing Interface (MS-MPI) Documentation

Mike Rosado
Senior Support Engineer
Microsoft Enterprise Platforms Support

What is a Microsoft Failover Cluster Virtual Adapter anyway?

jeffhugh — Fri, 13 Feb 2009 15:31:46 GMT

What is a Microsoft Failover Cluster Virtual Adapter anyway?

A question often asked is, "What is the Microsoft Cluster Virtual Adapter and what can I do with it?" The typical, and correct answer, is to leave it alone and let it just work for you. While that answer satisfies most, others may want just a little more by way of an explanation, so hopefully, this blog will provide that.

The networking model in Windows Server 2008 Failover Clustering was rewritten to accommodate new functionality which included being able to obtain IP addresses from DHCP servers and being able to locate Cluster nodes on separate, routed subnets. Additionally, communications went from being UDP Broadcast transmissions to UDP Unicast with a smattering of TCP connections thrown in for good measure. What this all adds up to is more reliable and robust communication connectivity within the Cluster, no matter where the Cluster nodes were located. It no longer matters if Cluster nodes are located in the same physical rack in the same datacenter or in a server rack in a server room in a remote datacenter located at the end of an OC3 WAN connection. This now makes the Cluster more tolerant of single points of failure, e.g. Network Interface Card (NIC) card (and hence the new driver name 'Network Fault-Tolerant or NetFT.sys). The only real minimum requirement is multiple (at least two), redundant communication paths between all nodes in the Cluster. This way, the Cluster network driver (NETFT.SYS) could build a complete routing structure to provide the redundant communication connectivity the Cluster would need to keep applications and services highly available.

Note: Not having at least two networks available for cluster communications will result in a Warning (violation of a 'best practice') being recorded during the Cluster validation process. This is noted in the hardware requirements under Network Adapters and cable section.

To provide some examples of this new functionality and still not get deep into the new networking model, I generated a cluster log from a cluster node so I could illustrate how this new network model is reflected as the cluster service starts. In the cluster log, several entries are associated with NETFT. Some of these include, but may not be limited to, the following:

NETFT - Network Fault-Tolerant
TM - Topology Manager (discovers and maintains the cluster network topology. Reports failures of any networks or network interfaces. configures the Microsoft Failover Cluster Virtual Adapter)
IM - Interface Manager (Responsible for any network interfaces that are part of a cluster configuration)
NETFTAPI - NETFT Application Programming Interface (API)
FTI - Fault-Tolerant Interface

As the cluster service starts, there are events registered indicating NETFT is preparing for communications with other pieces of the cluster architecture -

00000784.000007cc::2009/01/30-14:26:38.199 INFO [NETFT] FTI NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for TM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] TM NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [CS] Starting IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] IM NetFT event handler ready for events.

As connectivity is established with other nodes in the cluster, routes are added -

00000784.00000648::2009/01/30-14:26:39.744 INFO [NETFT] Added route <struct mscs::FaultTolerantRoute>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realLocal>172.16.0.181:~3343~</realLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realRemote>172.16.0.182:~3343~</realRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualLocal>fe80::2474:73f1:4b12:8096:~3343~</virtualLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualRemote>fe80::8b6:30ea:caa3:8da7:~3343~</virtualRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Delay>1000</Delay>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Threshold>5</Threshold>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Priority>99</Priority>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Attributes>1</Attributes>
00000784.00000648::2009/01/30-14:26:39.744 INFO </struct mscs::FaultTolerantRoute>

Additional events are registered as the routes to the nodes become 'reachable' -

00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.000004f4::2009/01/30-14:26:39.759 INFO [FTI] Got remote route reachable from netft evm. Setting state to Up for route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~.
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] Marking Route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~ as up
00000784.000001f8::2009/01/30-14:26:39.759 INFO [TM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTW] NetFT is ready after 0 msecs wait.
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTI] Route is up and NetFT is ready. Connecting to node W2K8-CL2 on virtual IP fe80::8b6:30ea:caa3:8da7%15:~3343~
00000784.0000061c::2009/01/30-14:26:39.759 INFO [CONNECT] fe80::8b6:30ea:caa3:8da7%15:~3343~: Established connection to remote endpoint fe80::8b6:30ea:caa3:8da7%15:~3343~.

A consequence of the changes made to the Cluster networking model is the fact that the Cluster network driver now manifests itself as a network adapter, a hidden adapter, but an adapter nonetheless.

While this is hidden from normal view (by default) in Device Manager (must select “Show hidden devices” to see it), it is plainly visible when listing the network configuration of a Cluster node using the ipconfig /all command line.

Like other adapters, the Microsoft Failover Cluster Virtual Adapter has a MAC address and both IPv4 and IPv6 addresses assigned to it. The IPv4 address is an Automatic Private Internet Protocol Addressing (APIPA) address and the IPv6 address is a non-routable Link-Local address, but that does not matter as all cluster communications are tunneled through the networks supported by the physical NICs as shown here using the route information obtained during the cluster service startup.

The MAC address that is assigned to the Microsoft Failover Cluster Virtual Adapter is based on the MAC address of one of the physical NICs

The Cluster network driver (netft.sys) is a kernel mode driver and is started and stopped by the Cluster Service.

The Cluster network driver has an entry under HKLM\System\CurrentControlSet\Services.

Additionally, there is an entry for the Microsoft Failover Cluster Virtual Adapter in the routing table for each Cluster node. Here are sample outputs for the three sections of the route print command executed on a Cluster node. The first part shows the listing of all the interfaces on the node. Interface 15 is the Microsoft Failover Cluster Virtual Adapter.

This next screen shows the IPv4 Route Table which reflects three entries for the Microsoft Failover Cluster Virtual Adapter.

And finally, the adapter appears in the IPv6 Route Table (If 15).

So, how can one get in trouble? Here are a couple of ways:

Disable the Microsoft Failover Cluster Virtual Adapter.
Sysprep an installation of Windows Server 2008 with the Failover Cluster feature installed. This will cause an error in the Cluster Validation Process.
Modifying any properties of the adapter.

Hopefully, this gives you a better feel for this new functionality in Windows Server 2008 Failover Clusters, and like I stated at the beginning of the blog, the correct answer is to not do anything to the adapter - just let it work for you. Thanks and we hope this has been helpful.

Chuck Timon and John Marlin
Senior Support Escalation Engineers
Microsoft Enterprise Platforms Support

Sorting out some myths and facts concerning Windows Server 2008 Failover Clustering

jeffhugh — Mon, 26 Jan 2009 15:58:32 GMT

You may be wondering why, at this point in time, we are publishing a blog such as this. That is a good question, but the answer is rather straightforward - because even though Windows Server 2008 has been out for awhile now, we are starting to see an increase in the number of customers starting to use the Failover Clustering feature. Some of these customers are still trying to apply the concepts from previous versions of Microsoft Clustering. Doing this sometimes causes problems and we end up getting calls. In an effort to head-off some of those calls, here are some myths and facts about Windows Server 2008 Failover Clustering. We hope these will help.

1. If the Cluster solution purchased from the hardware vendor does not appear on the Hardware Compatibility list as documented in KB309395, then it is not considered to be a supported solution.

Myth: This is no longer true. In Windows Server 2008, the compatibility requirements have changed. All that is required now is that the Cluster solution consists of hardware that has received a Windows Server 2008 Logo and that the solution passes the built-in Cluster validation process with no failures. You are still considered as supported if there are warnings. A Warning means a 'best practice' has been violated and probably should be addressed to ensure for high availability. To learn more about the validation process, read Failover Cluster Step-by-Step Guide: Validating Hardware for a Failover Cluster. Additionally, you can read about the support policy in The Microsoft Support Policy for Windows Server 2008 Failover Clusters article.

2. I am planning on using the same hardware that I am currently using for my Windows Server 2003 cluster. It is on the HCL, so it should work.

Myth: It is incorrect to assume that hardware purchased to run a Windows Server 2003 cluster will work with a Windows Server 2008 Failover Cluster. Please check with you hardware vendor to verify if this is true or not. You can go to the Microsoft website and see which vendors are participating in the Failover Cluster Configuration Program. If your vendor is a part of this, then you next step is to look it up on the Windows Server Catalog and see what particulars need to be in place for it to work properly. There may be specific drivers, firmware, BIOS, etc updates that need to be in place before going to Windows Server 2008. If you are planning to use the same hardware, it is covered in the next topic.

3. I remember when I upgraded my Windows 2000 Server Cluster to Windows 2003, I evicted one node from the Cluster, joined the Windows 2003 node to the Cluster and then ran the process one more time. In the end, I had a brand new Windows Server 2003 Cluster. I am sure that I can do this again in Windows Server 2008.

Myth: This is no longer true. There have been so many design changes made in Windows Server 2008 Failover Clustering, that the method of upgrading Cluster nodes is no longer supported. Servers that are running the Cluster Service cannot be upgraded to Windows Server 2008. The Cluster Service must be removed before the upgrade process can be executed. What you can do is evict one of the nodes and rebuild it with Windows 2008 and install the Failover Clustering feature. Then, create a single node Cluster and migrate the resources over. Migration information is covered in this blog or the

Failover Cluster Step-by-Step Guide: Migrating Cluster Settings from Windows Server 2003 to Windows Server 2008. Please remember, you would need to run the Cluster Validation and must pass once you have all nodes up and running with Windows 2008 Failover Clustering to be supported.

4. It takes a 'rocket' science degree to be able to configure and maintain a cluster.

Myth: This is simply not true In Windows Server 2008. One of the primary design goals for Windows Server 2008 Failover Clusters was to make setting up and maintaining them easier and less time consuming. Using wizard-based processes, it is simply a matter of providing the requested information as you step through a process that ensures all configuration settings required for proper functioning of the cluster are put in place. In the end, the cluster just works. This is not to say there isn't a learning curve or that old habits will be hard to break, but let's face it...it's not your grandfather's Cluster anymore. To start learning about Windows Server 2008 High Availability Technologies, you go look at the Clustering and HA Resources blog for all sorts of topics (whitepapers, guides, videos, etc).

5. Clusters are just too sensitive. I mean, look at the quorum disk. If that fails, I am out of business.

Myth: In Windows Server 2008 Failover Clusters, the concept of 'quorum' has taken on a whole new meaning. It no longer means a disk which, if it fails, could take the Cluster down. It is more of a concept involving attaining a sufficient number of votes from the Cluster membership for highly available services to continue to be provided to end users. For more information about the new quorum model, read the Failover Cluster Step-by-Step guide; Configuring the Quorum in a Failover Cluster.

6. There are too many restrictions on Clusters. The one that always comes up is that all of the Cluster nodes must reside on the same subnet. In a multi-site scenario, I have to stretch VLANs across for all networks to make it work and have to worry about network latency, not to mention some cost factors.

Myth: This is no longer true. The changes made to the Cluster Networking model now allow Cluster nodes to be located on separate, routable subnets. This provides more flexibility when designing Clusters. It also lifts the critical restriction (500 millisecond round-trip times) for multi-site clusters. As long as there are at least two properly configured and functioning network paths on each node, the new Cluster Network Driver can figure out how to get to each node in the cluster using the best possible route.

7. Cluster networking is too hard to understand. There are just too many concepts of private, public, heartbeat, disabling NetBIOS, no NIC Teaming, Windows Network Priority versus Cluster Communication priority, and the list goes on. It just seems there are too many things to deal with.

Myth: This is no longer true. The old way of doing business involved providing multiple network connections between the nodes of the Cluster and dedicating at least one of them specifically to internal Cluster communications. The Internal Cluster Communications adapter that is on this isolated network could not be supported by 'teamed' network cards. In Windows Server 2008 Failover Clusters, all we ask is that there be at least two fully functioning, properly configured networks that the Cluster can use for

communications with other nodes. If this configuration is not in place, a Warning will be registered when the validation process is run. Again, a warning means that 'best practices' are not in place.

8. When I create a Cluster in Windows Server 2008, I am not asked to provide a domain user account. Can I change it later on to the account I want?

Fact: That is correct and no. The whole security model in Windows Server 2008 Failover Clusters has changed beginning with the removal of the requirement for a domain user account to run the Cluster Service. Now, the Cluster Service runs under the Local System account. Additionally, all Cluster Network Name (now called Client Access Points or CAP) resources register a Computer Object in Active Directory when they come online. The Computer Objects, by default, are placed in the Computers OU. They can be moved or pre-created elsewhere if desired. For more information, review the Failover Cluster Step-by-Step Guide: Configuring Accounts in Active Directory.

9. Connections to Cluster file shares cannot be made using IP Addresses, so now all my mapped network drives that use IP Addresses to connect to user shares will no longer work.

Fact: This is true. To connect to Cluster shared folders in Windows Server 2008 Failover Clusters requires the access point be either the NetBIOS or Fully Qualified Domain Name (FQDN) corresponding to the Network Name resource. This is because there is a new 'scoping' feature which is part of the changes that went into the product to support highly available File Servers. For more information on 'scoping,' you can refer to the File Share 'Scoping' in Windows Server 2008 Failover Clusters blog.

10. I can no longer share subdirectories like I used to in Windows Server 2003 Clusters. There is no File Share resource type any longer.

Fact: This is true. Quite a few of our customers have been impacted by the loss of this capability but are slowly but surely coming to grips with it. It helps to understand a little bit of the history behind this functionality. To assist with that, the Windows Clustering & High Availability Product Group wrote a short blog about it. It is important to understand how the new functionality works especially if you are going to migrate from Windows Server 2003 File Server clusters to Windows Server 2008. This blog also provides access to additional TechNet content on migration. Be sure to also review the new File Server Migration Toolkit as it can be used as well.

11. I've been working with Clusters for a long time and I sometimes seem to have issues with the shared storage and disk signature changes.

Myth: Working with shared storage in Windows Server 2008 Failover Clusters is a much better experience. The Cluster storage model, like the networking mode, has been redesigned. The Cluster Disk Driver (CLUSDISK.SYS) is no longer in a 'direct' line to the storage stack, but sits off to the side of the Windows Disk stack.

Access to storage is now the primary job of Partition Manager. If the Cluster Service needs to interact with storage, the Cluster Disk Driver communicates via the Partition Manager driver (PARTMGR.SYS). Additionally, the way Cluster reserves a disk is different in Windows Server 2008. Instead of using the SCSI-2 protocol features of Reserve\Release\Reset, Cluster Service now uses SCSI-3 Persistent Reservations to reserve a disk. And, for even more flexibility and stability, the Cluster Service uses multiple attributes to identify a piece of storage (Disk Signature and Disk Unique ID). If at least one of these two attributes match, the Cluster should be able to bring a disk online and update this change in its Cluster registry. This means that if only the disk signature changes, Cluster will bring the drive online and you may never even know it changed. No more DUMPCFG or Cluster Recovery!! If you are totally replacing a disk, you can add it to the nodes; bring up the properties of the disk resource being replaced and hit the REPAIR button. It will bring up another window for you to select which disk you are replacing it with and you can then bring it online. Done! No more DUMPCFG or Cluster Recovery!!

12. I cannot find the Cluster Log in the Cluster subdirectory (%windir%\cluster). Does this mean we no longer use it?

Myth: The 'Eventing' Model is new in Windows Server 2008. Actually, it is carried forward from Windows Vista into the server product. The Cluster Log has 'evolved' into a trace log format (.etl) and this logging is configured to start at boot time. To access the data in a readable text Cluster Log requires using the cluster.exe CLI to dump out the information. The Windows Clustering & High Availability Product Group has written a blog that discusses this as well as talks about using TRACERPT.EXE to convert the .ETL into a readable HTML format.

13. In Windows 2003 Cluster, if there was a resource that had problems, I had to sit there and let it fail multiple times on one node, watch it go to another node and fail there multiple times and then just bounce around. This takes time that I do not have to get to where I can correct the problem. Being that I am new to Windows 2008 Clustering, I am afraid I am going to spend unnecessary cycles waiting for this same thing to occur if I do something wrong.

Myth: In Windows Server 2008 Failover Clusters, the default recovery behavior has changed. In Windows Server 2008 Failover Clusters, the resource tries only one restart in a 15-minute period before the resource is marked as "Failed." Then, the Service group or the Application group to which the resource belongs is failed over to another node in the cluster. This new behavior improves the high availability model by giving resources only one restart attempt on each node in the cluster before a failover occurs. As soon as an unsuccessful attempt to restart a failed resource on each node in the cluster is made, the resource is marked as "Failed."

14. Trying to find information about Windows 2003 Clusters was a long tedious process sometimes as there are so many places to go on the Microsoft pages. Am I going to have the same ‘travels’ with Windows 2008?

Myth: The Windows Clustering & High Availability Product Group has created a Failover Clustering Portal which serves as a more centralized repository for high availability information. They have also created this blog that breaks things out a little more.

Hopefully, this short blog will help clear up some of the questions or concerns floating around out there about Windows Server 2008 Failover Clusters. As always, the CORE Team strives to provide information that we hope will be useful. Some of this information does originate from the types of issues we deal with on a daily basis in the various technology areas we support. Thanks for your attention.

Chuck Timon and John Marlin
Senior Support Escalation Engineers
Microsoft Enterprise Platforms Support

Configuring Auditing for a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 19 Jan 2009 21:57:19 GMT

The Windows Server 2008 Failover Clustering auditing feature has been requested by customers and provides the capability to monitor, or audit, cluster access. The cluster auditing feature can be enabled to audit accesses (Success and\or Failure) of an object. The object, in this case, will be the Microsoft Failover Cluster. This is accomplished by auditing client accesses to a cluster using Cluster APIs. This basically means that a client trying to access a cluster using either the Failover Cluster Management snap-in (Cluadmin.msc), the cluster.exe command line or any custom application that calls cluster APIs, will be subjected to auditing events if configured.

The OpenCluster() call initiates a new client session by binding to an RPC Server endpoint (CluAPI RPC Endpoint). This means once a user binds to a CluAPI RPC endpoint either locally or remotely, the user will be authenticated to ensure they are a valid Cluster Administrator. This is accomplished by an AccessCheckAndAuditAlarm call inside cluster anytime there is an OpenCluster() or a Security Callback.

To demonstrate auditing in a Windows Server 2008 Failover Cluster, I will configure auditing on a 2-Node cluster. To facilitate the demonstration, I create a Domain Global Group called Cluster Admins and then place three domain users in that group (Bill, Ted and Chuck). The Cluster Admins global group will be added to each cluster node’s local administrators group so the domain users in the group will have permission to administer the cluster. A fourth Domain User, Tom, is not authorized to administer the cluster but will be used to demonstrate the auditing functionality (Tom has the Log on locally right to the cluster nodes). The auditing policy will be configured at the domain level and enabled in the cluster. Finally, we will walk through a typical scenario where this feature may come in handy.

Here is the membership for the Cluster Admins global group.

After configuring the Cluster Admins group and adding it to the local administrators group on each node, create and then configure the auditing policy on a Domain Controller.

Auditing is not enabled by default either locally, on a machine, or in the domain by way of a group policy object (GPO). Here I enable auditing at the domain level by way of a Group Policy Object (GPO) (recommended best practice) so it can be applied to various clusters distributed throughout the domain. Using the Group Policy Management snap-in, create a new GPO Link called Cluster Auditing.

Note: Consider setting the Enforced option if you want to ensure this policy is not disabled by another less restrictive policy.

Once the policy is created, Edit it to actually define the settings. The policy, as shown here, is set to audit Object access and is set to audit Successes and Failures (this is an individual\business decision and can be changed).

After the policy is configured, confirm the settings in the interface.

Then, apply the policy to the cluster nodes as shown here.

This restricts the policy to the defined objects in Active Directory. In this case, it is the Authenticated Users group and the Nodes in the 2-Node cluster.

On each node in the cluster, refresh the group policy by running the command shown here.

To verify the setting, run the gpresult /v command and examine the output.

The output also details the settings for the policy (Cluster Auditing). It should be the same as configured in the Domain.

Once the policy has been applied to the cluster nodes, configure auditing in the cluster. In the Failover Cluster Management snap-in, examine the Properties of the cluster itself. Selecting the Cluster Permissions tab, access the Advanced configuration options.

Select the Auditing tab and Add the objects you want to audit. Here, I have added the Cluster Admins, Domain Admins and Authenticated Users groups.

After completing this action, auditing is enabled. No restart of the cluster service is required. Here is an example from the Security log on one of the nodes in the cluster.

If a user (in this example Tom) that does not have permission to administer the cluster, tries to access the cluster using either the Failover Cluster Management snap-in (cluadmin.msc) or using the cluster.exe CLI, they receive an error as shown here.

The corresponding Audit Failure entry in the Security log is shown here.

Now that we have discussed how to configure a domain level auditing policy so cluster can take advantage of it, here is a simple scenario that demonstrates how this capability might be used.

Scenario

Gary is an Administrator in a corporate IT Department. He notices a highly available File Server group hosted in one of the clusters in his organization is not running on the cluster node it is supposed to. He needs to find out what happened so he reviews the cluster node system logs and does not see a failure that would explain the group moving over to another node in the cluster. He knows auditing is configured so he starts his investigation.

The first place he looks is in the Failover Cluster Operational log for the node where the group should be Online. He finds the following event in the log.

This tells him when the File Server group (CONTOSO-FS1) was taken Offline. He then inspects the corresponding Operational log on the node currently hosting the group and finds -

This tells him when the group came Online on the node.

With this information in hand, Gary inspects the Security logs on the cluster nodes and discovers one of his Cluster Admins was working on the cluster during the same timeframe.

Gary follows up with Ted to find out why the group was moved - mystery solved.

There you have it - quick and dirty on how auditing can be used with Failover Clusters in windows Server 2008. I hope this information may be of some use to you. Let us know.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support