Ask the Core Team : Windows Server 2008

Resource Hosting Subsystem (RHS) In Windows Server 2008 Failover Clusters

jeffhugh — Mon, 23 Nov 2009 12:39:43 GMT

In this blog, I would like to explore some of the inner-workings of the Resource Host Subsystem (RHS) which is responsible for monitoring the health of the various cluster resources being provided as part of highly available services in a Failover cluster. A Windows Server 2008 Failover Cluster is capable of providing high availability services using a variety of resources some of which are included as part of the Failover Cluster feature and others are as part of ’cluster-aware’ applications like SQL and Exchange. Resources are designed to work together and are typically organized in Resource Groups (Figure 1). For example, a group of resources supporting a highly available File Server may consist of one or more of the following types of resources – Client Access Point (IP Address(s) + Network Name resource), Physical Disk (Storage), and a File Server. A highly available SQL Instance could contain the following resources - Client Access Point (IP Address + Network Name resource), Physical Disk (Storage), SQL Server and SQL Server Agent. Cluster resources are supported by special ‘plugins’ or resource Data Link Libraries (DLLs) that include coding to allow them to properly integrate\interoperate with the cluster service.

Figure 1

A Windows Server 2008 Failover Cluster is capable of hosting an unlimited number of resources. The management of these resources is the responsibility of the Resource Control Manager (RCM) and the Resource Host Subsystem (RHS) which provide this functionality as part of the Cluster Service itself (Figure 2).

Figure 2

The Resource Control Manager (RCM) is part of the overall cluster architecture and is responsible for implementing failover mechanisms and policies for the cluster service as well as establishing and maintaining the dependency tree (Figure 3) for each resource (e.g. a File Server resource requires a dependency on a Client Access Point and a Storage resource).

Figure 3

The Resource Control Manager maintains the state for individual resources (Online, Offline, Failed, Online Pending, and Offline Pending) as well as for Resource Groups (Online, Offline, Partial Online, and Failed). The Resource Control Manager can execute the following actions on a group of resources – Move, Failover and Failback. Which action is executed depends on several factors including the current ‘health’ of resources in the group, administrative actions taken on the group (e.g. Move Group), or the current policies in effect for the group. Here is an example (Figure 4) of Failover and Failback Group Policies –

Figure 4

Individual resources have policies (Figure 5) that apply to them as well.

Figure 5

The Resource Hosting Subsystem (RHS) is responsible for initially hosting all resources that come Online in the cluster in one default process – rhs.exe (Resource Host Monitoring process) (Figure 6).

Figure 6

Note: The rhs.exe *32 process supports 32-bit resource DLLs running in the cluster.

In previous versions of Microsoft clustering, this was called the resource monitor process (resrcmon.exe) (Figure 7).

Figure 7

There is one exception to this rule which has been implemented in the Windows Server 2008 R2 Failover Clustering feature. In Windows Server 2008 R2, the Cluster Group which consists of the Cluster Network Name resource, one or more associated IP address resources and a ‘witness’ resource and the Available Storage group are considered to be ‘critical’ cluster resource groupings and are hosted in an rhs.exe process separate from all the other cluster resources.

The Resource Hosting Subsystem (RHS) conducts periodic health checks of all cluster resources to ensure they are functioning properly. This is accomplished by executing IsAlive and LooksAlive processes which are specific to the type of resource. Examples of these are documented in the following KB article –

KB 914458 - Behavior of the LooksAlive and IsAlive functions for the resources that are included in the Windows server Clustering component of Windows Server 2003.

How often health checks are conducted is determined by the specific resource DLL or by a policy set by the cluster administrator. An example of this policy is shown in Figure 5. Should a resource fail to respond to a low-level LooksAlive check, a more in-depth IsAlive check is conducted. If a resource fails an IsAlive check, additional policies are executed until such time it is determined that a resource cannot run on a particular node in the cluster. When that point has been reached, RHS notifies the Resource Control Manager which will report the resource as Failed to the cluster service and a Failover is executed to move the Resource Group to another node in the cluster provided the default policy (Figure 8) is in effect.

Figure 8

There are times when a cluster administrator will choose not to implement the default policy shown in Figure 8 for specific ‘non-critical’ resources. This reduces instability in the cluster which could adversely impact clients connected to highly available service(s).

The IsAlive and LooksAlive health monitoring function is but a small part of what can be done with cluster resources. Figure 9 shows a listing of additional Resource DLL Entry-Point functions.

Figure 9

Note: Information on the Failover Cluster APIs can be found on MSDN.

Failure of an IsAlive call into a resource is but one way resources can become unavailable in the cluster. Other ways include:

Deadlocks in a resource DLL
Crashes in a resource DLL
RHS process itself terminates in the cluster
Cluster service fails on the node
Operating system failures (e.g. resource exhaustion)

Most of us who have been working with clusters for a long period of time understand what happens if a resource fails a critical health check. I want to spend a little time discussing resource deadlocks.

What is a resource ‘deadlock’? Basically, there are two common reasons for instability within a resource DLL. The resource DLL itself crashes (e.g. access violation in the resource DLL) or the resource fails to respond to a command in a timely fashion. Every time a call is made into a resource, a timer is started. If a response is not received within a specific period of time (configurable), the resource is considered to be deadlocked and the RHS process hosting that resource will be terminated and the resource will be placed in a newly created RHS process thereby isolating it from all the other resources running in the default rhs.exe process. When a deadlock happens, the Failover Cluster service registers an event in the cluster log. Here is an example of a deadlock occurring in the ‘Cluster Name’ resource –

000008c8.00002528::2009/06/17-20:07:57.900 WARN [RCM] ResourceControl(GET_NETWORK_NAME) to Network Name (email) returned 5910.

00000f1c.00000f28::2009/06/17-20:07:58.009 ERR [RHS] RhsCall::DeadlockMonitor: Call LOOKSALIVE timed out for resource 'Cluster Name'.

00000f1c.00000f28::2009/06/17-20:07:58.009 ERR [RHS] Resource Cluster Name handling deadlock. Cleaning current operation and terminating RHS process.

000008c8.00001cc4::2009/06/17-20:07:58.009 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'Cluster Name', gen(0) result 4.

000008c8.00001cc4::2009/06/17-20:07:58.009 WARN [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'Cluster Name' has crashed or deadlocked; marking it to run in a separate monitor.

Figure 10

Entries are also made in the Windows System Event Log. Here is an example –

06/17/2009 04:07:58 PM Error Server1.contoso.com. 1230 Microsoft-Windows-FailoverCluste Resource Control NT AUTHORITY\SYSTEM Cluster resource 'Cluster Name' (resource type '', DLL 'clusres.dll') either crashed or deadlocked. The Resource Hosting Subsystem (RHS) process will now attempt to terminate, and the resource will be marked to run in a separate monitor.

06/17/2009 04:07:58 PM Critical Server1.contoso.com. 1146 Microsoft-Windows-FailoverCluste Resource Control NT AUTHORITY\SYSTEM The cluster resource host subsystem (RHS) stopped unexpectedly. An attempt will be made to restart it. This is usually due to a problem in a resource DLL. Please determine which resource DLL is causing the issue and report the problem to the resource vendor.

Figure 11

Information on these specific Failover Cluster error messages can be found on TechNet. The information for the two events shown in Figure 11 is shown in Figure 12.

Figure 12

In Windows Server 2008 R2, RHS events are registered with Windows Error Reporting. These events can be viewed in the Action Center under Control Panel. All RHS issues will be listed under the category ‘Failover Cluster Resource Host Subsystem.’

Examining the properties of a cluster resource highlights some of the information we have been discussing. Figure 13 points out some of the pertinent properties of a resource.

Figure 13

MonitorProcessID: Indicates the Process Identifier (PID) in task manger of the rhs.exe process associated with this resource. If multiple resources have been placed in their own RHS process, it can be difficult to discern which process is associated with which resource. Examining the properties of the specific resource can help.

Note: The Process ID is not displayed by default in Task Manager. You need to add the Column to the display by selecting View in the Menu Bar and from the drop down list select Select Columns. Check the box for PID (Process Identifier).

SeparateMonitor: Indicates if the resource has been placed in a separate monitor (0:No, 1:Yes).

IsAlivePoleInterval: Default is as shown indicating it is using the default setting for this specific resource type.

LooksAlivePollInterval: Default is as shown indicating it is using the default setting for this specific resource type.

DeadlockTimeout: Default setting indicating 5 minutes.

Resource deadlock detection was actually introduced in Windows Server 2003 clusters, however it was not turned on by default. Figure 14 illustrates this.

Figure 14

Deadlock detection is turned on by default in Windows Server 2008 (RTM + R2) and cannot be disabled.

So, what is the moral of this story? It is important to understand that cluster resource deadlocks are a symptom of a larger problem. The deadlock itself is not the problem….cluster is a victim of a problem that can exist either internal to the cluster node itself or somewhere external to the cluster. Applying a logical troubleshooting methodology can help understand where the problem may exist. But, to do that requires a couple of pieces of knowledge –

Identification of the specific resource that is deadlocked.
What is the entry point that is failing?
What is the entry point trying to do?

Using the example provided in Figures 10 and 11, we can see there was a deadlock in the cluster name resource during a LooksAlive entry point. Understanding what is being evaluated for a LooksAlive process for a Network Name resource may help identify the problem which could end up being local to the node or could perhaps involve connectivity to a DNS server on the network. Referring back to KB 914458, the cluster resource DLL (ClusRes.dll) is responsible for Network Name resource health checking (IsAlive\LooksAlive tests). Some of the tests that are conducted include:

· Determining if the Network Name (NetBIOS Name) is still registered on the network stack on the node. Opening a command prompt on a node and running an nbtstat –n command to view the local NetBIOS name table, will show the registrations for cluster Network Name resources. Here is an example of a Network Name supported a Client Access Point for a File Server –

Inspecting the Parameter data for the resource in the cluster registry hive, confirms the information –

Determine the result of a DNS registration attempt (dynamic DNS is required for this test).
If the Require DNS property is set and registration fails, then the IsAlive\LooksAlive test fails.

If all DNS registrations fail and the NetBIOS name is no longer registered locally on the node, the Network Name is no longer considered reachable and the resource is placed in a Failed state. Recovery processes are initiated by the cluster service on the local node first. If local recovery fails, the Group containing the Failed Network Name resource could be moved to another node in the cluster.

What are some things that can be done to help avoid, or at least mitigate, situations where a deadlock may occur? While not set in stone, here are some of my personal recommendations:

Make sure the operating system (OS) is running with the latest service pack plus any post-service pack updates that pertain to Failover Cluster, networking or storage connectivity.
If running highly available Microsoft applications like SQL or Exchange, ensure they are updated as well.
Consult with the storage vendor and ensure the shared storage is updated and configured correctly to work in a Microsoft Failover Cluster. Most storage vendors maintain a current support matrix.
Ensure there are reliable and redundant communications paths between all nodes in the cluster.
Ensure there is reliable connectivity between all nodes in the cluster and Active Directory.
Document all Third party products that are running in the cluster and ensure they are fully updated. Third party products that interact with storage or network connectivity are always potential suspects.
Use the cluster validation process to help troubleshoot issues seen in a cluster.
If you are a Cluster Administrator, you must be aware of all changes being implemented in the corporate infrastructure to determine potential impacts on highly available services.

Hopefully, you will find this information useful. Thanks again and please come back.

Additional References:

http://blogs.msdn.com/clustering/archive/2009/06/27/9806160.aspx

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Failover Cluster Validation Firewall Error in Windows Server 2008 R2

jeffhugh — Mon, 28 Sep 2009 18:02:58 GMT

An issue involving a firewall configuration error in the cluster validation process just surfaced here in Microsoft Support so I thought I would post a quick blog in an effort to not only inform our readership, but to ‘nip this in the bud’ before we start seeing more.

After running a Windows Server 2008 R2 Failover Cluster validation report, you may see the following error –

“An error occurred while executing the test. There was an error verifying the firewall configuration. An item with the same key has already been added”

The error, as is, does not provide a clear direction to take when trying to troubleshoot. Thanks to the efforts of Cluster Product Group, the source of the issue was identified and a quick data collection process can be executed to help determine the ‘root’ cause.

The firewall configuration error is reported if any of the network adapters across the cluster nodes being validated have the same Globally Unique Identified (GUID). This can be determined by running the following WMI query on each node in the cluster and comparing the results. I chose to run the query inside PowerShell to display sample data in a formatted list-

GetWMI Win32_NetworkAdapter | fl Name,GUID

The sample output above shows the information associated with the three physical network adapters that exist in one of the nodes in my cluster. After the data is gathered from each node in the cluster, you just need to compare it and identify the duplicate GUID information.

The next logical question is, “How does one find themselves in this predicament?” In the cases we have encountered thus far, the cluster nodes were being deployed in an unsupported manner. In each case an ‘image’ was being used to deploy the nodes. We discovered that the operating system image was not properly prepared before being deployed by, for example, running sysprep.

Hopefully this information will be useful and will help avoid further occurrences of this issue. Thanks again and please come back.

Additional References:

Failover Cluster Step-by-Step Guide: Validating hardware for a Failover Cluster

KB 943984: The Microsoft Policy for Windows Server 2008 Failover Clusters

Deployment Tools Technical Reference

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Top Issues for Microsoft Support for Windows Server 2008 Hyper-V (Q4)

jeffhugh — Mon, 14 Sep 2009 14:52:42 GMT

It is time for the final installment of a year-long segment on the top issues in Hyper-V. It is appropriate since Windows Server 2008 R2 has finally released, and we can look forward to tracking\reporting any issues we may find in the new version of Hyper-V. As always, the issues are categorized below with the top issue(s) in each category listed with possible resolutions and additional comments as needed. I think you will notice that the issues for Q4 have not changed much from Q1\Q2\Q3. Hopefully, the more people read our updates, the fewer occurrences we will see for some of these and eventually they will disappear altogether (if you have been following this blog series, you will notice some already have). Additionally, we continue to highly recommend the installation of Windows Server 2008 Service Pack 2 on all servers running the Hyper-V Role.

Deployment\Planning

Issue #1

Customers looking for Hyper-V documentation.

Resolution: Information is provided on the Hyper-V TechNet Library which includes links to several Product Team blogs. Additionally, the Microsoft Virtualization site contains information that can be used to get a Hyper-V based solution up and running quickly.

Installation Issues

Issue #1

After the Hyper-V role is installed, a customer creates a virtual machine, but it fails to start with the following error:

The virtual machine could not be started because the hypervisor is not running

Cause: Hardware virtualization or DEP was disabled in the BIOS.

Resolution: Enable Hardware virtualization or DEP in the BIOS. In some cases, the server needs to be physically shutdown in order for the new BIOS settings to take effect.

Issue #2

A customer was experiencing an issue on a pre-release version of Hyper-V.

Resolution: Upgrade to the release version (KB950050) of Hyper-V.

Issue #3

After the latest updates off Windows Update are installed or KB950050 is installed, virtual machines fail to start with one of the following error messages:

An error occurred while attempting to chance the state of the virtual machine vmname .
‘ vmname ’ failed to initialize.
Failed to read or update VM configuration.

An error occurred while attempting to change the state of virtual machine vmname .
" VMName " failed to initialize
An attempt to read or update the virtual machine configuration failed.
" VMName " failed to read or update the virtual machine configuration: Unspecified error (0x80040005).

Cause: This issue occurs because virtual machine configurations that were created in the beta version of the Hyper-V are incompatible with later versions of the Hyper-V.

Resolution: Perform the steps documented in KB949222.

Virtual Devices or Drivers

Issue #1

Synthetic NIC was listed as an unknown device in device manager.

Cause: Integration Components needed to be installed.

Resolution: Install Integration Components (IC) package in the VM.

Issue #2

Corrupted virtual hard disk (VHD) file.

Cause: The most common cause was a power outage or the server wasn’t shutdown properly.

Resolution: Restore the VHD file from backup.

Issue #3

Stop 0x00000050 on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: This issue can occur if a Hyper-V virtual machine is configured with a SCSI controller but no disks are attached (driver issue - Storvsp.sys).

Resolution: Perform the steps documented in KB969266.

Issue #4

Stop 0x0000001A on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: Vid.sys

Resolution: Install hotfix KB957967 to address this issue.

Snapshots

Issue #1

Snapshots were deleted

Cause: The most common cause is that a customer deleted the .avhd files to reclaim disk space (not realizing that the .avhd files were the snapshots).

Resolution: Restore data from backup.

For more information on Snapshots, please refer to the Snapshot FAQ: http://technet.microsoft.com/en-us/library/dd560637.aspx.

Issue #2

Snapshots were lost

Cause: Parent VHD was expanded (not supported). If snapshots are associated with a virtual hard disk, the parent vhd file should never be expanded. This is documented in the Edit Disk wizard:

Resolution: Restore data from backup.

Issue #3

Snapshots fail to merge with error 0x80070070

Cause: Low disk space.

Resolution: Free disk space to allow the merge to complete or move the .VHD and .AVHD file(s) to a volume with sufficient disk space and manually merge the snapshots.

Integration Components

Issue #1

On Windows Server 2008, when you attempt to install the Integration Components in a Hyper-V virtual machine running Windows Vista Service Pack 2, the installation may fail with the following error:

An error has occurred: One of the update processes returned error code 1.

Cause: This issue occurs if the management operating system (parent partition) that has the Hyper-V role installed does not have Service Pack 2 installed. If you have a virtual machine that’s running Windows Vista Service Pack 2, you need to use the Vmguest.iso from Service Pack 2 to install the Integration Components.

Resolution: Perform the steps documented in KB974503.

Issue #2

Attempting to install the Integration Components on a Server 2003 virtual machine fails with the following error:

Unsupported Guest OS

An error has occurred: The specified program requires a newer version of Windows.

Cause: Service Pack 2 for Server 2003 wasn’t installed in the virtual machine.

Resolution: Install SP2 in the Server 2003 VM before installing the integration components.

Virtual machine State and Settings

Issue #1

You may experience one of the following issues on a Windows Server 2008 system with the Hyper-V role installed or Microsoft Hyper-V Server 2008:

When you attempt to create or start a virtual machine, you receive one of the following errors:

The requested operation cannot be performed on a file with a user-mapped section open. ( 0x800704C8 )
‘VMName’ Microsoft Synthetic Ethernet Port (Instance ID {7E0DA81A-A7B4-4DFD-869F-37002C36D816}): Failed to Power On with Error 'The specified network resource or device is no longer available.' (0x80070037).
The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3)

Virtual machines disappear from the Hyper-V Management Console.

Cause: This issue can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files.

Resolution: Perform the steps documented in KB961804.

Issue #2

Customer has multiple Hyper-V servers and virtual machines are getting duplicate MAC addresses.

Resolution: Configure the Hyper-V servers to use unique MAC address ranges by modifying the MinimumMacAddress and MaximumMacAddress registry values on each Hyper-V server. This issue is documented on TechNet: http://technet.microsoft.com/en-us/library/dd582198(WS.10).aspx. On Server 2008 R2, the MAC address ranges can be configured in the UI.

Issue #3

Virtual machines have a state of "Paused-Critical"

Cause: Lack of free disk space on the volume hosting the .vhd or .avhd files.

Resolution: Free up disk space on the volume hosting the .vhd or .avhd files.

High Availability (Failover Clustering)

Issue #1

Virtual machine settings that are changed on one node in a Failover Cluster are not present when the VM is moved to another node in the cluster.

Cause: The "Refresh virtual machine configuration" option was not used before attempting a failover.

Resolution: We have a KB article (KB 2000016) which discusses this issue for Windows 2008. On Windows 2008 R2, the experience has improved. If the virtual machine settings are modified within the Failover Cluster Management console, changes that are made to the VM will be saved to the Cluster (i.e. synchronized across all nodes in the cluster). If you make changes to the VM using the Hyper-V Manager Console, you must select the refresh virtual machine configuration option before the VM is moved to another node. This issue is documented in the Windows Server 2008 R2 help file. There is also a blog that discusses this.

Issue #2

How to configure Hyper-V on a Failover Cluster.

Resolution: A step-by-step guide is available which covers how to configure Hyper-V on a Failover Cluster.

Backup (Hyper-V VSS Writer)

Issue #1

You may experience one of the following symptoms if you try to backup a Hyper-V virtual machine:

· If you back up a Hyper-V virtual machine that has multiple volumes, the backup may fail. If you check the VMMS event log after the backup failure occurs, the following event is logged:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin

Source: Microsoft-Windows-Hyper-V-VMMS

Event ID: 10104

Level: Error

Description:

Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

· The Microsoft Hyper-V VSS Writer may enter an unstable state if a backup of the Hyper-V virtual machine fails. If you run the vssadmin list writers command, the Microsoft Hyper-V VSS Writer is not listed. To return the Microsoft Hyper-V VSS Writer to a stable state, the Hyper-V Virtual Machine Management service must be restarted.

Resolution: An update (KB959962) is available to address issues with backing up and restoring Hyper-V virtual machines.

Issue #2

How to backup virtual machines using Windows Server Backup

Resolution: Perform the steps documented in KB958662.

Virtual Network Manager

Issue #1

Virtual machines are unable to access the external network.

Cause: The virtual network was configured to use the wrong physical NIC.

Resolution: Configure the external network to use the correct NIC.

Issue #2

After the customer configured a virtual machine to use a VLAN ID, the virtual machine is unable to access the network.

Cause: The VLAN ID used by the virtual machine didn’t match the VLAD ID configured on the network switch.

Resolution: How to configure a virtual machine to use a VLAN is covered in the Hyper-V Planning and Deployment guide.

Issue #3

How to configure a virtual machine to use a VLAN.

Resolution: How to configure a virtual machine to use a VLAN is covered in the Hyper-V Planning and Deployment guide.

Hyper-V Management Console

Issue #1

How to manage Hyper-V remotely.

Resolution: The steps to configure remote administration of Hyper-V are covered in a TechNet article. John Howard also has a very thorough blog on remote administration.

Miscellaneous

Issue #1

You may experience one of the following issues on a Windows Server 2003 virtual machine:

· An Event ID 1054 is logged to the Application Event log:

Event ID: 1054
Source: Userenv
Type: Error
Description:
Windows cannot obtain the domain controller name for your computer network. (The specified domain either does not exist or could not be contacted). Group Policy processing aborted.

· A negative ping time is displayed when you use the ping command.

· Perfmon shows high disk queue lengths

Cause: This problem occurs when the time-stamp counters (TSC) for different processor cores are not synchronized.

Resolution: Perform the steps documented in KB938448.

As always, we hope this has been informative for you.

BTW – Did I mention we are strongly recommending installing Windows Server 2008 SP2 on all Hyper-V server? Have a good one!

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

DPM 2007 - Troubleshooting protection for Hyper-V

tomac — Fri, 31 Jul 2009 00:22:03 GMT

This post is about Windows Server 2008 with the Hyper-V role installed, that are being protected by System Center Data Protection Manager 2007. There may be one or many Virtual Machines on each Host/Parent Partition, and they may be running Windows 2003 and/or Windows 2008. Supposing the DPM Agent is installed only on the Host/Parent partition of the Hyper-V server, you may find that DPM jobs fail intermittently on the 2003 VM’s, but the 2008 VM’s successfully complete. The following error may be encountered:

Type: Recovery point
Status: Failed
Description: DPM encountered a retryable VSS error. (ID 30112 Details:
Unknown error (0x800423f3) (0x800423F3))
End time: 4/23/2009 3:37:22 PM
Start time: 4/23/2009 3:36:38 PM
Time elapsed: 00:00:44
Data transferred: 0 MB
Cluster node -
Recovery Point Type Express Full
Source details: \Backup Using Child Partition Snapshot\%ServerName%
Protection group: %ProtectionGroupName%

We found these jobs fail when the Volume Shadow Copy Service (VSS service) on the guest VM is in a “Stopping” state and the only way to get the service in a good condition is to kill the process or reboot the VM. If the VSS service is in this “Stopping” state the next DPM job will fail. But if you first verify the VSS service is in a correct state (running or stopped) the DPM job will run successfully. However, once the DPM job is done you may see the VSS service stuck in the “Stopping” state. This service should automatically stop after 3 minutes of idle time but intermittently it may not stop. We experienced this behavior across several Hosts and almost all VM’s in a particular environment. The behavior is random but a few VM’s experience the problem more frequently than others. We also noticed if the VM is rebooted it will likely work without issues for a few days before the problem re-occurs.

When using vssadmin Windows Server command (see http://technet.microsoft.com/en-us/library/cc754968(WS.10).aspx), it appeared the “Microsoft Hyper-V VSS Writer” on the host was in a “Failed” state with a “Retryable” Last error state when the job fails. Ordinarily the writer will show a “Stable” state, and “No error” as follows.

When the jobs fail, the above command will return:

Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {59f449f9-2413-494d-b679-965bc56129fd}
State: [8] Failed
Last error: Retryable error

After installing Service Pack 2 for Windows 2008, or hotfix KB967560 (see Resolution section below) and running another DPM job on a different VM that has the VSS service in a non-stopping state, the job will run succesfully and place the Hyper-V VSS Writer back into a “Stable” state.

Quick reference for different possible scenarios:

VM in good state + Host in good state = Good backup
VM in bad state + Host in good state = Failed backup
VM in bad state + Host in bad state = Failed job
VM in good state + Host in bad state = Good job

Another possible symptom, when a DPM job is running you may notice on the Hyper-V Management screen next to the VM it displays this message: “Creating VSS Snapshot Set…”. This will continue to be displayed and when looking at the Volume Shadow Copy Service inside the W2K3 VM you may notice the service is stopped. Additionally, when running “vssadmin list writers” on the Host a message is displayed but no writers are visible:

“Waiting for responses. These may be delayed if a shadow copy is being prepared.”

When this condition occurs you may not be able cancel the DPM job. When trying to cancel the DPM job you may get the following in the detailed pane:

Type: Recovery point
Status: Attempting to cancel
End time: -
Start time: 4/28/2009 10:00:21 AM
Time elapsed: 01:01:22
Data transferred: -
Cluster node -
Recovery Point Type Express Full
Source details: \Backup Using Child Partition Snapshot\%Servername%
Protection group: %ProtectionGroupName%

Looking in the System Event log on the W2K3 VM during the times when the DPM jobs failed it may be clean. But the Application event log may be filled with the VSS errors below:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 8193
Date: 2/24/2009
Time: 8:46:34 AM
User: N/A
Computer: %SystemName%
Description:
Volume Shadow Copy Service error: Unexpected error calling routine IEventSystem::Store. hr = 0x80040206.

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12302
Date: 2/24/2009
Time: 5:58:49 AM
User: N/A
Computer: %SystemName%
Description:
Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers. Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.

When viewing the System and Application event logs on the DPM server neither have any entries for the same times as the job failures. But the DPM event log may have the following entry:

Event Type: Error
Event Source: DPM-EM
Event Category:None
Event ID: 2
Date: 2/19/2009
Time: 9:20:20 AM
User: N/A
Computer: %DPMServername%
Description:
Creation of recovery points for Backup Using Child Partition Snapshot\%ProtectedServerName-VM% on %HOSTName% have failed. The last recovery point creation failed for the following reason: (ID: 3159) DPM encountered a retryable VSS error. (ID: 30112)
DPM ID: 2^|^%DPMServername%^|^Recovery point creation failures^|^DPM^|^Backup^|^%HOSTName% ^|^a48c6c91-f4ae-4ed3-b5da-a3c22d980a48

RESOLUTION

The Hyper-v issue seems to be the result of the underlying state of VSS. VSS is hung in the "stopping" state because the registry writer is hung attempting to unregister a COM+ event subscription. This is a subscription for listening for COM messages from other VSS components. When analyzing the logs captured during the problem it was found the unsubscribe function had been waiting eight minutes when the trace ended (and still had not completed).

It could be that the machine is having COM issues. The VSS service is not going to be successful with processing subsequent jobs until this unsubscribe completes. If you experience any of the symptoms mentioned above, you should perform all of the action items noted below.

Action Item #1:

Verify all the Prerequisites are met for protecting Hyper-V with DPM:

Prerequisites and Known Issues with Hyper-V Protection
http://technet.microsoft.com/en-us/library/dd347840.aspx

Action Item #2:

Online backups are not possible if any of the following conditions are not met. Verify that all the W2K3 VM’s meet these requirements:

1. Hyper-V Integration components is installed and is running the latest version

NOTE: (On the Host/Parent partition you can check VMMS.exe = 6.0.6001.22352 (or newer) and in the guest, check vmbus.sys version 6.0.6001.22334 (or newer)

2. No Dynamic disks inside the guest.

3. All volumes are NTFS

4. All NTFS volumes must be >1GB and have >300MB free space.

5. Shadow copies within the VM are on the same volume or are Disabled

6. VM is in running state.

NOTE: Offline Backups of Windows 2000 Guest VMs fail. Cause: A synthetic SCSI Controller was configured for the VM with no drives attached. Windows 2000 Guests do not support the SCSI Controller, so it is not needed.

Action Item #3:

The root cause of symptoms noted in the Problem section appear to be COM related. After verifying the action items above install the following COM updates:

KB934016 "Availability of Windows Server 2003 Post-Service Pack 2 COM+ 1.5 Hotfix Rollup Package 12"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;934016

KB965230 "FIX: The COM+ Event System does not deliver timely or reliable statistics to subscribers of the IComTrackingInfoEvents event interface in Windows Server 2003"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;965230

KB968447 "The COM+ Event System stops processing the query for matching subscriptions when it detects a corrupted subscription on a Windows Server 2003-based computer"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;968447

Action Item #4:

Install the following two W2K3 VSS updates on the W2K3 virtual machines:

KB940349 “Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;940349

KB969219 “RPC 0x800706ba and 0x800706bf errors occur when backup software tries to create VSS shadow copies on a computer that is running Windows Server 2003 SP2”
http://support.microsoft.com/default.aspx?scid=kb;EN-US;969219

Install the latest VSS/Volsnap update on the W2K3 VM’s. If the Host is also running W2K3 it will be a good idea to also install:
KB967551 “Rollup update for the volsnap.sys driver in Windows Server 2003”
http://support.microsoft.com/default.aspx?scid=kb;EN-US;967551

Action Item #5:

If possible, install W2K8 SP2 since it will include the most recent Hyper-V writer updates. But, there are situations where installing SP2 will not be an option. As an alternative you can install KB967560 and KB971394 on the Windows Server 2008 Host machine.

KB967560 update is more recent then KB959978 which does address a known issue when you run a Windows Server 2003-based virtual machine on a Windows Server 2008 Hyper-V-based computer:

KB967560 “A backup operation fails on a two-node failover cluster that is running Windows Server 2008 after one of the disk resources is moved”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;967560

KB971394 "A backup of virtual machines fails when you use the Hyper-V VSS writer to back up virtual machines concurrently on a computer that is running Windows Server 2008"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;971394

How to obtain the latest service pack for Windows Server 2008
http://support.microsoft.com/kb/968849

ADDITIONAL INFORMATION:

Virtualization with Hyper-V: Supported Guest Operating Systems
http://www.microsoft.com/windowsserver2008/en/us/hyperv-supported-guest-os.aspx

Author:
Tom O’Malley
Microsoft Enterprise Support
Sr. Support Escalation Engineer

Technorati Tags: Data Protection Manager 2007,Hyper-V,Windows Server 2008

Expected Snapshot Merge Behavior for a Highly Available VM

jeffhugh — Fri, 29 May 2009 15:22:03 GMT

Hello my name is Sean Dwyer, and I'm a Support Escalation Engineer working in the Windows CORE team here at Microsoft.

While working with snapshots that are attached to a highly available VM in a Cluster, you may notice after deleting a snapshot, it does not merge as expected. You'll notice the merge process almost immediately ends and the VM begins to restart.

Let's explore a scenario I ran myself into the other day and then I'll explain why you may see this behavior.

Fig. 1.

Here I have configured a HA-VM of XP SP2 in Fig 1. and it is Online and happy, within the Cluster.

Fig. 2.

Switching to the Hyper-V Management MMC in Fig. 2, I've decided that my testing is over with this VM, and I need to clean up the snapshots. The 'Now' state is where I want the VM to remain, so I'm going to issue a Delete Snapshot Subtree command.

Fig. 3.

I choose 'Yes', and the snapshots are removed from the VM in Fig. 3.
Next, I choose to shut down the VM in Fig. 4, as I want to commit the merge now, and not wait until later.

Fig. 4.

The Merge process starts, but immediately finishes and the VM is restarting as seen in Fig. 5 and Fig. 6.

Fig. 5.

Fig. 6.

The merge didn't complete! What just happened?

Failover Cluster is doing its job!

When deleting a snapshot for a VM that is configured as a Highly Available Resource, you're taking action against the VM outside of the Cluster Management UI, and therefore, the Cluster service considers any shutdown events, or reboots, a failure of the resource and takes appropriate action.

In my example, the default behavior is restarting the VM as shown in Fig.7.

Fig. 7.

The end result is that the snapshot merge process has NOT taken place and I've still got AVHDs out there shown in Fig. 8.

Fig. 8.

So, what do we do, to get the Snapshots merged successfully? Easy!

We need to configure the Clustered VM Resource to allow the Snapshot Merge process to complete without being interrupted.

Open up the Properties of the VM, and then select the Offline Actions Tab.

You'll note, in Fig. 9, the default behavior is to Save the VM.

Fig. 9.

In Fig. 10, let's change this default behavior temporarily to Shutdown.

Fig. 10.

Once the change has been made, you can then choose to shutdown the VM either through Hyper-V Manager or through the Clustering UI, and the Snapshot Merge process will begin.

Once the Merge completes, reset the behavior of the VM in the Offline Actions Tab for the VM back to Save as shown in Fig. 11.

Fig. 11.

Once you start the VM, you'll be back in business!

I hope this blog post will help you continue to use our Clustering and Virtualization products successfully!

Sean Dwyer
Support Escalation Engineer
Microsoft Enterprise Platforms Support

Top Issues for Microsoft Support for Windows Server 2008 Hyper-V (Q3)

jeffhugh — Wed, 27 May 2009 17:24:55 GMT

It is time to update everyone on the issues our support engineers have been seeing for Hyper-V for the past quarter. The issues are categorized below with the top issue(s) in each category listed with possible resolutions and additional comments as needed. I think you will notice that the issues for Q3 have not changed much from Q1\Q2. Hopefully, the more people read our updates, the fewer occurrences we will see for some of these and eventually they will disappear altogether. There will probably be one more blog for the Q4 results. Additionally, I would like to mention that we are highly recommending the installation of Windows Server 2008 Service Pack 2 on all servers running the Hyper-V Role.

Deployment\Planning

Issue #1

Customers looking for Hyper-V documentation.

Installation Issues

Issue #1

A customer was experiencing an issue on a pre-release version of Hyper-V.

Resolution: Upgrade to the release version (KB950050) of Hyper-V.

Issue #2

After the latest updates off Windows Update are installed or KB950050 is installed, virtual machines fail to start with one of the following error messages:

An error occurred while attempting to chance the state of the virtual machine vmname .
‘ vmname ’ failed to initialize.
Failed to read or update VM configuration.

Cause: This issue occurs because virtual machine configurations that were created in the beta version of the Hyper-V are incompatible with later versions of the Hyper-V.

Resolution: Perform the steps documented in KB949222.

Issue #3

After the Hyper-V role is installed, a customer creates a virtual machine but it fails to start with the following error:

The virtual machine could not be started because the hypervisor is not running

Cause: Hardware virtualization or DEP was disabled in the BIOS.

Resolution: Enable Hardware virtualization or DEP in the BIOS. In some cases, the server may need to be physically shutdown in order for the new BIOS settings to take effect.

Virtual Devices\Drivers

Issue #1

Synthetic NIC was listed as an unknown device in device manager.

Cause: Integration Components needed to be installed.

Resolution: Install Integration Components (IC) package in the VM.

Issue #2

Stop 0x00000050 on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: This issue can occur if a Hyper-V virtual machine is configured with a SCSI controller but no disks are attached.

Resolution: Perform the steps documented in KB969266.

Issue #3

Stop 0x0000001A on a Microsoft Hyper-V Server 2008 or Server 2008 system with the Hyper-V role installed.

Cause: Vid.sys

Resolution: Install hotfix KB957967 to address this issue.

Snapshots

Issue #1

Snapshots fail to merge with error 0x80070070

Cause: Low disk space.

Resolution: Free up disk space to allow the merge to complete.

Issue #2

Snapshots were deleted

Cause: The most common cause is that a customer deleted the .avhd files to reclaim disk space (not realizing that the .avhd files were the snapshots).

Resolution: Restore data from backup.

For more information on Snapshots, please refer to the Snapshot FAQ: http://technet.microsoft.com/en-us/library/dd560637.aspx.

Issue #3

Snapshots were lost

Cause: Parent VHD was expanded (not supported). If snapshots are associated with a virtual hard disk, the parent vhd file should never be expanded. This is documented in the Edit Disk wizard:

Resolution: Restore data from backup.

Integration Components

Issue #1

A Windows 2000 (SP4) virtual machine with the Integration Components installed may shut down slowly.

Cause: This problem is caused by a bug in the Windows Software Trace Pre-Processor (WPP) tracing macro (outside of Hyper-V).

Resolution: KB959781 documents the workarounds for this issue on Server 2008.

Issue #2

Attempting to install the Integration Components on a Server 2003 virtual machine fails with the following error:

Unsupported Guest OS

An error has occurred: The specified program requires a newer version of Windows.

Cause: Service Pack 2 for Server 2003 wasn’t installed in the virtual machine.

Resolution: Install SP2 in the Server 2003 VM before installing the integration components.

Virtual machine State and Settings

Issue #1

You may experience one of the following issues on a Windows Server 2008 system with the Hyper-V role installed or Microsoft Hyper-V Server 2008:

When you attempt to create or start a virtual machine, you receive one of the following errors:

· The requested operation cannot be performed on a file with a user-mapped section open. ( 0x800704C8 )

· ‘VMName’ Microsoft Synthetic Ethernet Port (Instance ID

{7E0DA81A-A7B4-4DFD-869F-37002C36D816}): Failed to Power On with Error 'The specified network resource or device is no longer available.' (0x80070037).

· The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3)

Virtual machines disappear from the Hyper-V Management Console.

Cause: This issue can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files.

Resolution: Perform the steps documented in KB961804.

Issue #2

Creating or starting a virtual machine fails with the following error:

'General access denied error' (0x80070005).

Cause: This issue can be caused by the Intel IPMI driver.

Resolution: Perform the steps documented in KB969556.

Issue #3

Virtual machines have a state of "Paused-Critical"

Cause: Lack of free disk space on the volume hosting the .vhd or .avhd files.

Resolution: Free up disk space on the volume hosting the .vhd or .avhd files.

High Availability (Failover Clustering)

Issue #1

How to configure Hyper-V on a Failover Cluster.

Resolution: A step-by-step guide is now available which covers how to configure Hyper-V on a Failover Cluster.

Issue #2

Virtual machine settings that are changed on one node in a Failover Cluster are not present when the VM is moved to another node in the cluster.

Cause: The "Refresh virtual machine configuration" option was not used before attempting a failover.

Resolution: When virtual machine settings are changed on a VM that’s on a Failover Cluster, you must select the ‘Refresh virtual machine configuration’ option before the VM is moved to another node. There is a blog that discusses this.

Backup (Hyper-V VSS Writer)

Issue #1

You may experience one of the following symptoms if you try to backup a Hyper-V virtual machine:

· If you back up a Hyper-V virtual machine that has multiple volumes, the backup may fail. If you check the VMMS event log after the backup failure occurs, the following event is logged:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin

Source: Microsoft-Windows-Hyper-V-VMMS

Event ID: 10104

Level: Error

Description:

Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

Resolution: An update (KB959962) is now available to address issues with backing up and restoring Hyper-V virtual machines.

Issue #2

How to backup virtual machines using Windows Server Backup

Resolution: Perform the steps documented in KB958662.

Virtual Network Manager

Issue #1

Virtual machines are unable to access the external network.

Cause: The virtual network was configured to use the wrong physical NIC.

Resolution: Configure the external network to use the correct NIC.

Issue #2

Network connectivity issues

Cause: NIC teaming software

Resolution: Remove the NIC teaming software. Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Issue #3

Customers inquiring if Hyper-V supports NIC Teaming.

Resolution: Our support policy for NIC Teaming with Hyper-V is now documented in KB968703.

Hyper-V Management Console

Issue #1

How to manage Hyper-V remotely.

Resolution: The steps to configure remote administration of Hyper-V are covered in a TechNet article. John Howard also has a very thorough blog on remote administration.

Import/Export

Issue #1

Importing a virtual machine may fail with the following error:

A Server error occurred while attempting to import the virtual machine. Failed to import the virtual machine from import directory <Directory Path>. Error: One or more arguments are invalid (0x80070057).

Resolution: Perform the steps documented in KB968968.

Miscellaneous

Issue #1

You may experience one of the following issues on a Windows Server 2003 virtual machine:

· An Event ID 1054 is logged to the Application Event log:

· A negative ping time is displayed when you use the ping command.

· Perfmon shows high disk queue lengths

Cause: This problem occurs when the time-stamp counters (TSC) for different processor cores are not synchronized.

Resolution: Perform the steps documented in KB938448.

As always, we hope this has been informative for you.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Active Route Gets Removed on Windows 2008 Failover Cluster IP Address Offline

jeffhugh — Tue, 26 May 2009 21:31:50 GMT

We have received calls adding static routes on Windows Server 2008 Failover Clustering nodes and wanted to pass along some important information regarding this. The issue is that when you add a static persistent route to a network adapter that is on a Windows Server 2008 Failover Cluster and take a Clustered IP Address offline (or move it to another node), the “Active” route is removed and no connections can be made using this route even though it still shows as persistent. Once you bring the Clustered IP Address back online, the active route is returned.

First, I want to mention that the networking architecture in 2008 Failover Clustering has been rewritten from the ground up and now we have a our own internal route listings as well as our own adapter (Microsoft Failover Cluster Virtual Adapter). I will not into the specifics of it here, but you can read more about it in the blog “What is a Microsoft Failover Cluster Adapter anyway?”.

On to the problem and resolution. As a little setup, here is the configuration that I want to discuss.

ClusterNode1

Physical IP Address: 10.44.60.4

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

ClusterNode2

Physical IP Address: 10.44.60.3

Physical Subnet Mask: 255.255.0.0

Default Gateway: 10.44.60.1

Failover Cluster Virtual IP Address

IP Address: 10.44.60.6

I also have a backup server that I use to create backups using an IP Address of 10.51.0.1 and subnet mask 255.255.0.0 that will use the same default gateway above. Most Network Administrators would use the following ROUTE.EXE command to add a persistent static route to the local tables so that a connection can be made.

route -p add 10.51.0.0 mask 255.255.0.0 10.44.60.1

So with everything online (including the Failover Cluster Virtual IP Address) on ClusterNode1, I can do a ROUTE PRINT command to display my IP Address version 4 table and see this. As a side note, I am just pulling the necessary information from the Route Table.

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276    <<---
10.44.60.4           255.255.255.255 On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.60.6           255.255.255.255 On-link       10.44.60.4    276    <<--- Clustered IP Address
10.44.255.255        255.255.255.255 On-link       10.44.60.4    276   <<---
10.51.0.0            255.255.0.0      10.44.60.1    10.44.60.4     21    <<--- Static Route added
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255 On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                 <<--- Persistent Route added
===========================================================================

As long as the Clustered IP Address of 10.44.60.6 is online on this node, all is well. However, if I were to take the 10.44.60.6 IP Address offline, things change.

C:\>cluster res "IP Address 10.44.60.6" /offline
Taking resource ''IP Address 10.44.60.6'' offline...
Resource                 Group         Node             Status
--------------------     ----------    --------------- ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Offline

C:\>route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask          Gateway       Interface     Metric
10.44.0.0            255.255.0.0      On-link       10.44.60.4    276    <<---
10.44.60.4           255.255.255.255 On-link       10.44.60.4    276    <<--- Physical Node IP Address
10.44.255.255        255.255.255.255 On-link       10.44.60.4    276   <<---
224.0.0.0            240.0.0.0        On-link       10.44.60.4    276    <<---
255.255.255.255      255.255.255.255 On-link       10.44.60.4    276    <<---
===========================================================================
Persistent Routes:
Network Address      Netmask          Gateway Address   Metric
10.51.0.0            255.255.0.0      10.44.60.1        1                 <<--- Persistent Route added
===========================================================================

Notice here that the Clustered IP Address 10.44.60.6 as well as the 10.51.0.1 “Active” route is removed. Because the 10.51.0.0 route is removed, connectivity to the backup server is lost. If you bring the Clustered IP Address 10.44.60.6 online again, the “Active” routes are re-populated again and connectivity to the backup server is restored.

C:\>cluster res "IP Address 10.44.60.6" /online
Bringing resource ''IP Address 10.44.60.6'' online...
Resource                 Group         Node             Status
--------------------     ----------    --------------- ------
IP Address 10.44.60.6    Data Group    ClusterNode1     Online

According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.

The ROUTE.EXE command has additional parameters of METRIC and INTERFACE that you would need to specify that will bind the route to the card itself.

C:\>route /?

Manipulates network routing tables.

ROUTE [-f] [-p] [command [destination]

[MASK netmask] [gateway] [METRIC metric] [IF interface]

interface the interface number for the specified route.

METRIC specifies the metric, ie. cost for the destination.

So what you need to do first is determine what the interface is so that we can bind the route to it. When doing the ROUTE PRINT or NETSH command, it will give you the interfaces at the top first. Something similar to this:

C:\>route print

IPv4 Route Table

===========================================================================

Interface List

23 ...00 15 5d 4a ac 06 ...... Local Gigabit Controller

19 ...00 15 5d 4a ac 01 ...... Local Gigabit Controller #2

18 ...00 15 5d 4a ac 00 ...... Local Gigabit Controller #3

===========================================================================

-or-

C:\>netsh int ipv4 show int

Idx Met MTU State Name

--- --- ----- ----------- -------------------

18 50 4294967295 connected Local Gigabit Controller #3

19 5 1500 connected Local Gigabit Controller #2

23 5 1500 connected Local Gigabit Controller

I can go into the Network and Sharing Center if I have to to see which card is on this network. In my particular case, the “Local Gigabit Controller #3” is the one I want to use. So to get my persistent route to stay even though the Clustered IP Address goes offline, my command would be below. Please note that the METRIC is not needed as a requirement of the command.

route -p add 10.51.0.0 mask 255.255.0.0 0.0.0.0 metric 276 if 18

Now, the “Active” route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline. A good rule of thumb, moving forward, would be that if you are adding a static persistent route, start specifying the 0.0.0.0 and the interface as this is the proper supported commands from a networking perspective. This will result in the proper functioning no matter if Failover Clustering is configured or not.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Recovering a Deleted Cluster Name Object (CNO) in a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 27 Apr 2009 14:32:49 GMT

Greetings once again from the support trenches here on the CORE team. I want to talk a bit about a Windows Server 2008 Failover Cluster issue that appears to be on the rise. What we are seeing is the Computer Object for the Cluster Name (a.k.a. Cluster Name Object (CNO) being removed from Active Directory resulting in the Cluster Name no longer being able to function properly. This does not happen automatically. It requires some sort of human interaction either by consciously going into AD and deleting the object or running some script (process) that deletes it. However this is being done, it appears to us that the implications are not fully understood and there is no quick recovery from this. In this blog, I hope to provide information that will help avoid this scenario from happening within your organization. Along the way, I want to provide some 'value-add' information by discussing how the cluster computer objects relate to each other.

The first step to preventing this from happening in your organization is to be sure there is a clear understanding of the cluster security model in Windows Server 2008. Rather than spend a whole lot of time and space here rehashing what is already publicly available, I refer you to the following:

KB 947049: Description of the Failover Cluster Security Model in Windows Server 2008.

Failover Cluster Step-by-Step Guide: Configuring Accounts in Active directory

After reviewing the materials, you should have an understanding of how security works in Windows Server 2008 Failover Clusters and an appreciation for the importance of not removing (or disabling) the Computer Objects created in Active Directory by the cluster. By default, the Computer Objects created by the cluster are all placed in the Computers container. These can be relocated to another OU, or even pre-staged in an OU before the cluster is created. If pre-staging, be sure to review the requirements in the Step-by-step Guide already mentioned. As an example (Figure 1), I created a Cluster OU and moved the cluster nodes and their associated objects into the OU.

Figure 1

You may want to consider implementing a similar practice in your organization as it groups the cluster objects together thereby reinforcing the idea that this grouping of objects is 'special' in some way.

Before moving forward and discussing the actual recovery process, I want to spend a little time reviewing the cluster 'family tree' to help you gain an understanding of how cluster objects are related. To illustrate, I will use a cluster named W2K8-CLUS (Figure 2) in the CONTOSO domain.

Figure 2

This cluster is located in the Cluster OU shown in Figure 1. Using Regedit.exe, I open the cluster registry hive and inspect the properties for the cluster. I can see the name of the cluster and the resource GUID for the Cluster Name.

Figure 3

Expanding the Resource GUID corresponding to the Cluster Name, I inspect additional properties for the resource. Selecting the Parameters entry displays the ObjectGUID for the cluster Computer Object in Active directory (Figure 4).

Figure 4

In Figure 5, we see the attribute in Active directory (must enable Advanced Features before the Attribute Editor tab is visible). You can also use ADSIEdit to view the same information.

Figure 5

The Cluster Name Object (CNO) functions as the primary security context for the cluster. The CNO is responsible for creating any additional Computer Objects (Virtual Computer Objects (VCO)) associated with the cluster. These Computer Objects represent Network Name resources in a cluster. A Network Name resource is created as part of a Client Access Point (CAP). Each Computer Object created by a cluster CNO contains an Access Control Entry (ACE) for the CNO on the Access Control List (ACL) for the object. The CNO is also responsible for synchronizing the password for each VCO in the domain. The VCOs associated with a particular CNO can be determined either by manually inspecting the ACL for each VCO in AD, or the information can be obtained in the cluster registry.

Opening the cluster registry hive and inspecting the properties of the Cluster Name resource, we can see an entry called ObjectGUIDS. This is a listing for each Computer Object created by the CNO in Active directory. In Figure 6, I have four Computer Objects in Active Directory associated with this cluster.

Figure 6

One of them is a Computer Object (VCO) associated with the CAP representing a highly available Print Server (CONTOSO-PS1) in this cluster (Figure 7).

Figure 7

Well, there you have it…the cluster family tree.

So, what happens if the Cluster Name Object is deleted from Active Directory? A few important things –

· The Cluster Name, if Online, will stay Online but will fail to come Online again if the resource is cycled (it will be placed in a Failed state). This will prevent being able to connect to the cluster remotely when trying to administer the cluster.

· The security context for the cluster is lost. This prevents the passwords for all associated VCOs from being synchronized within the domain. Also, any user, service or other process needing permission to access cluster objects will fail to be authenticated.

· No more CAPs can be created in the cluster.

Besides the items listed above, there are other indications of problems. The Cluster Name resource in the Cluster Core Resources group will be in a Failed state. Attempts to bring the resource Online will generate a pop-up error (Figure 8)

Figure 8

A FailoverClustering error (Event ID 1207) will be registered in the System Log (Figure 9).

Figure 9

The cluster log will report a failure to locate the CNO Computer Object in Active Directory (Figure 10)

Figure 10

It is, therefore, very important the CNOs Computer Object in the domain not be deleted.

How does one recover from this? The supported way(s) to recover an Active Directory object that has been accidentally, or intentionally, deleted are described in the following articles and will not be covered in detail here–

KB840001: How to restore deleted user accounts and their group memberships in Active Directory

TechNet Content - Recovering Active Directory Domain Services

Additionally, there are 3^rd party solutions that can be used to protect Active Directory objects and\or recover them if deleted. Finally, as a last ditch effort, and when there is no other alternative, there is a free utility called ADRestore (32-bit only) that can be used to recover the Computer Object associated with the CNO. Please review the following information before deciding to use this utility –

Microsoft Supportability Newsletter – Using ADRestore tool to restore deleted objects

Either of these methods can be used, but they may end up being time consuming, expensive or both.

Once the Computer Object has been recovered from Active Directory, the Repair Active Directory object action can be used to restore functionality in the cluster (Figure 11).

Figure 11

Note: The logged on user that will perform the Repair action must have rights to administer the cluster and must have the right to Reset Passwords in the domain.

I personally believe ‘an ounce of prevention is worth a pound of cure.’ To that end, my top recommendation is to implement the steps outlined in the section Preventing unwanted deletions in the TechNet Content already mentioned above. Beginning with Windows Server 2008, objects in Active Directory, such as the Computer Object shown here (Figure 12), can be protected from accidental deletion by simply checking a box – Protect object from accidental deletion.

Figure 12

With this ‘guard’ in place, when an object is selected for deletion, the first pop-up is presented (Figure 13)

Figure 13

If Yes is selected, the next error is presented to the user (Figure 14) thus preventing deletion.

Figure 14

If this isn’t enough, there is more help coming in Windows Server 2008 R2. Domain Services in Windows Server 2008 R2 will include an optional feature called Active Directory Recycle Bin. This feature is not enabled by default and must be added. Details about the feature can be found on TechNet

TechNet Content – Active Directory Recycle Bin Step-by-Step Guide

That about wraps it up for this installment. As usual, we hope this information is useful. Come back and visit.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Windows 2008 Failover Cluster Validation Fails on ‘Validate SCSI-3 Persistent Reservation’

jeffhugh — Wed, 15 Apr 2009 15:02:42 GMT

We’ve been seeing a lot of calls lately from customers who are running the validation that’s required prior to installing and configuring failover clustering, and the validation fails in the ‘Storage’ portion of the tests. The specific error seen in the validation report is:

If you click on the ‘Validate SCSI-3 Persistent Reservation’ link in the report. It will take you to the detail section.

Validate SCSI-3 Persistent Reservation

Validate that storage supports the SCSI-3 Persistent Reservation commands.

Validating Cluster Disk 0 for Persistent Reservation support

Registering PR key for cluster disk 0 from node node1.cluster.com

Failed to Register PR key for cluster disk 0 from node node1.cluster.com status 1

Cluster Disk 0 does not support Persistent Reservation

If you dig a little deeper, you can also look at the ValidateStorage.txt file that’s located in the Windows\Cluster\Reports directory.

00000fd4.00000fd8::15:56:45.857 CprepDiskPRUnRegister: Enter CprepDiskPRUnRegister: ulSignature 0xd0426bb2
00000fd4.00000fd8::15:56:45.857 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:49.977 CprepDiskPRUnRegister: Failed to unregister PR key, status 1117
00000fd4.00000fd8::15:56:49.977 CprepDiskPRUnRegister: Exit CprepDiskPRUnRegister: hr 0x8007045d
00000fd4.00000fd8::15:56:54.097 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:54.097 CprepDiskIsPRPresent: Failed to read PR reservations, status 0
00000fd4.00000fd8::15:56:54.097 CprepDiskIsPRPresent: Exit CprepDiskIsPRPresent hr 0x0, Present 0
00000fd4.00000fd8::15:56:54.097 CprepDiskFind: found disk with signature 0xd0426bb2
00000fd4.00000fd8::15:56:54.097 DoIoctlAndAlloc: ControlCode 0x70050, retCode 1, status 122
00000fd4.00000fd8::15:56:54.097 CprepDiskGetArbSectors: Exit CprepDiskGetArbSectors: hr 0x0, SectorX 11 SectorY 12

So what is a “Persistent Reservation” (PR) and why should you care? A PR is a SCSI command, which clustering uses to protect LUN’s. When a LUN is reserved, no other computers on the SAN can access the disk, except the ones cluster controls. This is important to protect other machines from accessing the disk and corrupting the data on the disk.

Validate is a functional test tool that verifies that your storage supports all the necessary SCSI commands that clustering requires. It is critical that Validate tests pass, for your cluster to work correctly. The Storage tests are by far the most important, they should not be dismissed!

If you are reading this blog, then the bad news is that Validate has probably identified that your storage does not support Persistent Reservations, and is not compatible with Windows Server 2008 Failover Clustering. The good news is that it most likely will work, you just have to do a few things! All storage vendors and almost all current shipping models support Win2008 Failover Clustering, but many require firmware updates or configuration settings. Microsoft has been working closely with partners such as HP, EMC, IBM, NetApp, HDS, Fujitsu, Lefthand, Equallogic, Xiotech, NEC, LSI, Infortrend, 3PAR, Intransa, FalconStor, Nexsan, and even more… and they all work!

First things first, call your storage vendor and ask them if your storage is compatible AND configured for use with Windows Server 2008 Failover Clustering.

There are two things to verify:

Correct firmware version
Correct configuration settings

The storage vendor is really the right person to tell you how to correctly configure their arrays to work with Failover Clustering, so they are the right source. We can’t post the specific steps for each vendor but as we become aware of publicly available documentation from the SAN vendors, we’ll add them to this post as they start being published.

HP has a publicly available document detailing the steps needed to get SCSI-3 PR’s to work specific to their hardware (pages16-18)

Implementing Microsoft® Windows® Server 2008 Service Pack 2 beta on HP ProLiant servers

With any of these vendor links, although they may contain steps to resolve the PR problem, we still strongly recommend being directly engaged with the vendor to verify with them that these storage configuration changes are current, appropriate for your environment and hardware, and non destructive to your data. Microsoft makes no guarantees on any of the 3rd party links we are providing. They are solely intended to have information on hand to discuss with your particular vendor.

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Maximizing Limited Storage Resources in a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 09 Mar 2009 16:10:53 GMT

We have encountered scenarios where customers are implementing Windows Server 2008 Failover Clusters and they want to make quite a few services and applications highly available but, they are not able to purchase additional Storage to facilitate this. Or, it may be that another business within their organization has higher priority for existing storage assets and their 'lower' priority cluster will just have to make do with what storage has already been allocated. What is a cluster administrator to do under such circumstances?

In this blog, as an example, I will show you how to configure a highly available File Server Group so that it can also be used to support highly available Print Services. I am using File and Print Services because that is the most common scenario we see in customer environments. To implement this configuration we will be using the concepts explained in KB947050: Advanced resource configuration in Windows Server failover clusters.

Note: Even though these procedures are fully supported, the preference is to use dedicated storage for each highly available service or application. This allows the built-in wizard-based processes to be used thus ensuring the correct configuration. It is recommended that customers review their current storage utilization within each cluster and discuss with their local Storage Team to see if it would be possible recover any excess storage space on current LUNs which could then be used to create new LUNs.

The starting point will be an already configured File Server application (CONTOSO-FS1) providing some shared folders for a couple of business groups within an organization.

We will be using the storage in this file Server group to also host the files needed for a highly available Print Server. The procedures we will use will be executed outside of the normal wizard-based process for configuring a highly available Print Server. When using the normal wizard-based process, a highly available Print Server application looks something like this:

The resources in the group include a Client Access Point (CAP) (CONTOSO-PS1), a Print Spooler resource and a piece of storage for storing print jobs and printer drivers. Additionally, since this was configured using the wizard-based process, we also have the correct group type configured [101] so we will have access to the Print Management interface via the Failover Cluster Management snap-in:

The process we will use will not provide direct access to the Print Management snap-in from inside Failover Cluster Manager and, we will have to take that into account. Let's get started....

Since we are outside of the normal wizard-based process, the cluster will not be able to verify the installation of any prerequisite Roles and\or Features, so the first thing is to ensure we install the Print Server Role on all nodes in the cluster.

Once the Print Server role is installed, the Print Management snap-in is listed under Administrative Tools.

The Print Management snap-in provides management capabilities for print services running either in the context of the local node or a highly available Print Server using a configured Client Access Point (CAP).

With the Print Server role installed, we can move forward with the manual configuration of the resources we will need to place in the highly available File Server group.

The Print Spooler resource requires a dependency on a Network Name and Physical disk resource. Both of these resources already exist and are Online in the group so we could use the existing resources. However, we will choose to create a new Client Access Point (CAP) (CONTOSO-PS2) so the users can connect using another Network Name. In the Actions pane to the right, select Add a resource action and select a Client Access Point.

This starts a wizard where we create a NetBIOS name and IP Address (IP address information would not be requested if using DHCP).

Once the resource is created, bring it Online.

Next, create a Print Spooler resource.

The Print Spooler resource is created and placed in an Offline state. The resource is Offline because additional configuration is required before it can be brought Online. If the resource were brought Online at this point, it would fail and would take the whole group down because a failover would be initiated. This would disrupt any user connections to shared folders (more on this later).

To complete the configuration, Right-click on the resource and select Properties. On the General tab you can change the display name for the resource (optional) but you must enter a path to a folder on the storage where spooled print jobs and printer drivers can be stored. I created a Spooler directory on the storage so I enter the path information.

Next, select the Dependencies tab and add dependencies for the storage and the CAP.

Verify the default setting for the Policies and Advanced Policies tabs and then click OK. Bring the Print Spooler resource Online.

Test failover to other nodes in the cluster to be sure you have high availability.

Earlier, I mentioned we would not be able to manage highly available printers in the Failover Cluster Management snap-in when we configure a Print Spooler resource using the method above. If we were to open the Print Management snap-in, accessible in Administrative Tools, we would only see the local cluster node listed.

We have to manually add the new CAP we created for the Print Spooler resource.

Once we complete this action, both the local and the highly available Print Server will be visible in the management interface.

Looking at the final result where we have a single grouping of resources which consist of both File Server and Print Server resources, we need to consider what happens in case of any failure?

This is a critical question that needs to be answered because if the default behavior is implemented which is "if restart is unsuccessful, fail over all resources in this service or application", then all the resources in the application group will be taken Offline and moved to another node in the cluster and brought back Online.

In this scenario then, if the Print Spooler resource were to fail and could not be restarted on the node that owns it, the entire group, which includes the File Server resource and its associated shared folders, would be taken Offline and moved to another node in the cluster. This will interrupt all client connections to the shared folders until they are brought back Online on another node and the connections re-established. A decision may be required, in this situation, where perhaps it is more important that access to the File Server be maintained while the Print Spooler is in a Failed state. To accomplish this, a cluster administrator would have to modify the setting by Unchecking the box on both the Print Spooler resource and the associated Client Access Point so that failures of either of these will not result in a failover. You obviously would not do this for the disk resource because that is the single common resource being used by both services and would want that to failover.

To ensure the modified settings result in the desired behavior, we can simulate failures on the modified resources and observe the results. Here, I am simulating a failure on the Client Access Point for the Print Server.

After a single successful restart of the resource, execute another simulated failure and the resource will go into a Failed state but will not force a failover of the group.

So there you have it. A method for maximizing existing resources in a cluster. Before we wrap up, I want to emphasize again that the preferred method would be to have dedicated storage for each highly available application or service and not try to multi-purpose current storage.

I hope this information proves to be useful to someone and keep the cards and letters coming.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

What is a Microsoft Failover Cluster Virtual Adapter anyway?

jeffhugh — Fri, 13 Feb 2009 15:31:46 GMT

What is a Microsoft Failover Cluster Virtual Adapter anyway?

A question often asked is, "What is the Microsoft Cluster Virtual Adapter and what can I do with it?" The typical, and correct answer, is to leave it alone and let it just work for you. While that answer satisfies most, others may want just a little more by way of an explanation, so hopefully, this blog will provide that.

The networking model in Windows Server 2008 Failover Clustering was rewritten to accommodate new functionality which included being able to obtain IP addresses from DHCP servers and being able to locate Cluster nodes on separate, routed subnets. Additionally, communications went from being UDP Broadcast transmissions to UDP Unicast with a smattering of TCP connections thrown in for good measure. What this all adds up to is more reliable and robust communication connectivity within the Cluster, no matter where the Cluster nodes were located. It no longer matters if Cluster nodes are located in the same physical rack in the same datacenter or in a server rack in a server room in a remote datacenter located at the end of an OC3 WAN connection. This now makes the Cluster more tolerant of single points of failure, e.g. Network Interface Card (NIC) card (and hence the new driver name 'Network Fault-Tolerant or NetFT.sys). The only real minimum requirement is multiple (at least two), redundant communication paths between all nodes in the Cluster. This way, the Cluster network driver (NETFT.SYS) could build a complete routing structure to provide the redundant communication connectivity the Cluster would need to keep applications and services highly available.

Note: Not having at least two networks available for cluster communications will result in a Warning (violation of a 'best practice') being recorded during the Cluster validation process. This is noted in the hardware requirements under Network Adapters and cable section.

To provide some examples of this new functionality and still not get deep into the new networking model, I generated a cluster log from a cluster node so I could illustrate how this new network model is reflected as the cluster service starts. In the cluster log, several entries are associated with NETFT. Some of these include, but may not be limited to, the following:

NETFT - Network Fault-Tolerant
TM - Topology Manager (discovers and maintains the cluster network topology. Reports failures of any networks or network interfaces. configures the Microsoft Failover Cluster Virtual Adapter)
IM - Interface Manager (Responsible for any network interfaces that are part of a cluster configuration)
NETFTAPI - NETFT Application Programming Interface (API)
FTI - Fault-Tolerant Interface

As the cluster service starts, there are events registered indicating NETFT is preparing for communications with other pieces of the cluster architecture -

00000784.000007cc::2009/01/30-14:26:38.199 INFO [NETFT] FTI NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for TM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] TM NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [CS] Starting IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] IM NetFT event handler ready for events.

As connectivity is established with other nodes in the cluster, routes are added -

00000784.00000648::2009/01/30-14:26:39.744 INFO [NETFT] Added route <struct mscs::FaultTolerantRoute>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realLocal>172.16.0.181:~3343~</realLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realRemote>172.16.0.182:~3343~</realRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualLocal>fe80::2474:73f1:4b12:8096:~3343~</virtualLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualRemote>fe80::8b6:30ea:caa3:8da7:~3343~</virtualRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Delay>1000</Delay>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Threshold>5</Threshold>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Priority>99</Priority>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Attributes>1</Attributes>
00000784.00000648::2009/01/30-14:26:39.744 INFO </struct mscs::FaultTolerantRoute>

Additional events are registered as the routes to the nodes become 'reachable' -

00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.000004f4::2009/01/30-14:26:39.759 INFO [FTI] Got remote route reachable from netft evm. Setting state to Up for route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~.
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] Marking Route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~ as up
00000784.000001f8::2009/01/30-14:26:39.759 INFO [TM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTW] NetFT is ready after 0 msecs wait.
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTI] Route is up and NetFT is ready. Connecting to node W2K8-CL2 on virtual IP fe80::8b6:30ea:caa3:8da7%15:~3343~
00000784.0000061c::2009/01/30-14:26:39.759 INFO [CONNECT] fe80::8b6:30ea:caa3:8da7%15:~3343~: Established connection to remote endpoint fe80::8b6:30ea:caa3:8da7%15:~3343~.

A consequence of the changes made to the Cluster networking model is the fact that the Cluster network driver now manifests itself as a network adapter, a hidden adapter, but an adapter nonetheless.

While this is hidden from normal view (by default) in Device Manager (must select “Show hidden devices” to see it), it is plainly visible when listing the network configuration of a Cluster node using the ipconfig /all command line.

Like other adapters, the Microsoft Failover Cluster Virtual Adapter has a MAC address and both IPv4 and IPv6 addresses assigned to it. The IPv4 address is an Automatic Private Internet Protocol Addressing (APIPA) address and the IPv6 address is a non-routable Link-Local address, but that does not matter as all cluster communications are tunneled through the networks supported by the physical NICs as shown here using the route information obtained during the cluster service startup.

The MAC address that is assigned to the Microsoft Failover Cluster Virtual Adapter is based on the MAC address of one of the physical NICs

The Cluster network driver (netft.sys) is a kernel mode driver and is started and stopped by the Cluster Service.

The Cluster network driver has an entry under HKLM\System\CurrentControlSet\Services.

Additionally, there is an entry for the Microsoft Failover Cluster Virtual Adapter in the routing table for each Cluster node. Here are sample outputs for the three sections of the route print command executed on a Cluster node. The first part shows the listing of all the interfaces on the node. Interface 15 is the Microsoft Failover Cluster Virtual Adapter.

This next screen shows the IPv4 Route Table which reflects three entries for the Microsoft Failover Cluster Virtual Adapter.

And finally, the adapter appears in the IPv6 Route Table (If 15).

So, how can one get in trouble? Here are a couple of ways:

Disable the Microsoft Failover Cluster Virtual Adapter.
Sysprep an installation of Windows Server 2008 with the Failover Cluster feature installed. This will cause an error in the Cluster Validation Process.
Modifying any properties of the adapter.

Hopefully, this gives you a better feel for this new functionality in Windows Server 2008 Failover Clusters, and like I stated at the beginning of the blog, the correct answer is to not do anything to the adapter - just let it work for you. Thanks and we hope this has been helpful.

Chuck Timon and John Marlin
Senior Support Escalation Engineers
Microsoft Enterprise Platforms Support

Windows Server Backup 2008 Restore from Network Location

jeffhugh — Wed, 04 Feb 2009 21:02:29 GMT

Hello my name Scott McArthur and I am a Senior Support Escalation Engineer in the Windows Platforms group. Today’s blog submission is how to use Windows Server Backup to do a complete system restore over the network. This blog is specific to Windows Server 2008 but the process is basically the same for Windows Vista.

Assumptions

You have added the Windows Server Backup feature to your 2008 install and run a Complete PC Backup to a network location, \\server\share

The following steps show how to do this restore by using the Windows Recovery Environment UI in addition to a sample way to do this with command line.

If the network adapter or hard drive requires a third-party driver, you should put those drivers on a USB flash drive.

1. Start the computer by using the Windows Server 2008 DVD

2. On the first screen Click Next.

3. Select the “Repair your computer” option in the lower-left corner of screen.

4. It will show you any currently installed operating systems. Click Next.

If this screen is blank you may have to load a third-party driver for your mass storage driver. You can click Load Drivers to load the mass storage driver from your USB flash drive.

5. Click “Windows Complete PC Restore”.

6. It will report “A valid backup location could not be found”. Click cancel.

7. Select “Restore a different backup” then next.

8. Click Advanced.

9. If the network adapter driver is included with Windows Server 2008 you can click “Search for a backup on the network. If the network adapter driver is not included you have to click “install a driver” and browse to your driver to load it. In my test I was using a Hyper-V virtual machine with the legacy network adapter. The legacy network adapter driver is in Windows Server 2008 so that it just works. The synthetic driver is not included.

You would receive an error similar to the following when you try to connect to the server.

10. Click Yes to the “Are you sure you want to connect to the network” and then specify the path of your backup. You can use IP address instead to eliminate any netbios/dns issues.

11. Select the backup listed and then click Next.

12. Select the backup then Next.

13. You are presented with the restore options.

The exclude disks option enables you to exclude disks from the restore process.

The advanced button has the following options.

14. Click Finish to confirm the settings.

If you selected “Format and repartition disks” you are prompted with “Windows Complete PC restore will format the disks you chose to restore, which will erase all existing data”. Click “I confirm that I want to format the disks and restore the backup”.

You can monitor the progress through the final dialog box.

The restart will automatically occur or you can delay it.

For some scenarios, you may want to do this manually. The following lists the basic general steps to do this at the command prompt.

1. At the following screen select the “Command prompt” option.

2. Run the following commands at the command prompt.

If your network adapter is not included with Windows you need to load the driver using drvload.

Drvload driver.inf

Where driver.inf is the inf for the network adapter. You can load this from an usb flash drive or the local drive if it is available.

Start /w wpeinit

Run ipconfig to verify the driver loaded and you have a valid ip address. If you have to set a static IP address, you could use netsh. For example

“netsh interface ipv4 set address name = "<idx>" source=static address=<staticIP> mask=<SubnetMask> gateway=<DefaultGateway>”

Netsh Commands for Interface (IPv4 and IPv6)

http://technet.microsoft.com/en-us/library/cc770948.aspx

Net use z: \\server\share /user:username

Where \\server\share is the location that you saved your backup to. There should be a WindowsImageBackup folder in the root of the share/directory that you saved your backup to.

wbadmin get versions -backuptarget:\\server\share

You should output similar to the following

The times of the backups displayed are based on the timezone of the current
operating system you have booted into.
The timezone used currently is (GMT -08:00) Pacific Standard Time

Backup time: 1/8/2009 11:20 AM
Backup target: Network Share labeled \\server\share
Version identifier: 01/08/2009-19:20
Can Recover: Volume(s), File(s), Application(s), Bare Metal Recovery

From this you must have the Version Identifier for the next step.

Note this basically wipes the drive so be careful when you use the next command

wbadmin start sysrecovery –backuptarget:\\server\share -version:01/08/2009-19:20 -recreatedisks –restoreallvolumes -quiet

Hopefully this helps show how to do a basic restore of a Windows Server 2008 computer.

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

How To: Activate Windows Server 2008 Core Edition with MAK key over the phone

jeffhugh — Mon, 02 Feb 2009 17:39:50 GMT

Hello my name is Scott McArthur and I am a Senior Support Escalation Engineer in the Windows group. Today we are going to discuss the steps to activate Windows Server 2008 Core edition using a MAK key via the phone. It is much easier to activate via the Internet but for those computers that do not have access to the Internet (for example a Hyper-V virtual machine) you can use the phone. If you have more than one machine to activate you may want to look at the Volume Activation Management Tool 1.1. This supports batch activation for computers not connected to the Internet. This is very useful for isolated lab scenarios.

The steps in this blog are specific to core edition and a MAK Key but the same general steps could be applied to the non-core edition and other types of keys. In order to activate with a MAK key you must install using volume license media. By default volume license media will look for a KMS host to activate and do not require a key to install. If you are prompted during setup for a key then you are not installing from volume license media.

After Windows Server 2008 Core Edition is installed do the following at a cmd prompt

1. Install your MAK key using slmgr.vbs

slmgr.vbs -ipk XXXXXXXXXXXXXXXXXXXXXXXXX

Where XXXXXXXXXXXXXXXXXXXXXXXXX is your 5x5 MAK key supplied by Microsoft

You should receive the following confirmation:

2. View your Installation ID

slmgr.vbs -dti

Note: the –DTI option only shows up when you have installed a MAK or Retail key.

You will get the following dialogue box that has your Installation ID

Tip: The Installation ID is a 9x6 number. An easy way to split it up so it is easier to read back to the phone activation system is to hit CTRL-C to copy the contents of the dialogue box and paste into notepad. In notepad it will be broken out into the 9x6 groups and easier to read. For example

---------------------------
Windows Script Host
---------------------------
Installation ID: XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

3. Determine the phone number for your region:

notepad %systemroot%\system32\slui\phone.inf

For example here is the line for USA

USA,,,%United_States%,1-866-740-1256,%not_available%,%not_available%

4. Call the number for your region and you will be prompted for the Installation ID, Groups 1 through 9

Tip: You can press 2 on the keypad to proceed to the next group

5. Once you have entered your installation ID your confirmation ID will be read back to you. The Confirmation ID is read back to you from Group A to Group H. This is a 8x6 number

Groups

A	B	C	D	E	F	G	H
xxxxxx	xxxxxx	xxxxxx	xxxxxx	xxxxxx	xxxxxx	xxxxxx	xxxxxx

Type it into your notepad with no dashes so you can paste it into the next step

6. Activate with your Confirmation ID

slmgr.vbs –atp xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Note: The –atp option is only available when you have installed a MAK or retail key

You should receive the following confirmation

7. Verify that you are activated using slmgr.vbs

Cscript.exe %systemroot%\system32\Slmgr.vbs –dlv

Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.
Software licensing service version: 6.0.6001.18000
Name: Windows Server(R), ServerEnterpriseCore edition
Description: Windows Operating System - Windows Server(R), VOLUME_MAK_B channel
Activation ID: <removed>
Application ID: <removed>
Extended PID: <removed>
Installation ID: <removed>
Processor Certificate URL: http://go.microsoft.com/fwlink/?LinkID=48189
Machine Certificate URL: http://go.microsoft.com/fwlink/?LinkID=48190
Use License URL: http://go.microsoft.com/fwlink/?LinkID=48192
Product Key Certificate URL: http://go.microsoft.com/fwlink/?LinkID=48191
Partial Product Key: <removed>
License Status: Licensed

Check the “Description” to verify it says Volume_MAK channel and “License Status” says Licensed

If you run into any errors with this process please see the following articles

http://support.microsoft.com/default.aspx?scid=kb;EN-US;938450

http://support.microsoft.com/default.aspx?scid=kb;EN-US;942969

Here are the other commands available with slmgr.vbs

Windows Software Licensing Management Tool
Usage: slmgr.vbs [MachineName [User Password]] [<Option>]

MachineName: Name of remote machine (default is local machine)
User: Account with required privilege on remote machine
Password: password for the previous account

Global Options: -ipk <Product Key>
   Install product key (replaces existing key)
-ato      Activate Windows
-dli [Activation ID | All]
    Display license information (default: current license)
-dlv [Activation ID | All]
    Display detailed license information (default: current license)
-xpr
    Expiration date for current license state
Advanced Options:
-cpky
    Clear product key from the registry (prevents disclosure attacks)
-ilc <License file>
    Install license
-rilc
    Re-install system license files
-rearm
    Reset the licensing status of the machine
-upk
    Uninstall product key
-dti
    Display Installation ID for offline activation
-atp <Confirmation ID>
    Activate product with user-provided Confirmation ID

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Sorting out some myths and facts concerning Windows Server 2008 Failover Clustering

jeffhugh — Mon, 26 Jan 2009 15:58:32 GMT

You may be wondering why, at this point in time, we are publishing a blog such as this. That is a good question, but the answer is rather straightforward - because even though Windows Server 2008 has been out for awhile now, we are starting to see an increase in the number of customers starting to use the Failover Clustering feature. Some of these customers are still trying to apply the concepts from previous versions of Microsoft Clustering. Doing this sometimes causes problems and we end up getting calls. In an effort to head-off some of those calls, here are some myths and facts about Windows Server 2008 Failover Clustering. We hope these will help.

1. If the Cluster solution purchased from the hardware vendor does not appear on the Hardware Compatibility list as documented in KB309395, then it is not considered to be a supported solution.

Myth: This is no longer true. In Windows Server 2008, the compatibility requirements have changed. All that is required now is that the Cluster solution consists of hardware that has received a Windows Server 2008 Logo and that the solution passes the built-in Cluster validation process with no failures. You are still considered as supported if there are warnings. A Warning means a 'best practice' has been violated and probably should be addressed to ensure for high availability. To learn more about the validation process, read Failover Cluster Step-by-Step Guide: Validating Hardware for a Failover Cluster. Additionally, you can read about the support policy in The Microsoft Support Policy for Windows Server 2008 Failover Clusters article.

2. I am planning on using the same hardware that I am currently using for my Windows Server 2003 cluster. It is on the HCL, so it should work.

Myth: It is incorrect to assume that hardware purchased to run a Windows Server 2003 cluster will work with a Windows Server 2008 Failover Cluster. Please check with you hardware vendor to verify if this is true or not. You can go to the Microsoft website and see which vendors are participating in the Failover Cluster Configuration Program. If your vendor is a part of this, then you next step is to look it up on the Windows Server Catalog and see what particulars need to be in place for it to work properly. There may be specific drivers, firmware, BIOS, etc updates that need to be in place before going to Windows Server 2008. If you are planning to use the same hardware, it is covered in the next topic.

3. I remember when I upgraded my Windows 2000 Server Cluster to Windows 2003, I evicted one node from the Cluster, joined the Windows 2003 node to the Cluster and then ran the process one more time. In the end, I had a brand new Windows Server 2003 Cluster. I am sure that I can do this again in Windows Server 2008.

Myth: This is no longer true. There have been so many design changes made in Windows Server 2008 Failover Clustering, that the method of upgrading Cluster nodes is no longer supported. Servers that are running the Cluster Service cannot be upgraded to Windows Server 2008. The Cluster Service must be removed before the upgrade process can be executed. What you can do is evict one of the nodes and rebuild it with Windows 2008 and install the Failover Clustering feature. Then, create a single node Cluster and migrate the resources over. Migration information is covered in this blog or the

Failover Cluster Step-by-Step Guide: Migrating Cluster Settings from Windows Server 2003 to Windows Server 2008. Please remember, you would need to run the Cluster Validation and must pass once you have all nodes up and running with Windows 2008 Failover Clustering to be supported.

4. It takes a 'rocket' science degree to be able to configure and maintain a cluster.

Myth: This is simply not true In Windows Server 2008. One of the primary design goals for Windows Server 2008 Failover Clusters was to make setting up and maintaining them easier and less time consuming. Using wizard-based processes, it is simply a matter of providing the requested information as you step through a process that ensures all configuration settings required for proper functioning of the cluster are put in place. In the end, the cluster just works. This is not to say there isn't a learning curve or that old habits will be hard to break, but let's face it...it's not your grandfather's Cluster anymore. To start learning about Windows Server 2008 High Availability Technologies, you go look at the Clustering and HA Resources blog for all sorts of topics (whitepapers, guides, videos, etc).

5. Clusters are just too sensitive. I mean, look at the quorum disk. If that fails, I am out of business.

Myth: In Windows Server 2008 Failover Clusters, the concept of 'quorum' has taken on a whole new meaning. It no longer means a disk which, if it fails, could take the Cluster down. It is more of a concept involving attaining a sufficient number of votes from the Cluster membership for highly available services to continue to be provided to end users. For more information about the new quorum model, read the Failover Cluster Step-by-Step guide; Configuring the Quorum in a Failover Cluster.

6. There are too many restrictions on Clusters. The one that always comes up is that all of the Cluster nodes must reside on the same subnet. In a multi-site scenario, I have to stretch VLANs across for all networks to make it work and have to worry about network latency, not to mention some cost factors.

Myth: This is no longer true. The changes made to the Cluster Networking model now allow Cluster nodes to be located on separate, routable subnets. This provides more flexibility when designing Clusters. It also lifts the critical restriction (500 millisecond round-trip times) for multi-site clusters. As long as there are at least two properly configured and functioning network paths on each node, the new Cluster Network Driver can figure out how to get to each node in the cluster using the best possible route.

7. Cluster networking is too hard to understand. There are just too many concepts of private, public, heartbeat, disabling NetBIOS, no NIC Teaming, Windows Network Priority versus Cluster Communication priority, and the list goes on. It just seems there are too many things to deal with.

Myth: This is no longer true. The old way of doing business involved providing multiple network connections between the nodes of the Cluster and dedicating at least one of them specifically to internal Cluster communications. The Internal Cluster Communications adapter that is on this isolated network could not be supported by 'teamed' network cards. In Windows Server 2008 Failover Clusters, all we ask is that there be at least two fully functioning, properly configured networks that the Cluster can use for

communications with other nodes. If this configuration is not in place, a Warning will be registered when the validation process is run. Again, a warning means that 'best practices' are not in place.

8. When I create a Cluster in Windows Server 2008, I am not asked to provide a domain user account. Can I change it later on to the account I want?

Fact: That is correct and no. The whole security model in Windows Server 2008 Failover Clusters has changed beginning with the removal of the requirement for a domain user account to run the Cluster Service. Now, the Cluster Service runs under the Local System account. Additionally, all Cluster Network Name (now called Client Access Points or CAP) resources register a Computer Object in Active Directory when they come online. The Computer Objects, by default, are placed in the Computers OU. They can be moved or pre-created elsewhere if desired. For more information, review the Failover Cluster Step-by-Step Guide: Configuring Accounts in Active Directory.

9. Connections to Cluster file shares cannot be made using IP Addresses, so now all my mapped network drives that use IP Addresses to connect to user shares will no longer work.

Fact: This is true. To connect to Cluster shared folders in Windows Server 2008 Failover Clusters requires the access point be either the NetBIOS or Fully Qualified Domain Name (FQDN) corresponding to the Network Name resource. This is because there is a new 'scoping' feature which is part of the changes that went into the product to support highly available File Servers. For more information on 'scoping,' you can refer to the File Share 'Scoping' in Windows Server 2008 Failover Clusters blog.

10. I can no longer share subdirectories like I used to in Windows Server 2003 Clusters. There is no File Share resource type any longer.

Fact: This is true. Quite a few of our customers have been impacted by the loss of this capability but are slowly but surely coming to grips with it. It helps to understand a little bit of the history behind this functionality. To assist with that, the Windows Clustering & High Availability Product Group wrote a short blog about it. It is important to understand how the new functionality works especially if you are going to migrate from Windows Server 2003 File Server clusters to Windows Server 2008. This blog also provides access to additional TechNet content on migration. Be sure to also review the new File Server Migration Toolkit as it can be used as well.

11. I've been working with Clusters for a long time and I sometimes seem to have issues with the shared storage and disk signature changes.

Myth: Working with shared storage in Windows Server 2008 Failover Clusters is a much better experience. The Cluster storage model, like the networking mode, has been redesigned. The Cluster Disk Driver (CLUSDISK.SYS) is no longer in a 'direct' line to the storage stack, but sits off to the side of the Windows Disk stack.

Access to storage is now the primary job of Partition Manager. If the Cluster Service needs to interact with storage, the Cluster Disk Driver communicates via the Partition Manager driver (PARTMGR.SYS). Additionally, the way Cluster reserves a disk is different in Windows Server 2008. Instead of using the SCSI-2 protocol features of Reserve\Release\Reset, Cluster Service now uses SCSI-3 Persistent Reservations to reserve a disk. And, for even more flexibility and stability, the Cluster Service uses multiple attributes to identify a piece of storage (Disk Signature and Disk Unique ID). If at least one of these two attributes match, the Cluster should be able to bring a disk online and update this change in its Cluster registry. This means that if only the disk signature changes, Cluster will bring the drive online and you may never even know it changed. No more DUMPCFG or Cluster Recovery!! If you are totally replacing a disk, you can add it to the nodes; bring up the properties of the disk resource being replaced and hit the REPAIR button. It will bring up another window for you to select which disk you are replacing it with and you can then bring it online. Done! No more DUMPCFG or Cluster Recovery!!

12. I cannot find the Cluster Log in the Cluster subdirectory (%windir%\cluster). Does this mean we no longer use it?

Myth: The 'Eventing' Model is new in Windows Server 2008. Actually, it is carried forward from Windows Vista into the server product. The Cluster Log has 'evolved' into a trace log format (.etl) and this logging is configured to start at boot time. To access the data in a readable text Cluster Log requires using the cluster.exe CLI to dump out the information. The Windows Clustering & High Availability Product Group has written a blog that discusses this as well as talks about using TRACERPT.EXE to convert the .ETL into a readable HTML format.

13. In Windows 2003 Cluster, if there was a resource that had problems, I had to sit there and let it fail multiple times on one node, watch it go to another node and fail there multiple times and then just bounce around. This takes time that I do not have to get to where I can correct the problem. Being that I am new to Windows 2008 Clustering, I am afraid I am going to spend unnecessary cycles waiting for this same thing to occur if I do something wrong.

Myth: In Windows Server 2008 Failover Clusters, the default recovery behavior has changed. In Windows Server 2008 Failover Clusters, the resource tries only one restart in a 15-minute period before the resource is marked as "Failed." Then, the Service group or the Application group to which the resource belongs is failed over to another node in the cluster. This new behavior improves the high availability model by giving resources only one restart attempt on each node in the cluster before a failover occurs. As soon as an unsuccessful attempt to restart a failed resource on each node in the cluster is made, the resource is marked as "Failed."

14. Trying to find information about Windows 2003 Clusters was a long tedious process sometimes as there are so many places to go on the Microsoft pages. Am I going to have the same ‘travels’ with Windows 2008?

Myth: The Windows Clustering & High Availability Product Group has created a Failover Clustering Portal which serves as a more centralized repository for high availability information. They have also created this blog that breaks things out a little more.

Hopefully, this short blog will help clear up some of the questions or concerns floating around out there about Windows Server 2008 Failover Clusters. As always, the CORE Team strives to provide information that we hope will be useful. Some of this information does originate from the types of issues we deal with on a daily basis in the various technology areas we support. Thanks for your attention.

Chuck Timon and John Marlin
Senior Support Escalation Engineers
Microsoft Enterprise Platforms Support

Configuring Auditing for a Windows Server 2008 Failover Cluster

jeffhugh — Mon, 19 Jan 2009 21:57:19 GMT

The Windows Server 2008 Failover Clustering auditing feature has been requested by customers and provides the capability to monitor, or audit, cluster access. The cluster auditing feature can be enabled to audit accesses (Success and\or Failure) of an object. The object, in this case, will be the Microsoft Failover Cluster. This is accomplished by auditing client accesses to a cluster using Cluster APIs. This basically means that a client trying to access a cluster using either the Failover Cluster Management snap-in (Cluadmin.msc), the cluster.exe command line or any custom application that calls cluster APIs, will be subjected to auditing events if configured.

The OpenCluster() call initiates a new client session by binding to an RPC Server endpoint (CluAPI RPC Endpoint). This means once a user binds to a CluAPI RPC endpoint either locally or remotely, the user will be authenticated to ensure they are a valid Cluster Administrator. This is accomplished by an AccessCheckAndAuditAlarm call inside cluster anytime there is an OpenCluster() or a Security Callback.

To demonstrate auditing in a Windows Server 2008 Failover Cluster, I will configure auditing on a 2-Node cluster. To facilitate the demonstration, I create a Domain Global Group called Cluster Admins and then place three domain users in that group (Bill, Ted and Chuck). The Cluster Admins global group will be added to each cluster node’s local administrators group so the domain users in the group will have permission to administer the cluster. A fourth Domain User, Tom, is not authorized to administer the cluster but will be used to demonstrate the auditing functionality (Tom has the Log on locally right to the cluster nodes). The auditing policy will be configured at the domain level and enabled in the cluster. Finally, we will walk through a typical scenario where this feature may come in handy.

Here is the membership for the Cluster Admins global group.

After configuring the Cluster Admins group and adding it to the local administrators group on each node, create and then configure the auditing policy on a Domain Controller.

Auditing is not enabled by default either locally, on a machine, or in the domain by way of a group policy object (GPO). Here I enable auditing at the domain level by way of a Group Policy Object (GPO) (recommended best practice) so it can be applied to various clusters distributed throughout the domain. Using the Group Policy Management snap-in, create a new GPO Link called Cluster Auditing.

Note: Consider setting the Enforced option if you want to ensure this policy is not disabled by another less restrictive policy.

Once the policy is created, Edit it to actually define the settings. The policy, as shown here, is set to audit Object access and is set to audit Successes and Failures (this is an individual\business decision and can be changed).

After the policy is configured, confirm the settings in the interface.

Then, apply the policy to the cluster nodes as shown here.

This restricts the policy to the defined objects in Active Directory. In this case, it is the Authenticated Users group and the Nodes in the 2-Node cluster.

On each node in the cluster, refresh the group policy by running the command shown here.

To verify the setting, run the gpresult /v command and examine the output.

The output also details the settings for the policy (Cluster Auditing). It should be the same as configured in the Domain.

Once the policy has been applied to the cluster nodes, configure auditing in the cluster. In the Failover Cluster Management snap-in, examine the Properties of the cluster itself. Selecting the Cluster Permissions tab, access the Advanced configuration options.

Select the Auditing tab and Add the objects you want to audit. Here, I have added the Cluster Admins, Domain Admins and Authenticated Users groups.

After completing this action, auditing is enabled. No restart of the cluster service is required. Here is an example from the Security log on one of the nodes in the cluster.

If a user (in this example Tom) that does not have permission to administer the cluster, tries to access the cluster using either the Failover Cluster Management snap-in (cluadmin.msc) or using the cluster.exe CLI, they receive an error as shown here.

The corresponding Audit Failure entry in the Security log is shown here.

Now that we have discussed how to configure a domain level auditing policy so cluster can take advantage of it, here is a simple scenario that demonstrates how this capability might be used.

Scenario

Gary is an Administrator in a corporate IT Department. He notices a highly available File Server group hosted in one of the clusters in his organization is not running on the cluster node it is supposed to. He needs to find out what happened so he reviews the cluster node system logs and does not see a failure that would explain the group moving over to another node in the cluster. He knows auditing is configured so he starts his investigation.

The first place he looks is in the Failover Cluster Operational log for the node where the group should be Online. He finds the following event in the log.

This tells him when the File Server group (CONTOSO-FS1) was taken Offline. He then inspects the corresponding Operational log on the node currently hosting the group and finds -

This tells him when the group came Online on the node.

With this information in hand, Gary inspects the Security logs on the cluster nodes and discovers one of his Cluster Admins was working on the cluster during the same timeframe.

Gary follows up with Ted to find out why the group was moved - mystery solved.

There you have it - quick and dirty on how auditing can be used with Failover Clusters in windows Server 2008. I hope this information may be of some use to you. Let us know.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support