Aeval Shah's Blog

Windows Server 2012 Failover Clustering Dynamic Quorum - Revisited

2013-06-05T09:10:52Z

This is a follow on post to my original dynamic quorum post which can be found here. I have had a lot of scenario based questions around dynamic quorum and it’s behaviour and I have done some extensive testing to try and explain some scenarios with detail.

Scenario 1: Explaining dynamic quorum using a five node cluster and gracefully shutting down nodes in a sequential manner.

In this example I will turn off the cluster service on the cluster nodes one at a time. The quorum model has been set to node majority.

All nodes in the cluster are up and running.

Notice the ID's of the nodes. These ID's correspond to the joining order of the node in the cluster. This is the same as the HKLM\Cluster\Nodes registry key.

I stop nodes 2, 3 & 4 sequentially (using failover cluster manager to stop the cluster service). You can see below the nodes are down but more importantly my cluster is up and running. To maintain quorum a single vote is required which is coming from node 5. Through testing I have found the node with the highest ID will be the one casting the dynamic vote.

To prove the above I started node 2 and 3 and also stopped node 5 and 1. If my theory is correct the dynamic vote will be cast by node 2 as it has the highest ID. This can be seen in the next screen shot.

Now for the last man standing if I stop node 2 the dynamic weight should be transferred to node 3. You can see from the screen shot below that this is true. My cluster remains up and running as dynamic quorum has been able to calculate quorum throughout my testing i.e. quorum was maintained as the cluster service was turned off on some nodes and turned on other nodes.

This is all well and good and shows how dynamic quorum works but in the real world we do not always have the luxury of gracefully stopping the cluster service. In the next set of tests I will simulate failures as opposed to graceful shutdowns.

Scenario 2: Explaining dynamic quorum using a five node cluster and simulating failures.

In this example I will turn off the cluster node at the Hyper-V level. I will turn off (not shutdown) each node one at a time and show you what happens. The quorum model has been set to node majority.

All nodes up, running and voting to achieve quorum.

Turning off nodes 2 and 3 one by one.

Critical events have been logged by the cluster and you can see the results below from the remaining nodes

Dynamic quorum voting looks like this

Now depending on the node I turn off the node with the highest ID that is still up will have the dynamic vote to maintain the cluster.

Turn off node 1. Of the nodes remaining the node with the highest ID is node 4 therefore that should be the one casting the vote to maintain the quorum.

As you can see the above is correct. Node 4 is the one casting the vote.

Now for the last man standing scenario. I need to turn off node 4 and the dynamic weight needs to shift to node 5. This will then be the only node in the cluster up and running. However this is not the case when a cluster node goes down unexpectedly.

Conclusion

After a lot of testing I have come to the following conclusions;

Dynamic quorum will work for an x node cluster all the way down to a single node if cluster nodes are shutdown gracefully in a sequential manner allowing the cluster to maintain quorum and then calculate dynamic quorum.
If the nodes in the cluster are turned off i.e. power unplugged the cluster will try to maintain quorum. If it can it will then calculate the new dynamic quorum value. This will work all the way down to the last two nodes of the cluster.
Once you are down to the last two nodes and you turn off the node with the dynamic weight the cluster will then go down. See the screenshot below. This is happening as the cluster does not have a chance to sync up with the node turned off as there is no more communication therefore it cannot transfer the vote across.

Once you are down to the last two nodes and you turn off the node which is not voting the cluster will remain up. See the screen shot below shows this.

So in the real world the last man standing scenario will work if the last node online is the node which has the dynamic vote. Transfer of the vote is not possible if the node voting is turned off. It is only possible if there is a graceful shutdown.

If you find yourselves in a scenario where you are on the last two nodes of the cluster it may be worthwhile failing over services to the node with the dynamic vote as that will be the node that remains up if the other node goes down.

I hope that helps some of the scenario based questions for dynamic quorum.

Aeval

Premier Field Engineer - Failover Clustering & Virtualisation

Configuring Virtual Machines to run in separate Resource Hosting Subsystem (RHS) Processes

2013-02-05T15:48:51Z

I was recently working with a customer who was facing deadlock issues with their Windows Server 2008 R2 Hyper-V cluster. The RHS process would be terminated and a new process would spawn which meant virtual machines would be recycled. As a result of this my customer decided to isolate each virtual machine in it's own RHS process.

My task was to work out how to do this without causing downtime to the virtual machines and hosts. They wanted a simple way to set this up and also a way to find out which virtual machines were associated to which RHS process.

In this post I will discuss the steps I took to show and confirm the isolation of virtual machines in their own RHS process and how this could be achieved without any down time.

To understand what the RHS process is and how it is used I suggest you take a look at this article. It gives a nice explanation of the RHS process and how it fits into Failover Clustering. This article shows how you can quickly understand the cause of the RHS instability.

In order to proceed I configured my test lab as a three node cluster running a few virtual machines.

Step1. – Check VM is running in default RHS process.

The first step was to determine how to find out what process the VM was running in. For this example I was looking at a single VM (W2K8R2-Node1).

Below are extracts from Task Manager screen shots from all three cluster nodes to show how may RHS processes are running.

W2K8-CN1 – Cluster Node

W2K8-CN2 – Cluster Node

W2K8-CN3 – Cluster Node

The screenshot below confirms the VM and VM configuration have the default settings for the resource monitor process. The checkbox to run the resource in a separate Resource Monitor has not been checked.

This is as expected as by default there are only two RHS processes running. One RHS process is used to run cluster resources and the other is used for the storage.

Running the following PowerShell command will give me the list of processes the VM and VM configuration are running in.

#Get a list of VMs and correlate to RHS.exe process on a per VM basis

PS C:\> Get-ClusterResource |?{$_.OwnerGroup -like "W2K8R2-Node1"} | ft Cluster, OwnerNode, OwnerGroup, SeparateMonitor, ResourceType, @{Label='RHS Process ID';Expression={$_.MonitorProcessID}} -AutoSize

This is the output of the above command. From this I can see that both the VM and VM configuration are running on the RHS process with ID 2092 on host W2K8-CN2

Note: The VM and VM configuration are both listed. This means if we want to separate the VM entirely we also want to reconfigure the VM configuration to run in it’s own RHS process.

Step 2 – Change VM and VM configuration to run in their own process.

By running the following command I was able to change the VM and VM configuration to run in their own RHS process. I chose to do this via PowerShell as this would need to be done for a lot of virtual machines and the GUI would not be practical.

# Set VM to run in its own RHS process.

PS C:\> Get-ClusterResource | ?{$_.OwnerGroup -like "W2K8R2-Node1"} | %{$_.SeparateMonitor='True'}

Checking in the GUI I can see the relevant tick box has now been set

Running the following PowerShell command will give me the list of processes the VM and VM configuration are running in

#Get a list of VMs and correlate to RHS.exe process on a per VM basis

This is the output of the above command. From this I can see that both the VM and VM configuration are running on the RHS process with ID 2092 on host W2K8-CN2. Therefore setting the VM and VM configuration to run their own RHS process has no impact on the VM or VM configuration

Step 3 – Live migrate VM to spawn a new RHS process

In order to start the resource in a new RHS process the resource group needs to be moved to another node. Fortunately for a VM we can perform a Live Migration therefore the VM can start in a new RHS process without downtime.

Note: Both the VM and VM configuration should start in their own new process.

Running the following PowerShell command (once the VM has been live migrated) will give me the list of processes the VM and VM configuration are running in

#Get a list of VMs and correlate to RHS.exe process on a per VM basis

This is the output of the above command. I can see that both the VM and VM configuration are now running in their own processes.

I can also see four RHS processes on W2K8-CN1

Step 4: What happens to the RHS process after Live Migration?

We need to determine what happens to the RHS processes once the virtual machine has been live migrated to another node in the cluster.

The VM was live migrated to another host and this is what I saw on the new host. The processes are different as to what they were on the previous node.

The below shows there are four RHS processes on W2K8-CN3 which is as you would expect as the VM had been live migrated to that node. The previous host W2K8-CN1 still has four RHS processes. This implies that the live migration spawned new processes on the destination host but nothing removed the RHS processes from the original host.

RHS processes on new server (W2K8-CN3)

RHS processes on new server (W2K8-CN1)

I needed to determine what was running in those RHS process and the following command shows me just that.

#Get a list of VMs and correlate to RHS.exe process on a per cluster basis

PS C:\> Get-ClusterResource | ft Cluster, OwnerNode, OwnerGroup, SeparateMonitor, ResourceType, @{Label='RHS Process ID';Expression={$_.MonitorProcessID}} -AutoSize

This is the output. Although the two processes for VM and VM configuration on W2K8-CN1 are still up they are not running any resources (3160 & 3424).

Step 5: Live migrate the VM back to the previous host

Live migrate the virtual machine back to W2K8-CN1 which is where the first processes were spawned to isolate the VM and VM configuration

The clippings below show that both hosts still have four RHS processes running.

RHS processes on W2K8-CN1

RHS processes on W2K8-CN3

I ran the following command to check the RHS process for the VM and VM configuration .

#Get a list of VMs and correlate to RHS.exe process on a per VM basis

I can see that the process which were originally used are re-used for the VM and VM configuration .

Step 6: Remove the Isolation and see what happens.

The following command should remove the isolation but it does not work

# Set VM to run in in default RHS process. i.e. remove isolation

PS C:\> Get-ClusterResource | ?{$_.OwnerGroup -like "W2K8R2-Node1"} | %{$_.SeparateMonitor='False'}

Instead I had to use the cluster.exe command line to remove the isolation checkbox. This has to be done for both the VM and VM configuration

Remove resource isolation for VM

C:\> cluster res "Virtual Machine W2K8R2-Node1" /prop SeparateMonitor=0

Remove resource isolation for VM configuration

C:\> cluster res "Virtual Machine Configuration W2K8R2-Node1" /prop SeparateMonitor=0

We can now see the check box has been removed from the settings.

Step 7: Live migrate to see if processes are removed

Live migrate to another node and notice what happens to the RHS processes on the source server.

RHS Processes on original host after VM has been live migrated still remains at four.

RHS Process on W2K8-CN1

RHS Process on W2K8-CN3 - where the VM was live migrated to has changed to two.

Step 8: Live migrate back to original node

Live migrate the VM back to the original node. This should then only be using the default RHS process and the number of RHS processes running should drop from four to two.

RHS processes on W2K8-CN1

Note: We now have only two RHS processes

RHS processes on all hosts

W2K8-CN1

W2K8-CN2

W2K8-CN3

The following command checks what is running in each RHS process

#Get a list of VMs and correlate to RHS.exe process on a per cluster basis

PS C:\> Get-ClusterResource | ft Cluster, OwnerNode, OwnerGroup, SeparateMonitor, ResourceType, @{Label='RHS Process ID';Expression={$_.MonitorProcessID}} -AutoSize

This is the output. All VMs and VM configuration are running in the default RHS processes.

Conclusion

From the above testing I determined that the virtual machines can be isolated to their own RHS processes without causing any downtime. The VM can be live migrated from node to node and it will remain isolated from the default RHS process. The VM configuration also needs to be isolated.

There is still more testing to be done in terms of the RHS processes still running but not running any VM or VM configuration. I suspect this is by design of the cluster and an RHS process will only terminate if the thorough health check (IsAlive previously) fails.

This in itself posses some questions as there is nothing running inside the process therefore the health check will not run and therefore the process will not be terminated.

Scaling this up to a hundred virtual machines means in the cluster there will be 200 RHS processes running split over the cluster nodes. There could potentially be more as once a VM has been live migrated and spawns a new RHS process this is not terminated if the VM is live migrated again.

As you can see further testing needs to be done but this at least proves there is a way to achieve isolation without downtime. If running a lot of RHS processes does not impact performance then it may be feasible. Do these additional RHS processes cause an impact on the cluster itself especially with the health checks and the logging of information?

Depending on how my customer decides to proceed I will post an update with any further testing scenarios and results.

Aeval

Premier Field Engineer - Failover Clustering & Virtualisation

Windows Server 2012 Failover Clustering Dynamic Quorum

2012-08-21T14:58:00Z

Introduction to Quorum

Quorum in Windows 2008 R2 referred to a consensus , that is, a majority of votes is required in order to reach quorum and maintain stability of the cluster. A new option created in Windows Server 2012 which was also back ported to Windows Server 2008 R2 SP1 was the ability to stop a node being able to participate in the voting process. This meant that you could configure a node not to have a vote. This is a property of the node called NodeWeight and is configurable via PowerShell.

This is a very useful feature as you have the ability to remove the cluster nodes in a secondary site from voting and thus have the controlling votes in the primary site. This makes managing the quorum simpler and also prevents the cluster database from being updated by the nodes in the secondary site if the nodes in the primary site are down.

Another new feature introduced in Windows Server 2012 is the concept of dynamic quorum.

What is Dynamic Quorum

Dynamic quorum is the ability of the cluster to recalculate quorum on the fly and still maintain a working cluster. This is a huge improvement as we are now able to continue to run a cluster even if the number of nodes remaining in the cluster is less than 50%. This was not possible before but the dynamic quorum concept now allows us to do this. In fact we can reduce the cluster down to the last node (known as last man standing) and still maintain quorum.

Dynamic quorum is enabled by default and recalculates quorum when a node(s) in the cluster is down. Dynamic quorum will only work when these two conditions are met.

The cluster has already achieved quorum.
Sequential failures of nodes have occurred.

If multiple nodes in a cluster go down at the same time dynamic quorum will not recalculate the number of votes required to maintain the cluster. Instead a re-group will occur and once the cluster has determined quorum can be maintained dynamic quorum will kick in when a subsequent node failure occurs. As long as the two conditions above have been met.

How Dynamic Quorum Works

Below is an explanation of the behaviour of the dynamic quorum as tested in the lab. Below is a three node cluster with no failures. The quorum is set to node majority and dynamic quorum is enabled. This is the default.

As you can see all nodes are up and running in the cluster.

Using PowerShell we see the following properties for the node weights and dynamic quorum.

To demonstrate dynamic quorum and the last man standing scenario we turn off the cluster service on Contoso-N3

This is what the node weight and dynamic weight looks like. Notice that the DynamicWeight property of Contoso-N2 is set to 0. This is the dynamic quorum kicking in.

From this we can see that the dynamic quorum realised that there were two nodes left and removed the vote from one of them. From testing we have determined that the node with the lowest ID from the surviving two nodes has it's vote removed. The node with the highest ID has a vote and is used to maintain quorum.

When you have two nodes remaining in a cluster you cannot have a majority as the majority of two is two. In order to avoid the cluster from shutting down one of the votes is removed therefore we only require one vote to maintain the cluster.

Contoso-N1 is the node with the vote. Contoso-N2 had its DynamicWeight property changed to 0. We now stop the cluster service on Contoso-N1.

This is what the dynamic weights now look like. The vote was transferred from Contoso-N1 to Contoso-N2 and this the cluster remains up and running and is Contoso-N2 is truly the last man standing.

Dynamic quorum will only work if the cluster has a failure of a node or several nodes in a sequential order. Each time dynamic quorum is calculated the cluster must already have quorum in order for the new dynamic value to be implemented. These are the two rules mentioned above.

Aeval

Premier Field Engineer - Failover Clustering & Virtualisation

Windows Server 2008 R2 Failover Clustering - Best Practice Guide

2012-05-15T14:46:00Z

Overview

Hi everyone, this article is aimed at providing a best practice guide for building a Windows Server 2008 R2 Failover Cluster. The article provides details on hardware and software pre-requisites and configuration best practices. This guide also provides best practices for different areas of Failover Clustering configuration i.e. storage and network.

Pre-requisites for Failover Clustering
Hardware Requirements

It is recommended to use servers with the same or similar hardware components. You should only use hardware that is compatible with Windows Server 2008 R2. Depending on the cluster you are building you will need multiple network cards and or HBAs. Be sure to finalise the architecture prior to purchasing hardware to ensure you have everything you need before you start building the cluster.

Network Adapters

The hardware component must be compatible with Windows Server 2008 R2. If you use iSCSI the network adapters must be dedicated to either network communications or iSCSI.

Ensure the network infrastructure that connects your cluster to the network has redundancies built in. This could mean multiple network cards (teamed), multiple switches and multiple routers. The purpose of this is to ensure there is no single point on failure in the network infrastructure.

Storage Adapters

It is recommended to use identical host bus adapters (HBA) to communicate with the storage. The drivers and firmware should also be identical. If you want to use different HBA’s verify this with your hardware vendor. Multi-Path I/O software also needs to be identical on the cluster nodes. Windows Server 2008 R2 only supports SCSI Primary Commands-3 (SPC-3) therefore the backend storage must also be compatible with this. Parallel SCSI is no longer supported therefore check your storage specifications prior to building your cluster.

You must ensure that the HBA or network card used is dedicated to the storage. The network used for iSCSI cannot be used for network communications. You cannot use teamed network adapters because they are not supported with iSCSI.

Storage

You must use storage that is compatible with Windows Server 2008 R2. Failover clustering natively supports basic disks. To you use dynamic disks you will need to speak to your hardware vendor and use any software provided by them for dynamic disks.

The partition style must be either master boot record (MBR) or GUID partition table (GPT). The file system is recommended to be NTFS. The witness disk must be formatted with NTFS.

Storage Area Network (SAN) Storage

Ensure the SAN is compatible with Windows Server 2008 R2. Ensure the storage drivers, firmware and software is compatible with Failover Clustering. The storage must support SCSI Primary Commands-3 (SPC-3). In particular, the storage must support Persistent Reservations as specified in the SPC-3 standard. The miniport driver used for the storage must work with the Microsoft Storport storage driver.
Storage should be isolated to a cluster in that a LUN provided to a cluster must not be accessible from another cluster through zoning and masking. If you can have a SAN dedicated for a cluster that would be even better.

Consider using multi path I/O which provides the highest level of redundancy and availability.

Software Requirements

The servers participating In Failover Clustering must be running the same version of Windows Server 2008 R2 that supports the feature. Service packs, patches and hot fixes must be at the same level on all nodes. Mixing of full server installations and Core server installations is not supported.

Network Settings

Network adapters should ideally be identical for each network and it is recommended the configuration of the adapters be the same i.e. speed & duplex settings. This will ensure the network does not behave differently on different cluster nodes. Each private network not routed to the main network infrastructure should be assigned a unique subnet.

DNS

DNS must be running on the network and the cluster nodes must be using DNS for name resolution. When
objects are created in Active directory and DHCP is in use on the cluster nodes the relevant A records will also be created in DNS.

Domain Role

Each cluster node should be in the same Active Directory domain and they should all be running using the same role. It is recommended than each node in the clusters is a member server of the same domain.

Administrative Account

The account used to build and create the cluster must have full administrative access on the cluster nodes. A domain user account can be used but it must have Create Computer Objects and Read All Properties rights on the domain.

The account used to build the cluster will also be used to create the Computer Name Object and therefore needs the permissions mentioned above.

Server Setup & Configuration

The following sections talks about the general server configuration in terms of hardware and software. It is recommended to understand the pre-requisites before reading the following sections.

General Guidance

All nodes participating in a cluster should be identical. That is they should have the same configuration at the software level and at the hardware level. Although having identical hardware is not a pre-requisite it is recommended as it makes life easier for the server administrator when updating drivers as the same one can be used on all the nodes. It also provides consistency to the cluster.

Changes made to a node must be replicated to all other nodes of the same cluster. Any changes made should be tested in a test environment first before changes are made to production nodes.

Hardware Configuration

Specific hardware is no longer required however the cluster must pass validation i.e. no errors detected to be officially supported. In a geographically dispersed cluster as there is no shared storage you will see errors in the validation report but this is expected and the cluster will still be supported.

It is highly recommended to remove all single points of failure in terms or hardware and infrastructure.

Power Supply

It is recommended that all cluster nodes have 2 or more power supplies to provide redundancy.
If a node has multiple power supplies each power supply should be connected to a different power source to help prevent any power issues from taking out the node. Each node in the cluster should use a different power source to ensure that a single power strip does not take out the whole cluster.

Data centre infrastructure should also allow for total power loss and should have measures in place to provide power from UPS and/or on site generators.

Network Switches

Having two switches for a network allows the cluster nodes to connect to the LAN and provides redundancy from a network point of view.

For each network available (especially the public – client facing network) two network switches should be used for the cluster nodes to ensure a switch failure does not disrupt network communications.

If you are using iSCSI for storage the cluster should be connected to this network using multiple redundant switches to ensure the backed storage is always accessible.

Host Bus Adapters (HBAs)

It is recommended to use multiple single HBAs to connect to the backend storage. Dual HBAs can be a single point of failure as although they provide a multi-path to the storage a failure of the HBA can disconnect the node from the storage. Each HBA should be using a separate network infrastructure to connect to the SAN(s).

Boot from SAN

Operating System Configuration

Setting up the operating system correctly can increase performance and reliability. Additionally in the event of an operating system error the system can record activities and performance information to help troubleshoot and aid in root cause analysis. In order to capture this information performance monitoring and kernel debugging must be set accordingly.

Page File Configuration

It is good practice to place the page file on another partition other than the boot partition; however, in previous versions of Windows this configuration would not allow you to capture a memory dump. In Windows Server 2008 R2 the page file can be located on another partition which is not the same as the boot partition and will still capture memory dumps.

The Min and Max sizes of the page file should be set to the same value in order to improve performance. Having the Min and Max sizes different can result in slow system performance if the page file has to grow. The page file can become fragmented due to not having contiguous disk space to use and would have to place the data elsewhere on the disk.

The page file can be located on another LUN in cases where the boot partition is small or where the node is booting from SAN.

Kernel Memory Dumps

It is recommended to configure each cluster node to capture a kernel memory dump. A kernel memory dump captures the memory being used by the kernel at the time when the system stops unexpectedly. This information can be useful for troubleshooting and determining the root cause.

By default memory dumps are overwritten i.e. if a stop error occurs again the previous dump file will no longer exist. It is advisable to either copy the dump file to another machine for analysis or configure the server to not overwrite existing dump files.

Service Packs, Hotfixes & Critical Updates

Service packs need to be installed on each node of the cluster so that each node is on the same service pack level. This ensures device drivers, services and executable files are the same version. Hotfixes & Critical updates should also be installed on each cluster node to ensure any device drivers, services and executable files are the same version. Ensure any update installed is tested in a test environment first before deploying it to production.

Device Drivers

There are many device drivers on a system and it is highly recommended to ensure that these driver versions are consistent across all nodes of a cluster. Differences in driver versions can cause instability on a cluster and can impact the availability.
Software installed on a cluster node may also install a driver and this also needs to be mirrored across all nodes. If tools are used for troubleshooting which install drivers these should be removed once the troubleshooting has completed to ensure consistency between the cluster nodes.

Roles, Role Services & Features

Each role, role service and feature required in a cluster should be installed on each node in the cluster. If a role, role service or feature is removed from a cluster node it should also be removed from all other cluster nodes.

Installed Services

All services should be identical on each node of the cluster. Any services required for the cluster must be installed on all nodes of the cluster.

Performance Baselines

Once the cluster is in production it is good to run performance monitoring for a period of time to understand the server load. This can be used as a base line and you can compare future performance data to this base line and see where the performance has changed.

Black Box Monitoring

It is recommended to have monitoring running on all your cluster nodes. This monitoring should be setup so all the relevant counters are logging information to a binary file. Circular logging should also be used to overwrite previous data. This also ensures the captured data file does not grow too large and therefore does not eat up disk space. This black box monitoring is useful when a performance issue occurs as rather than wait for performance data to be gathered once an issue has been raised the performance data is already to hand.

The captured data can also be compared to a server baseline to see what the impact on the server is over time i.e. adding more highly available services or adding more users which consume the services.

Anti-Virus Configuration

Anti-virus products can interfere with the performance of a server and can cause issues with the cluster. The anti-virus should be configured to exclude the cluster specific resources on each cluster node. The following should be excluded on each cluster node;

(1) A witness disk if used
(2) A file share witness if used
(3) The cluster folder %systemroot%\cluster on all cluster nodes

Network Setup & Configuration

There should be adequate networks configured for the cluster to ensure there is no bottle neck for users,
cluster nodes and storage. Depending on the purpose of the cluster you may need an additional network for Hyper-V Live Migration.

Follow the pre-requisites to ensure you are following the best practices for network configuration. The binding orders of the network should be set accordingly. Typically you have the bindings in the following order Public, Private, Storage and Live Migration.

Public

The public network is the network which clients will connect to gain access to the highly available resources on the cluster. This network will be connected to the main network infrastructure in your organization. This is automatically selected when the cluster is built by determining which network has a default gateway.

Private

All networks in a cluster are used for intra cluster communication. You can fine tune this to ensure intra cluster communication uses a specific network and can fall back to another network if required. The private network should also be used for management tasks ensuring no additional overhead is being added to the public, storage and Live Migration networks.

Storage

If you are using an iSCSI network this network must be dedicated for storage. You should also consider using NIC teaming for additional redundancy.

Heartbeat

The heartbeat for the cluster nodes is set by default to send a heartbeat every second. The heartbeat configuration also by default will allow five missed heartbeats before a cluster node is deemed as unavailable. This can be configured to increase the interval and increase/decrease the threshold. This is of particular interest when using a high latency network (WAN) when implementing a geographically dispersed cluster.

Live Migration

When using Hyper-V in a Failover Cluster you may want to use the Live Migration feature. This requires a dedicated 1GB network to copy the system state and memory pages from one cluster node to another when Live Migration is initiated. Live migration is used for planned maintenance and cannot be used for unplanned outages.

The Hyper-V guests themselves should be location on a Cluster Shared Volume to assist with Live Migration and to ensure the VM is not taken offline to move a cluster disk to another node.

Storage Configuration

Ensure all HBAs on all nodes are at same hardware devices and ensure the firmware/driver for the HBA is the same version. Speak to your storage vendor with regards to drivers versions. Also read any documentation regarding the driver as it will probably highlight which version of the storport.sys driver is required.

Speak to your hardware vendor/engineer to implement LUN masking or Zoning to ensure that LUNS are isolated from each other. Ideally each cluster should be backed up by a dedicated SAN and it is recommended to have multiple independent paths to the storage.

If you are implementing an iSCSI solution for the storage it is recommended to use dedicated network interface cards so the storage traffic is independent of the public facing and intra cluster communication networks.

Cluster Setup & Configuration
Cluster Validation Wizard

The cluster validation wizard should be run before the initial cluster build takes place and it should also be run once you have completed building and configuring your cluster. It is also recommended to run the wizard every time to make a change to the cluster.

The storage tests will fail once the cluster is in production as the tests will not offline any disks that are in use unless explicitly told to do so. It is recommended to have a small LUN present on the cluster which is not is use so the storage tests can take place. This will ensure there are no issues with the storage and arbitration.

Quorum Models

The quorum model is automatically selected when you initially build the cluster. You can make changes to the quorum model on the fly without impacting the cluster. Always check the current quorum model so you are aware of what it is and check the chosen model against the recommended model.

Witness Resource

Depending on the architecture of the cluster you may want to implement a witness resource in the form of a witness disk or a file share witness. The witness disk should be 512MB in size (can be larger for large and print file clusters).

A file share witness is recommended for geographically dispersed clusters where there is no shared storage. The witness disk will contain a backup of the cluster database and this will be loaded as a 0.cluster hive on the node which currently owns the witness disk.

The file share witness does not keep a copy of the cluster database. Instead it maintains a log of the Paxos tag to provide versioning.

A single file share witness can be used by multiple clusters. It is recommended to make the file share witness highly available in its own cluster to provide the additional vote for multiple clusters. Within the file share witness folder a new folder will be created for each cluster so the data is isolated.

Core Cluster Group

The cluster group should only contain the Network Name, IP and witness disk resources (if required). No other resource should be in this group. If you are using software provided by the storage vendor you may have an additional resource within this group which is acceptable.

The dependencies for these resources should not be altered unless your storage vendor advises you to due to the software used to manage/provide LUNs to the cluster.

Other Groups

Most of the advanced settings should be left at their defaults unless specifically recommended by Microsoft or third party vendors.

Affect the group should only be cleared if the resource is non-critical i.e. secondary network is used for backups and this network allows client access for backup purposes only. As a result of this network failing, if you do not want the whole group to failover uncheck affect the group.

All resources should have all nodes listed in the Possible Owners list. You can set anti affinity rules to ensure groups are kept on separate cluster nodes to ensure the services remain highly available and that the cluster nodes split the load evenly.

Run this resource in a separate Resource Monitor should not be selected unless recommended by Microsoft for troubleshooting or by a 3rd party vendor for a specific reason. Anti-Affinity can be used to ensure the services are dispersed among the cluster nodes.

Chkdsk on Cluster Disks

When cluster disks states are changed to online pending (i.e. when a failover is initiated or when a group is moved) the system will check the dirty bit on the disk and see if it is set. If the dirty bit has been set the disk will remain in an online pending state whilst chkdsk is run on the entire volume.

This can lead to a significant amount of downtime and therefore should be avoided. You should plan on a monthly or quarterly basis to have a maintenance window during which you can manually check the dirty bit ad run chkdsk on the relevant volume(s). Below is a list of the options you can set using PowerShell or Cluster.exe.

Value	Description
0 (Default)	ShallowCheck open files in root of volume. Check dirty bit
1	FullCheck recursively on all files. Check dirty bit
2	Run chkdsk every time the volume is mounted
3	ShallowCheck. If corrupt run chkdsk. If not corrupt run chkdsk in read-only mode. Online will proceed when chkdsk is running in read-only mode
4	Never run chkdsk
5	FullCheck. If corrupt do not bring online. User intervention required.

Anti-Affinity

To ensure groups do not end up on the same node set the AntiAffinityClassName of the group. You can set this to the same string value for all groups you do not want to appear on the same node.

Network Priority

The network priority should be set for each network participating in the cluster. The metric value on the network represents a cost which determines which network to use for what communication.

By default the network that has been assigned a default gateway will have a metric value of 10,000. Other networks without a gateway defined will be assigned a value of between 1000 and 10,000. The first network without a gateway will be given a metric value of 1000 and subsequent networks will be given values in increments of 100 i.e. 1100, 1200, 1300.

Intra cluster communication occurs on networks that have the lower metric value i.e. cost. You need to set the iSCSI and Live Migration networks metric value higher than the private network. If the private network is down the next available network with the lowest will be used.

Set the network priorities according to the network design in your environment.

Failover/Failback Policies

All nodes must be able to run load that might be on them so if you have printers installed ensure the drivers are on all cluster nodes to ensure the printer can operate after a failover to any node. There should also be enough capacity in terms of resources available on a node to take on additional capacity from other nodes.

Failback causes another outage, however small and therefore it is recommended to set this to occur outside of normal working hours. In a 24x7 operation this may not be acceptable and therefore you should disable failback altogether.
Always test failover and failback prior to production to ensure each node can host the services without issue.

Security Considerations

A dedicated account to run the cluster service is no longer required. Instead a cluster name object (CNO) is created in Active Directory when the cluster is built. All further security checks are performed using the context of the CNO.

A domain admin account is required to build the cluster as it is this account that creates the CNO. Once the CNO has been created all further virtual computer objects (VCO) are created using the CNO. As with any Active Directory user there is a limit of 10 imposed on the number objects an account can create. This will need to be changed if you need the CNO to create more than 10 VCO’s.

The CNO object can be pre-staged prior to building the cluster. The pre-staged CNO needs to be set to disabled in order for the newly built cluster to use it.

You can now also set permissions on the cluster to allow users to view the cluster configuration in with read-only privileges.

Patch Management

Installing patches on a cluster can be challenging as there will be a period of time when the cluster nodes are out of sync. It is recommended to install the patches on a test cluster before deploying the updates to the production cluster. This ensures that any adverse effects are seen before impacting production.

Hotfixes are designed to address specific issues and should only be installed if you are facing the symptoms described in the relevant hotfix knowledge base article.

Security Updates should be installed as they are released by Microsoft. Once Microsoft has released the security updates these should be tested in a test environment and deployed to production once deemed ok.

Always ensure you have full backups which have been tested and verified prior to installing updates to a cluster.

Aeval Shah
Premier Field Engineer - Failover Clustering & Hyper-V

Windows Server 2008 R2 Failover Clustering - Avoiding Chkdsk on Failover

2012-05-15T14:20:00Z

Overview

In Windows Server 2003 if corruption on a physical disk was encountered the chkdsk process was run to attempt to repair the corruption. The repair attempt would occur when a disk resource was moved or failed over to another cluster node. The disk was taken offline, moved and then would be brought online. During the online stage the disk was checked to see if the dirty bit had been set and if so a chkdsk would be run.

Cluster admins would see the cluster disk in an online pending state. Once the chkdsk was completed the cluster disk would be brought online. Users would not be able to access the shares on the disk and would result in calls to the help desk. If this was a SQL Server installation using the disk SQL Server would not be brought online as it has a dependency on the disk. If this was the quorum disk the whole cluster would be offline until the chkdsk completed.

New Parameter in Windows Server 2008 R2

In Windows 2008 R2 Failover Clustering is now a new parameter (DiskRunChkDsk) that can be set, using cluster.exe or PowerShell, that determines what action to take if the dirty flag has been set on a cluster disk. The options for the setting are described below.

Value	Description
0 (default)	ShallowCheck open files in root of volume. Check dirty bit
1	FullCheck recursively on all files. Check dirty bit
2	Run chkdsk every time the volume is mounted
3	ShallowCheck. If corrupt run chkdsk. If not corrupt run chkdsk in read-only mode. Online will proceed when chkdsk is running in read-only mode
4	Never run chkdsk
5	FullCheck. If corrupt do not bring online. User intervention required.

Check Dirty Flag On Disks

There are several ways to check if the disk is marked as dirty on Windows Server 2008 R2. You can do this using a scripting language to interrogate WMI, cluster.exe, PowerShell or FSUtil. I prefer the PowerShell way as you can check all disks which are available on the node from where you run the command.

Checks all logical disks on a node for the dirty bit using PowerShell

Get-WMIObject Win32_LogicalDisk | ft DeviceID, VolumeDirty

These commands will allow you to check the dirty bit on a specified disk
Chkntfs X:

Fsutil dirty query x:

For testing you can also manually set the dirty bit of a disk
Fsutil dirty set x:

Check & Change the Default Setting

By default the parameter has a setting that will run a shallow check on the disk if it is marked as dirty when the disk is brought online. To totally avoid running chkdsk you change the parameter to 4. This will ensure chkdsk is not run even if he dirty bit has been set. You must ensure there are procedures in place to allow you manually check the disks and plan maintenance to run chkdsk on the disks manually if the flag is set.

Below are the PowerShell commands to check the parameter and set it's value.

# Check the parameter on a disk

Get-ClusterResource "<Cluster Disk Name>" | Get-Parameter | fl DiskRunChkDsk

# Example

Get-ClusterResource "Witness Disk" | Get-Parameter | fl DiskRunChkDsk

From the screen shot below we can see the current value of the property DiskRunChkDsk

# Set the parameter on a disk ( is the x is the number we want to set)

Get-ClusterResource "<Cluster Disk Name>" | Set-Parameter DiskRunChkDsk = x

# Example

Get-ClusterResource "Witness Disk" | Set-Parameter DiskRunChkDsk = 4

From the screen shot below we can see the current value of the property DiskRunChkDsk has changed to 4

You can check if the dirty bit has been set on all CSV volumes using the command

Get-WMIObject Win32_Volume | ft Caption, DirtyBitSet -autosize

For CSVs you can run the following commands to check and set the value of the parameter DiskRunChkDsk

# Check the values of the parameters

Get-ClusterSharedVolume | Get-ClusterParameter

# Set the value of the DiskRunChkDsk parameter

Get-ClusterSharedVolume | Set-ClusterParameter DiskRunChkDsk 4

Aeval Shah

Premier Field Engineer - Failover Clustering & Hyper-V