Live Migration, Cluster Shared Volumes & Networks

gmarchetti — Fri, 02 Oct 2009 02:42:49 GMT

The recommendation for people setting up live migration clusters is to isolate different kinds of traffic on their own networks:

Public network to access the cluster and the virtual machines running on it
“Private” cluster heartbeat network
“live migration” network
iSCSI network, if required to access shared storage

How do we determine what traffic goes where?

For public and private, the failover cluster manager tool is self-explanatory:

We select the appropriate cluster network properties. If we want to limit such network to private traffic, we do not allow clients to connect through it.

If we don’t want the cluster to use such network at all, e.g. because it is dedicated to iSCSI, we select the “Do not allow…” button.

How about the live migration traffic, though? It can be quite heavy, as we are copying memory pages from one host to another. We can select in which order to use cluster networks for such traffic through the failover cluster manager

The property requires some digging: expand “services and applications”, select the virtual machine in question, then in the main panel right-click on “virtual machine <name>” and you’ll see tab called “network for live migration”. You can then select and sort in order of priority the networks that you want to use. By default, live migration will select a network that is NOT used for CSV traffic. Note that you may have networks in this panel that were not selected for cluster use before. If you use iSCSI, de-select the relevant entry to make sure that the live migration traffic does not go through that network.

This brings me to cluster shared volumes. One of the great features of CSVs is that if the storage link (iSCSI, fibre) becomes unavailable for any reason on a node, storage traffic can be redirected over the cluster network to another node and hence to the storage device. But which cluster network?

Inter-node communications and CSV traffic will use the available network authorized for cluster use that has the lowest metric value. We can see the metrics with old cluster.exe

C:\Windows\system32>cluster net /prop
Listing properties for all networks:

T Network              Name                     Value
-- -------------------- ------------------------ -----------------------
SR Cluster Network 1    Name                     Cluster Network 1
MR Cluster Network 1    IPv6Addresses            <cut on purpose>
MR Cluster Network 1    IPv6PrefixLengths        <..>
MR Cluster Network 1    IPv4Addresses            <cut on purpose>
MR Cluster Network 1    IPv4PrefixLengths        <..>
SR Cluster Network 1    Address                  <..>
SR Cluster Network 1    AddressMask              <..>
S Cluster Network 1    Description
D Cluster Network 1    Role                     3 (0x3)
D Cluster Network 1    Metric                   10001 (0x2711)
D Cluster Network 1    AutoMetric               0 (0x0)
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses            <..>
MR Cluster Network 2    IPv4PrefixLengths        <..>
SR Cluster Network 2    Address                  <..>
SR Cluster Network 2    AddressMask              <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1000 (0x3e8)
D Cluster Network 2    AutoMetric               1 (0x1)

Note the 3 values:

Role: 1 for a private network, 0 for ignored by cluster, 3 for mixed traffic
Metric: the “weight” of the connection, generally in the 10,000 range for public networks, 1,000 for private ones. If a network has a default gateway, it is considered public; if not, private. Should there be more than one private or public network, the metric is incremented by 100 in order of enumeration (e.g. private network 2 will have a default metric of 1,100)
Autometric: 1 if the metric is set automatically by the cluster, 0 if you have set it manually.

So in my simple case the heartbeat network will also be used for CSV traffic. If you have more than 1 private network and you want to prioritize them, you can set the metric with cluster.exe, e.g.

C:\Windows\system32>cluster net "Cluster Network 2" /prop metric=1001

C:\Windows\system32>cluster net "Cluster Network 2" /prop

Listing properties for 'Cluster Network 2':

T Network              Name                     Value
-- -------------------- ------------------------ -----------------
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses                 <..>
MR Cluster Network 2    IPv4PrefixLengths             <..>
SR Cluster Network 2    Address                       <..>
SR Cluster Network 2    AddressMask                   <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1001 (0x3e9)
D Cluster Network 2    AutoMetric               0 (0x0)

Redirection of the traffic is automatic: if a network becomes unavailable, the next-lowest-metric one will be used. If another network with a lower metric becomes available, it will be used from that point onwards.

In Summary

By default, live migration traffic will be put on the network with the second-lowest metric. CSV traffic will be put on the the network with the lowest metric. In this simple example, I just have a public and private network, so the public one is used for live migration and the private one for csv and cluster traffic.

Windows Server 2008 High-Availability Clusters

gmarchetti — Wed, 23 Jan 2008 04:32:00 GMT

It is no secret that HPC Server 2008 will offer the option to make the head node of a HPC cluster highly available. This feature is not in beta 1, but it is being developed. It will exploit fail-over mechanisms provided by Server 2008 (enterprise edition or better), so I thought I'd mention some highlights in this area too.

High-availability clusters are difficult to set up and troubleshoot on several platforms. With Windows Server 2003 we made progress in simplifying them, but limitations are still significant:

You need a configuration that is fully and specifically certified as a cluster in order to obtain support when things go wrong.
There is very limited support for geo-clusters, because of limitations in intra-cluster communications, no awareness of storage location and cluster quorum models. Also, geo-clusters require yet another level of certification.
Writing cluster-aware applications is not easy. It requires knowledge of cluster-specific APIs in order to produce “resources” usable by the cluster software. Scripting generic application fail-over is supported, but limited in functionality.
Troubleshooting by reading cluster logs requires very deep knowledge to interpret the cryptic messages therein.

These limitations may hamper adoption, especially in such environments - like HPC - where Windows has not been popular.

Windows Server 2008 introduces some significant improvements that address most of those issues:

1. Configuration validation
A test tool is built into the product. It will analyze nodes and shared storage (if any) before they join a cluster. It can also be used as a troubleshooting tool, as long as the storage you want to analyze is offline. The tool will point out any issues with the hardware and the configuration that may make them unsuitable for a fail-over cluster. It will finally replace the cluster HCL. So, if the hardware passes validation, then the configuration is officially supported.

2. Simplified resource setup
A wizard-driven process allows you to select which roles you want to cluster (e.g. file server, print server, virtual server), then sets up cluster resources and dependencies appropriately.

3. Improved SAN support
Windows server 2008 issues persistent reservations on shared storage to establish ownership of LUNs, it does not use bus resets any longer. Bus resets are disruptive on SANs where several systems on several platforms may share the same storage bus. This implies that the storage must support persistent reservations. Shared parallel SCSI is deprecated.

4. Changed quorum model
The administrator can choose the most appropriate quorum model for the configuration. Several are possible:

Majority node sets with witness disk: each node gets a vote and so does a witness disk. The cluster will survive the failure of any 1 vote (including the shared witness disk).
Majority of nodes: storage is replicated amongst them, but does not get a vote. The cluster is active until the majority of nodes is running.
Majority of nodes and file share witness: nodes get a vote and so does a file share on a separate server. This is ideal for geo-clusters, as the witness file share can be in a 3rd site. The geo-cluster will thus survive the loss of any one site.

5. Improved networking
The nodes need no longer be on the same private subnet and the timeout of the “ping” among nodes is configurable. This makes it possible to route private traffic between locations and removes any a-priori restrictions on distance. Obviously, practical restrictions remain and will depend on how much the clustered applications and their users will tolerate.

These are just a few of the innovations available in server 2008 clusters. You may want to try them out for yourself by building a simple cluster on a set of virtual machines. You don’t need shared storage any longer, but if you want to try a quorum with witness disk, you can set up one of those machines as an iscsi target.

Get started

If you want to try Windows Server 2008 clusters, a virtual lab is available on Technet Events.
There is also an excellent screencast by David Northey on http://edge.technet.com/Media/Windows-Server-2008-Failover-Clustering/
Official training will be launched shortly.

HPC, Virtualization and Random Thoughts : High Availability

Live Migration, Cluster Shared Volumes & Networks

In Summary

Windows Server 2008 High-Availability Clusters