Rob's SQL Server Blog : Clustering

SQL Server 2005 Multi-Site Clustering with Windows Server 2008

robcarrol — Mon, 16 Mar 2009 01:12:00 GMT

I was working recently with a customer who was looking to deploy a SQL Server 2005 cluster across 2 geographically dispersed sites using Windows Server 2008. They were looking to utilise the new clustering improvements in Windows Server 2008 to build a highly available SQL Server solution. The customer required automatic failover between the sites in the event of a disaster, but their current solution required manual intervention by an administrator in order to failover to the disaster recovery site. Automatic failover would increase application availability, and reduce the complexity of the solution. Each site has it's own SAN storage and the customer planned to replicate data between each site using SRDF replication.

This led me to do further research into clustering SQL Server in this type of environment. Windows Server 2008 introduces greater flexibility in the choice of Quorum configuration. The concept of quorum moves away from the requirement of a shared storage resource, but now refers to the number of votes needed to establish a majority. All nodes and a witness resource can get a vote, which removes the disk as the single point of failure as in previous clustering models. The 4 Quorum Models available are:

Node and Disk Majority
Disk Only
Node Majority
Node and File Share Majority

As there is no shared storage between the nodes in a multi-site cluster, 2 of these Quorum models are suitable for multi-site clustering: Node Majority and Node and File Share Majority. Node and Disk Majority and Disk Only should only be used in a multi-site cluster if specifically directed by your storage vendor as your disk replication software needs to support these configurations.

Node and File Share Majority:

This allows the creation of up to 16 nodes with no shared disks. A file share acts as a witness, meaning that a 2 node cluster will have have 3 votes, so connectivity can be lost by either one of the nodes or the witness and the cluster can still continue to function.

A cluster quorum configured to use a node-and-file-share majority is a great solution for multi-site clusters. The file share witness can reside at a third site independent of either site hosting a cluster node for high disaster resilience. A single file server can serve as a witness to multiple clusters (with each cluster using a separate file share witness on the file server).

This configuration gives the highest resilience as the cluster can automatically recover from the loss of any one site without manual intervention.

The File Share Witness (FSW) needs to be in the sane forest as the nodes and be running Windows Server 2003 or Windows Server 2008. For maximum resilience, it is best to locate the FSW at a 3rd site separate from the cluster nodes. The FSW does not need to be attached to shared storage and should NOT be a node in the same cluster.

Node Majority:

A node-majority cluster consists of 3 or more nodes without shared storage. Each of the nodes has a vote and there is no shared disk vote. A majority of votes are necessary to operate the cluster if 2 nodes fail in a 3 node cluster, then the remaining node drops out of the cluster. An administrator can manually over-ride this and force the remaining node to start. When the other nodes come back, majority quorum is achieved again and the cluster comes back online seamlessly.

This configuration works best with an odd number of cluster nodes as it is not enough to have half the cluster nodes functioning in this model. If four nodes were set up in a node-majority configuration, the cluster would continue to operate with the loss of one node but not with the loss of two nodes. You should use an odd number of nodes with Node Majority as 4 nodes can only survive 1 failure, which is the same as 3 nodes.

The node-majority quorum configuration can work when there is more than one cluster node at each site. Consider a multi-site cluster consisting of five nodes, three of which reside at Site A and the remaining two at Site B. With a break in connectivity between the two sites, Site A can still communicate with three nodes (which is greater than 50 percent of the total), so all of the nodes at Site A stay up. The nodes in Site B are able to communicate with each other, but no one else. Since the two nodes at Site B cannot communicate with the majority, they drop out of cluster membership. (Were Site A is to go down in this case, in order to bring up the cluster at Site B, it would require manual intervention to override the non-majority.)

As a result, the Node Majority configuration does not give automatic failover between sites as nodes 4 and 5 cannot achieve quorum. In this situation, you would need to manually force a failover.

SQL Server Networking Considerations:

Windows Server 2008 now allows nodes in the same cluster to reside in different network subnets and communicate across network routers. However, be aware that SQL Server 2005 and 2008 still require all cluster nodes to reside in the same network subnet, so you will still need to set up virtual local area networks (VLANs) to connect geographically separated cluster nodes. This can have some benefits with regard to client response times though, as DNS replication may impact client re-connection times in the event of a failover from one site to another. VLAN's allow DNS names to stay the same, so can increase availability.

Storage Considerations:

As there is no shared storage between the nodes in a multi-site cluster, the main consideration is how to keep the data replicated between the sites. The choice of 3rd-party replication solution is important and can have a major effect on how you deploy your cluster. As such, you should work closely with your storage vendor from an early stage in the design process.

Synchronous replication results in no data loss, but requires shorter distances between nodes and higher bandwidth to avoid write latency from impacting performance. Asynchronous allows you to stretch cluster nodes across longer distances, however there is a potential for data loss in the event of a failure. Asynchronous data replication also assumes a large enough network bandwidth to keep up with data changes and does not significantly impact application performance.

Conclusion:

As data replication is key in a multi-site SQL Server cluster, work with your storage vendor from an early stage to ensure they support your cluster configuration. Multi-site clustering allows you to achieve high availability and disaster recovery, however it can be a costly and complex solution. You should evaluate your business requirements first and then decide on the best technology to meet these. It could be the case that Database Mirroring, for example, could be used to give you the required level of resilience across geographical sites.

In this case, the customer chose to implement a 2-node, 3-site solution using the Node and File Share Majority quorum model, with a File Share Witness located in the 3rd site. This gives site-level resilience in the event of a disaster and also allows automatic failover between the cluster nodes without having to re-write client applications, meeting the business requirements.

Additional Resources:

Windows Server 2008 Multi-Site Clustering Whitepaper

TechNet Webcast: Geographically Dispersed Failover Clustering in Windows Server 2008

TechNet Webcast: Failover Clustering and Quorum in Windows Server 2008 Enterprise

Support Webcast: Microsoft SQL Server 2005 Failover Clustering on Windows Server 2008

How to: Create a New SQL Server 2005 Failover Cluster (Setup)

Support Webcast: Microsoft SQL Server 2005 Failover Clustering on Windows Server 2008

robcarrol — Mon, 19 May 2008 22:53:00 GMT

A new Level 300 support webcast has been scheduled for Monday the 9th of June, 10:00 AM Pacific Time. The summary of the session is as follows:

This Support WebCast focuses on how to plan, implement, and administer a Microsoft SQL Server 2005 failover cluster on Windows Server 2008. This session provides step-by-step instructions about how to install SQL Server 2005 clustered instance on a Windows Server 2008 cluster. It also discusses the options you can use to move SQL Server 2005 failover cluster from Windows Server 2003 to Windows Server 2008.

http://support.microsoft.com/default.aspx?scid=kb;en-us;953170&sd=rss&spid=2855

Windows Failover Clustering Overview

robcarrol — Wed, 07 May 2008 19:07:00 GMT

The host node in the failover cluster performs a "looks alive" check every 5 seconds. An IsAlive check is performed every 60 seconds using SELECT @@SERVERNAME. If this fails the IsAlive retries 5 times and then attempts to reconnect to the instance of SQL. If all fail, then the SQL Server resource fails. Depending on the failover threshold, configuration of SQL resource, Windows Clustering will either attempt to restart on same node or failover to another available node.

During failover, Windows Clustering starts the SQL Server service for that instance on the new node, and goes through the recovery process to start the databases. After the service is started and the master database is online, the SQL Server resource is considered to be up. User databases will then go through the normal recovery process: any completed transactions in the t-log are rolled forward, and any incomplete transactions are rolled back. The length of the recovery process is dependent on how much activity must be rolled forward or rolled back upon startup.

Set the recovery interval of the server to a low number to avoid long recovery times and to speed up the failover process. SQL Server generates automatic checkpoints based on the "recovery interval" setting. Long running transactions can lead to much longer restart times than specified in the recovery interval option.

Failover/Failback Strategies

The cluster group containing SQL Server can be configured for automatic failback to the primary node when it becomes available again. By default, this is set to off.

To Configure:

Right-click the group containing SQL Server in the cluster administrator, select 'properties' then 'failback' tab.
To prevent an auto-failback, select 'Prevent Failback', to allow select 'Allow Failback' then one of the following options:

Immediately: Not recommended as it can disrupt clients
Failback between n and n1 hours: allows a controlled failback to a preferred node (if it's online) during a certain period.

Configure Node Failover Preferences

When you use more than 2 nodes, it's important to consider which nodes should own resources in the event of a failover. For example, in an n+1 configuration, each SQL Server group should have the idle node second in the list of preferred owners. N.B. Do not use cluster admin to remove nodes from the resource definition. USe SQL Server setup for that functionality.

To Configure:

Right-click SQL Server group in the cluster administrator and select properties
On the 'General' tab, the preferred owners list box contains all cluster nodes that can potentially own resources in that group, and the current order in which they will failover
Click 'Modify' to change this order

Configure Thresholds for a Resource

Right-click the cluster resource and then select 'Propereties'
Click 'Advanced'
Select 'Do not restart' if the cluster service should not attempt to restart. Restart is the default
If 'Restart' is selected:

Affect the Group: uncheck to prevent a failure of the selected resource from causing the SQL Server group to failover
Threshold: number of times the cluster service will attempt to restart the resource, and period is the amount of time in seconds between retries

Do not modify the 'LooksAlive' and 'IsAlive' settings
Unless necessary, do not alter the 'Pending Timeout'. This is the amount of time the resource is either in the online or pending or offline pending states before the the cluster service puts it in either offline or failed state

Configure Thresholds for a Group

Right-click the group containing the SQL Server virtual server then click properties
Click the failover tab
to configure the failover policy, in the threshold box enter the number of times the group is configured to failover within a set span of hours. In the period box, entrer the set span of hours
Once the resource group reaches the set number of failovers, it will stay offline. However, other cluster resources, such as cluster IP, could be left online

Cluster Resource Dependencies

Resource	Dependency
SQL IP Address (Virtual Server Name)	NONE
SQL Network Name (Virtual Server Name)	SQL IP Address
SQL Server	Disk Resource(s), SQL Network Name
SQL Server Agent	SQL Server
SQL Server Full Text	Disk Resource(s)
Analysis Services (2005 only)	Disk Resource(s), Network Name

Cluster Heartbeat

Cluster nodes use the "heartbeat" signal to check whether each node is alive at both the OS level and SQL Server level. The node hosting the SQL Server resources uses the Service Control Manager to check every 5 seconds whether the SQL Server service appears to be running. This "LooksAlive" check does not impact performance but does not perform a thorough check; the check will succeed if the service appears to be running even though it might not be operational. As a result, a deeper check must be performed; this "IsAlive" check runs every 60 seconds.

IsAlive:

Runs every 60 seconds
Runs an @@SERVERNAME T-SQL query against SQL Server to determine whether the server can respond to requests
Does not gaurantee that all user databases are available or are performing within necessary performance/response-time requirements

If IsAlive Check fails:

Retried 5 times and then it attempts to reconnect to the instance of SQL Server
If all 5 retries fail, the server resource fails
Depending on the failover threshold config, the failover cluster will either restart the resource on the same node or it will failover to another available node

The IsAlive query tolerates a few errors, but ultimately it fails if it's threshold is exceeded

During failover of the SQL Server instance, SQL Server resources start up on the new node and SQL Server goes through the recovery process to restart the databases. After the service is started and the master database is alive, the SQL Server resource is considered to be up. User databases will go through the normal recovery process. Completed transactions in the transaction log are rolled forward (the Redo phase), incomplete transactions are rolled back (the Undo phase).

In SQL Server 2005 Enterprise Edition, each user database is available to the user once the Redo phase is complete. For all other editions (and all 2000 editions), each user database is unavailable until the Undo phase completes. Length of recovery process depends on how much activity needs to be rolled forward or back upon startup.

The 'recovery interval' sp_configure option of the server can be set to a low number to avoid longer Redo recovery times and to speed up the failover process. Undo recovery time can be reduced by using shorter transactions so that uncommitted transactions do not have much to roll back.

Recommended Heartbeat Configurations

Two or more independent networks must connect the nodes of the cluster to avoid a single point of failure
Use of 2 LAN's is typical (MS PSS does NOT support the config of a cluster with nodes connected by only one network)
At least two of the cluster networks must be configured to support heartbeat communications between the cluster nodes to avoid a single point of failure
To do so, configure the roles of these networks as either "Internal Cluster Communications Only" or "All Communications" for the cluster service
Typically, one of these networks is a PRIVATE INTERCONNECT dedicated to internal cluster communication.
Each cluster network must fail independently of all other cluster networks.

The cluster networks must not have a component in common that can cause both to fail simultaneously.
The use of a multiport network adapter, for example to attach a node to two cluster networks would not satisfy this requirement in most cases as the ports are not independent

Remove all unnecesary network traffic from the network adapter that is set to INTERNAL CLUSTER COMMUNICATIONS ONLY (also known as the "heartbeat" or "private" network adapter, to eliminate possible communication issues
Clustering communicates using Remote Procedure Calls (RPC) on IP sockets with User Datagram Protocol (UDP) packets