Rob's SQL Server Blog : Windows

SQL Server 2005 Multi-Site Clustering with Windows Server 2008

robcarrol — Mon, 16 Mar 2009 01:12:00 GMT

I was working recently with a customer who was looking to deploy a SQL Server 2005 cluster across 2 geographically dispersed sites using Windows Server 2008. They were looking to utilise the new clustering improvements in Windows Server 2008 to build a highly available SQL Server solution. The customer required automatic failover between the sites in the event of a disaster, but their current solution required manual intervention by an administrator in order to failover to the disaster recovery site. Automatic failover would increase application availability, and reduce the complexity of the solution. Each site has it's own SAN storage and the customer planned to replicate data between each site using SRDF replication.

This led me to do further research into clustering SQL Server in this type of environment. Windows Server 2008 introduces greater flexibility in the choice of Quorum configuration. The concept of quorum moves away from the requirement of a shared storage resource, but now refers to the number of votes needed to establish a majority. All nodes and a witness resource can get a vote, which removes the disk as the single point of failure as in previous clustering models. The 4 Quorum Models available are:

Node and Disk Majority
Disk Only
Node Majority
Node and File Share Majority

As there is no shared storage between the nodes in a multi-site cluster, 2 of these Quorum models are suitable for multi-site clustering: Node Majority and Node and File Share Majority. Node and Disk Majority and Disk Only should only be used in a multi-site cluster if specifically directed by your storage vendor as your disk replication software needs to support these configurations.

Node and File Share Majority:

This allows the creation of up to 16 nodes with no shared disks. A file share acts as a witness, meaning that a 2 node cluster will have have 3 votes, so connectivity can be lost by either one of the nodes or the witness and the cluster can still continue to function.

A cluster quorum configured to use a node-and-file-share majority is a great solution for multi-site clusters. The file share witness can reside at a third site independent of either site hosting a cluster node for high disaster resilience. A single file server can serve as a witness to multiple clusters (with each cluster using a separate file share witness on the file server).

This configuration gives the highest resilience as the cluster can automatically recover from the loss of any one site without manual intervention.

The File Share Witness (FSW) needs to be in the sane forest as the nodes and be running Windows Server 2003 or Windows Server 2008. For maximum resilience, it is best to locate the FSW at a 3rd site separate from the cluster nodes. The FSW does not need to be attached to shared storage and should NOT be a node in the same cluster.

Node Majority:

A node-majority cluster consists of 3 or more nodes without shared storage. Each of the nodes has a vote and there is no shared disk vote. A majority of votes are necessary to operate the cluster if 2 nodes fail in a 3 node cluster, then the remaining node drops out of the cluster. An administrator can manually over-ride this and force the remaining node to start. When the other nodes come back, majority quorum is achieved again and the cluster comes back online seamlessly.

This configuration works best with an odd number of cluster nodes as it is not enough to have half the cluster nodes functioning in this model. If four nodes were set up in a node-majority configuration, the cluster would continue to operate with the loss of one node but not with the loss of two nodes. You should use an odd number of nodes with Node Majority as 4 nodes can only survive 1 failure, which is the same as 3 nodes.

The node-majority quorum configuration can work when there is more than one cluster node at each site. Consider a multi-site cluster consisting of five nodes, three of which reside at Site A and the remaining two at Site B. With a break in connectivity between the two sites, Site A can still communicate with three nodes (which is greater than 50 percent of the total), so all of the nodes at Site A stay up. The nodes in Site B are able to communicate with each other, but no one else. Since the two nodes at Site B cannot communicate with the majority, they drop out of cluster membership. (Were Site A is to go down in this case, in order to bring up the cluster at Site B, it would require manual intervention to override the non-majority.)

As a result, the Node Majority configuration does not give automatic failover between sites as nodes 4 and 5 cannot achieve quorum. In this situation, you would need to manually force a failover.

SQL Server Networking Considerations:

Windows Server 2008 now allows nodes in the same cluster to reside in different network subnets and communicate across network routers. However, be aware that SQL Server 2005 and 2008 still require all cluster nodes to reside in the same network subnet, so you will still need to set up virtual local area networks (VLANs) to connect geographically separated cluster nodes. This can have some benefits with regard to client response times though, as DNS replication may impact client re-connection times in the event of a failover from one site to another. VLAN's allow DNS names to stay the same, so can increase availability.

Storage Considerations:

As there is no shared storage between the nodes in a multi-site cluster, the main consideration is how to keep the data replicated between the sites. The choice of 3rd-party replication solution is important and can have a major effect on how you deploy your cluster. As such, you should work closely with your storage vendor from an early stage in the design process.

Synchronous replication results in no data loss, but requires shorter distances between nodes and higher bandwidth to avoid write latency from impacting performance. Asynchronous allows you to stretch cluster nodes across longer distances, however there is a potential for data loss in the event of a failure. Asynchronous data replication also assumes a large enough network bandwidth to keep up with data changes and does not significantly impact application performance.

Conclusion:

As data replication is key in a multi-site SQL Server cluster, work with your storage vendor from an early stage to ensure they support your cluster configuration. Multi-site clustering allows you to achieve high availability and disaster recovery, however it can be a costly and complex solution. You should evaluate your business requirements first and then decide on the best technology to meet these. It could be the case that Database Mirroring, for example, could be used to give you the required level of resilience across geographical sites.

In this case, the customer chose to implement a 2-node, 3-site solution using the Node and File Share Majority quorum model, with a File Share Witness located in the 3rd site. This gives site-level resilience in the event of a disaster and also allows automatic failover between the cluster nodes without having to re-write client applications, meeting the business requirements.

Additional Resources:

Windows Server 2008 Multi-Site Clustering Whitepaper

TechNet Webcast: Geographically Dispersed Failover Clustering in Windows Server 2008

TechNet Webcast: Failover Clustering and Quorum in Windows Server 2008 Enterprise

Support Webcast: Microsoft SQL Server 2005 Failover Clustering on Windows Server 2008

How to: Create a New SQL Server 2005 Failover Cluster (Setup)

Updated: Support Policy for SQL Server on Virtualised Platforms

robcarrol — Wed, 08 Oct 2008 18:48:00 GMT

The support policy for SQL Server running on virtualised platforms has now been updated to include support for SQL Server 2005 running on Hyper-V

http://support.microsoft.com/?id=956893

Microsoft provides technical support for SQL Server 2005 and for SQL Server 2008 that are running in the following hardware virtualization environments:

Windows Server 2008 with Hyper-V
Microsoft Hyper-V Server 2008
Configurations that are certified through the Server Virtualization Validation Program (SVVP) http://windowsservercatalog.com/svvp.aspx?svvppage=svvp.htm

Any non-Microsoft virtualization software that is not a validated configuration through the SVVP program will be handled under the policy in the following KB article:

http://support.microsoft.com/kb/897615/

The following restrictions apply to support for SQL Server products in any hardware virtualization environment:

Guest Failover Clustering is not supported in a virtual machine.
The virtual machine must be running an operating system that is supported for the specific SQL Server version and edition and that is supported for virtualization
The SQL Server product must be a supported version under its current Microsoft Support Lifecycle policy http://support.microsoft.com/?pr=lifecycle

Support Policy for SQL Server on Hyper-V

robcarrol — Tue, 19 Aug 2008 21:10:00 GMT

Microsoft have now published the support policy for running SQL Server on Hyper-V. There is support for SQL Server 2008 initially, with plans to include SQL Server 2005 at a later date.

http://support.microsoft.com/?id=956893

Weird Problem with Virtual PC 2007

robcarrol — Wed, 16 Jul 2008 18:54:00 GMT

Had a weird problem with Virtual PC 2007 where the blue screen would flash up when I ran the program, yet the console window would not appear even though Virtual PC was running in the task manager and the system tray. The Virtual PC console had effectively disappeared. All this was bad enough, but this happened to me when I was visiting a customer site and just about to demo some cool new features of SQL Server 2008 ! I had even checked my demos before setting out that morning, so this was very weird (and not to mention embarrassing) ! After uninstalling and reinstalling and scratching my head for a while, I came across the solution here...

http://www.essjae.com/blog/default.asp?id=43

Sysinternals Live Beta Launched

robcarrol — Mon, 02 Jun 2008 02:10:00 GMT

Microsoft has launched the Sysinternals Live beta, allowing execution of Sysinternals tools directly from the web. Enter a tool’s Sysinternals Live path into Windows Explorer or a command prompt as \\live.sysinternals.com\tools\<toolname> or view the entire Sysinternals Live tools directory in a browser at http://live.sysinternals.com.

http://technet.microsoft.com/sysinternals

Hyper-V RC1 Released

robcarrol — Wed, 21 May 2008 18:32:00 GMT

Hyper-V Release Candidate 1 for Windows Server 2008 was released on the 20th May. This is a full functionality release and provides improvements to security, stability, performance, user experience, forward compatibility of configurations, and the programming model. Further details and downloads available here:

http://www.microsoft.com/virtualization/ws08.mspx

It has also been announced that Microsoft have been using Hyper-V in production for several months to host the MSDN and TechNet websites:

http://blogs.technet.com/virtualization/archive/2008/05/20/msdn-and-technet-powered-by-hyper-v.aspx

Support Webcast: Microsoft SQL Server 2005 Failover Clustering on Windows Server 2008

robcarrol — Mon, 19 May 2008 22:53:00 GMT

A new Level 300 support webcast has been scheduled for Monday the 9th of June, 10:00 AM Pacific Time. The summary of the session is as follows:

This Support WebCast focuses on how to plan, implement, and administer a Microsoft SQL Server 2005 failover cluster on Windows Server 2008. This session provides step-by-step instructions about how to install SQL Server 2005 clustered instance on a Windows Server 2008 cluster. It also discusses the options you can use to move SQL Server 2005 failover cluster from Windows Server 2003 to Windows Server 2008.

http://support.microsoft.com/default.aspx?scid=kb;en-us;953170&sd=rss&spid=2855

SQL Server and Windows Memory

robcarrol — Thu, 15 May 2008 17:02:00 GMT

SQL Server 2000 Enterprise Edition introduced support for the use of Windows 2000 Address Windowing Extensions (AWE) to address 8GB of memory on Windows 2000 Advanced Server and 32GB of memory on Windows 2000 Datacentre. With AWE, SQL Server can reserve memory not in use by other applications and the OS. Each instance must statically allocate the memory it needs. AWE memory can only be used for the data cache and not for executables, drivers, DLL's, etc.

SQL Server 2000 on Windows 2003/Windows 2000 Advanced Server/Windows 2000 Datacentre:

The use of /PAE and AWE allows SQL Server 2000 to use more than 4GB of RAM. Without /PAE, SQL can only use up to 3GB (with /3GB switch enabled in boot.ini). To allow AWE to use the the memory range above 16GB on Windows 2000 Datacentre, the /3GB switch must not be enabled in the server's boot.ini file. When you allocate SQL Server AWE memory on a 32GB system, Windows 2000 may require at least 1GB memory to manage AWE.

SQL Server 2000 Enterprise and Developer Editions are the only editions that can use AWE.

USING AWE MEMORY (SQL Server 2000)

Run the database engine under a Windows account with the "lock data pages in memory" option enabled. SQL Server setup will automatically grant this permission. If you start an instance from the command prompt (sqlservr.exe), you must manually assign this permission using gpedit.msc. Set the sp_configure option AWE ENABLED and restart SQL Server.

Instances of SQL Server 2000 do not dynamically manage the size of the address space when you enable AWE memory. When you enable AWE and start an instance of SQL 2000, one of the following occurs:

If sp_configure max server memory has been set and there are at least 3GB RAM free on the computer, the instance acquires the amount of memory specified in max server memory
If the amount of memory available is less than max server memory (but more than 3GB), then the instance acquires almost all of the available memory and may leave only up to 128MB of memory free
If max server memory has not been set and there is at least 3GB of free memory available, then the instance acquires almost all of the available memory and may leave only up to 128MB of memory free
If there is less than 3GB of free memory available, memory is dynamically allocated and, regardless of the parameter setting for AWE ENABLED, SQL Server will run in non-AWE mode

Evaluating Memory Usage

Use System Monitor to obtain the correct amount of SQL Server memory usage:

Total Server Memory (KB)
Select memory usage from sysperfinfo

Running Multiple instances with AWE on SQL Server 2000

Each instance must have a max server memory setting.

Sum of max server memory values for all instances should be less than the amount of physical memory on the server. If the sum greater, some of the instances will either not start or will have less memory than is specified in max server memory.

Windows 2000 Usage Considerations:

Before configuring Windows 2000 for AWE memory:

Add the /PAE switch to boot.ini
Add the /3GB switch to enable Windows 2000 Advanced Server and Windows 2000 Datacentre to support a 3GB virtual address space. This allows user applications to address 3 GB of virtual memory and reserves 1 GB of virtual memory for the operating system. However, if there is more than 16 GB of physical memory available on a computer, Windows 2000 needs 2 GB of virtual memory address space for system purposes and therefore can support only a 2-GB virtual address space.
In order to allow AWE to use the memory range above 16 GB, be sure the /3gb parameter is not in the boot.ini file. If it is, Windows 2000 will be unable to address any memory above 16 GB. When allocating SQL Server AWE memory on a 32-GB system, Windows 2000 may require at least 1 GB of available memory to manage AWE. Therefore, when starting an instance of SQL Server with AWE enabled, it is recommend you do not use the default max server memory setting, but instead limit it to 31 GB or less

Using Failover Clustering:

If you are using SQL Server 2000 failover clustering and AWE memory, you must ensure that the summed value of the max server memory settings for all the instances is less than the lowest amount of physical RAM available on any of the servers in the failover cluster. If the failover node has less physical memory than the original node, the instances of SQL Server 2000 may fail to start or may start with less memory than they had on the original node.

SQL SERVER 2005

To enable AWE for SQL Server 2005:

Run the SQL Server account under a Windows account with "Lock Pages in Memory" permissions assigned.
Use sp_configure to set "awe enabled" option to 1
The "Maximize Data Throughput for Network Application" in Control Panel must be selected.

Support for AWE is available only in the SQL Server 2005 Enterprise, Standard and developer editions and only applies to 32-bit OS's. SQL Server 2005 Analysis Services (SSAS) cannot take advantage of AWE mapped memory. If available physical memory is less than the user mode VAS, AWE cannot be used.

Lock pages in memory: Prevents paging the data to virtual memory or disk. Set to OFF by default in SQL Server 2005.

SQL Server 2005 Enterprise Edition can access up to 64GB of memory on Windows 2000 and Windows Server 2003.

Instances of SQL Server 2005 running on Windows 2000 use static AWE memory allocation, instances running on Windows Server 2003 use dynamic AWE memory allocation.

To support over 4GB of physical memory, the /PAE switch must be added to boot.ini file. If more than 16GB of physical memory, the OS needs 2GB of virtual address space for system purposes and can therefore support only a 2GB user mode virtual address space. Remove /3GB from boot.ini file in order to use this memory above 16GB.

The SQL Server buffer pool can fully utilise AWE mapped memory, however only database pages can be dynamically mapped to and unmapped from SQL Server's virtual address space and take full advantage of memory allocated through AWE.

AWE does not directly help supporting additional users, threads, databases, queries and other objects that permanently reside in the virtual address space.

SQL Server 2005 Features Comparison

Windows Memory (PAE/AWE/3GB)

robcarrol — Thu, 15 May 2008 15:20:00 GMT

PAE (Physical Address Extension)

PAE is the added ability of the IA32 processor to address more than 4GB of physical memory. The following OS's can use PAE to take advantage of physical memory beyond 4GB:

Microsoft Windows 2000 Advanced Server
Microsoft Windows 2000 Datacentre Server
Microsoft Windows Server 2003 Enterprise Edition
Microsoft Windows Server 2003 Datacentre Edition

To enable PAE, use the /PAE switch in your server's boot.ini file.

N.B. In Windows Server 2003, PAE is enabled automatically if your server is using hot-add memory devices.In all other cases, you must add the /PAE switch to boot.ini

Typically, a process running under Windows 2000 or Windows 2003 can address up to 2GB of memory address space (3GB if the /3GB switch is used in boot.ini) with some of the memory being physical memory and some being virtual. The more programs (and therefore, more processes) you run, the more memory you commit up to the full 2GB of address space. When this occurs, the paging process increases dramatically and performance may be negatively affected.

Windows 2000 and 2003 memory managers use PAE to provide more physical memory to a program. This reduces the need to swap the memory of the page file and results in increased performance. The program itself is not awareof the memory size, the memory management and allocation of the PAE memory is handled by the memory manager independently of the programs that run.

When the /3GB switch is used is conjunction with the /PAE switch, the OS does not use any memory in excess of 16GB. This is caused by kernel virtual memory space considerations. If the system restarts with the /3GB switch in boot.ini, and the system has more than 16GB available (physical memory), the additional RAM is not used by the OS. Restarting without the /3GB switch enables use of all the physical memory.

AWE (Address Windowing Extensions)

AWE is a set of API's to the memory manager's functionsthat enables programs to address more memory than is available through standard 32-bit addressing. It enables programs to reserve physical memory as non-paged memory to the program's working set of memory. Enables memory intensive programs, such as RDBMS, to reserve large amounts of physical memory for data without having to be paged in and out of a paging for for usage. Data is swapped in and out of of the working set and reserved memory is in excess of the 4GB range. Additionally, the range of memory in excess of 4GB is exposed to the memory manager and the AWE functions by PAE. Without PAE, AWE cannot reserve memory in excess of 4GB.

In Summary:

PAE is a function of the Windows 2000 and Windows Server 2003 memory manager that provides more physical memory to a program that requests memory
The program is not aware that any of the memory that it uses resides in the range greater than 4GB
AWE is an API that enables programs to reserve large chunks of memory
Reserved memory is non-pagable and is only accessible to that program
You can't allocate more than 4GB of memory to an application on Windows Server 2000 because PAE is not available. Also cannot use the /3Gb switch in the boot.ini file with Windows Server 2000
Need to use Windows 2000 Advanced Server or Windows 2000 Datacentre.
PAE (36-bit physical addressing mode) allows up to 8GB of physical memory on Windows 2000 Advanced Server and 32GB on Windows 2000 Datacentre
Maximum amount of memory supported on Windows Server 2003 is 4GB. Windows Server 2003 Enterprise Edition SP2 supports 64GB of physical RAM and Windows 2003 Datacentre SP2 supports 128GB. You can use the /3GB switch in boot.ini in all versions of Windows Server 2003

Memory Limits for Windows Releases
Operating Systems and PAE support

Windows Failover Clustering Overview

robcarrol — Wed, 07 May 2008 19:07:00 GMT

The host node in the failover cluster performs a "looks alive" check every 5 seconds. An IsAlive check is performed every 60 seconds using SELECT @@SERVERNAME. If this fails the IsAlive retries 5 times and then attempts to reconnect to the instance of SQL. If all fail, then the SQL Server resource fails. Depending on the failover threshold, configuration of SQL resource, Windows Clustering will either attempt to restart on same node or failover to another available node.

During failover, Windows Clustering starts the SQL Server service for that instance on the new node, and goes through the recovery process to start the databases. After the service is started and the master database is online, the SQL Server resource is considered to be up. User databases will then go through the normal recovery process: any completed transactions in the t-log are rolled forward, and any incomplete transactions are rolled back. The length of the recovery process is dependent on how much activity must be rolled forward or rolled back upon startup.

Set the recovery interval of the server to a low number to avoid long recovery times and to speed up the failover process. SQL Server generates automatic checkpoints based on the "recovery interval" setting. Long running transactions can lead to much longer restart times than specified in the recovery interval option.

Failover/Failback Strategies

The cluster group containing SQL Server can be configured for automatic failback to the primary node when it becomes available again. By default, this is set to off.

To Configure:

Right-click the group containing SQL Server in the cluster administrator, select 'properties' then 'failback' tab.
To prevent an auto-failback, select 'Prevent Failback', to allow select 'Allow Failback' then one of the following options:

Immediately: Not recommended as it can disrupt clients
Failback between n and n1 hours: allows a controlled failback to a preferred node (if it's online) during a certain period.

Configure Node Failover Preferences

When you use more than 2 nodes, it's important to consider which nodes should own resources in the event of a failover. For example, in an n+1 configuration, each SQL Server group should have the idle node second in the list of preferred owners. N.B. Do not use cluster admin to remove nodes from the resource definition. USe SQL Server setup for that functionality.

To Configure:

Right-click SQL Server group in the cluster administrator and select properties
On the 'General' tab, the preferred owners list box contains all cluster nodes that can potentially own resources in that group, and the current order in which they will failover
Click 'Modify' to change this order

Configure Thresholds for a Resource

Right-click the cluster resource and then select 'Propereties'
Click 'Advanced'
Select 'Do not restart' if the cluster service should not attempt to restart. Restart is the default
If 'Restart' is selected:

Affect the Group: uncheck to prevent a failure of the selected resource from causing the SQL Server group to failover
Threshold: number of times the cluster service will attempt to restart the resource, and period is the amount of time in seconds between retries

Do not modify the 'LooksAlive' and 'IsAlive' settings
Unless necessary, do not alter the 'Pending Timeout'. This is the amount of time the resource is either in the online or pending or offline pending states before the the cluster service puts it in either offline or failed state

Configure Thresholds for a Group

Right-click the group containing the SQL Server virtual server then click properties
Click the failover tab
to configure the failover policy, in the threshold box enter the number of times the group is configured to failover within a set span of hours. In the period box, entrer the set span of hours
Once the resource group reaches the set number of failovers, it will stay offline. However, other cluster resources, such as cluster IP, could be left online

Cluster Resource Dependencies

Resource	Dependency
SQL IP Address (Virtual Server Name)	NONE
SQL Network Name (Virtual Server Name)	SQL IP Address
SQL Server	Disk Resource(s), SQL Network Name
SQL Server Agent	SQL Server
SQL Server Full Text	Disk Resource(s)
Analysis Services (2005 only)	Disk Resource(s), Network Name

Cluster Heartbeat

Cluster nodes use the "heartbeat" signal to check whether each node is alive at both the OS level and SQL Server level. The node hosting the SQL Server resources uses the Service Control Manager to check every 5 seconds whether the SQL Server service appears to be running. This "LooksAlive" check does not impact performance but does not perform a thorough check; the check will succeed if the service appears to be running even though it might not be operational. As a result, a deeper check must be performed; this "IsAlive" check runs every 60 seconds.

IsAlive:

Runs every 60 seconds
Runs an @@SERVERNAME T-SQL query against SQL Server to determine whether the server can respond to requests
Does not gaurantee that all user databases are available or are performing within necessary performance/response-time requirements

If IsAlive Check fails:

Retried 5 times and then it attempts to reconnect to the instance of SQL Server
If all 5 retries fail, the server resource fails
Depending on the failover threshold config, the failover cluster will either restart the resource on the same node or it will failover to another available node

The IsAlive query tolerates a few errors, but ultimately it fails if it's threshold is exceeded

During failover of the SQL Server instance, SQL Server resources start up on the new node and SQL Server goes through the recovery process to restart the databases. After the service is started and the master database is alive, the SQL Server resource is considered to be up. User databases will go through the normal recovery process. Completed transactions in the transaction log are rolled forward (the Redo phase), incomplete transactions are rolled back (the Undo phase).

In SQL Server 2005 Enterprise Edition, each user database is available to the user once the Redo phase is complete. For all other editions (and all 2000 editions), each user database is unavailable until the Undo phase completes. Length of recovery process depends on how much activity needs to be rolled forward or back upon startup.

The 'recovery interval' sp_configure option of the server can be set to a low number to avoid longer Redo recovery times and to speed up the failover process. Undo recovery time can be reduced by using shorter transactions so that uncommitted transactions do not have much to roll back.

Recommended Heartbeat Configurations

Two or more independent networks must connect the nodes of the cluster to avoid a single point of failure
Use of 2 LAN's is typical (MS PSS does NOT support the config of a cluster with nodes connected by only one network)
At least two of the cluster networks must be configured to support heartbeat communications between the cluster nodes to avoid a single point of failure
To do so, configure the roles of these networks as either "Internal Cluster Communications Only" or "All Communications" for the cluster service
Typically, one of these networks is a PRIVATE INTERCONNECT dedicated to internal cluster communication.
Each cluster network must fail independently of all other cluster networks.

The cluster networks must not have a component in common that can cause both to fail simultaneously.
The use of a multiport network adapter, for example to attach a node to two cluster networks would not satisfy this requirement in most cases as the ports are not independent

Remove all unnecesary network traffic from the network adapter that is set to INTERNAL CLUSTER COMMUNICATIONS ONLY (also known as the "heartbeat" or "private" network adapter, to eliminate possible communication issues
Clustering communicates using Remote Procedure Calls (RPC) on IP sockets with User Datagram Protocol (UDP) packets