KaitlinG

Clustered File Share Migration–Windows Server 2003 to Windows Server 2008 R2

2011-03-09T21:09:39Z

This is an interesting issue that I have seen recently. When migrating cluster resources from Windows Server 2003 to Windows Server 2008 R2, some of the resources may not migrate successfully. It works from 2003 to 2008, or 2008 to 2008 R2, but not from 2003 to 2008 R2. There is an article that describes this issue and the available workarounds noted below.

The workarounds unfortunately all have negative impacts depending on the size and criticality of the implementation we are dealing with. The 3rd method states to ‘manually recreate’ the file shares. This may not be a viable option if you have a very large number of shares and lots of varying permissions configured. The 2nd method recommends to migrate the shares using the File Server Migration Toolkit. Unfortunately, this may not work for people who are planning to utilize the existing storage. Method 1, which is also the ‘recommended’ method, is then going to be the most ideal method for most people.

The thing that is not mentioned in the 'recommended' workaround however, is that the resources must be brought online on the 'interim' 2008 cluster server before they can be successfully migrated to the R2 system. This could be inconvenient for anyone that does not want to take the time to actually copy the files over or mount the storage on the 2008 server simply to make this migration work. This was the issue that I ran into recently. We have numerous file share clusters configured that are running hundreds of individual file share resources in each cluster group. This would be a huge undertaking to have to copy the shares or manually recreate them. And, since the existing storage would be used and there are loads of permissions setup, the only method that is really applicable for us was method 1.

So, I devised a plan to make this go as smoothly and painless as possible.

This can be done simply by using a Virtual machine for the 'interim' 2008 cluster server node which is what we did.

Install Windows Server 2008 on your temporary server.
Configure the Failover Cluster Feature and create a new temporary single node cluster.
Create or attach one small volume or more if you wish to migrate multiple resources at one time (in my test, I simply used a single 50MB iSCSI disk for my migration.)
Assign this disk the same drive letter as the source Windows Server 2003 cluster share drive letter on the Windows 2008 cluster.
Prepare the cluster shared disk for the file shares.

Map a drive to the root of the shared cluster disk on the Windows 2003 cluster that is hosting the file shares.
Run the below Robocopy command from the Windows Server 2008 system.
robocopy <Source Drive Letter>:\ <Destination Drive Letter>:\ /e /z /xjf /xjd /tee /xf * /lev:2 /loc:v:\RobocopyLog.txt
NOTICE: This command will only copy the directories and not the actual files.
Unless you want to copy the entire directory structure, be sure to include the /lev: switch – this allows you to set how many levels deep that you want the copy to go. We configured this because we have a very deep folder structure and only needed the first couple of levels to make sure we got all of the shared folders.
After running the command, verify that all of the proper folders were successfully copied

Run the Cluster Migration Wizard from the 2008 cluster. Follow through the wizard to migrate your file shares from the 2003 cluster to the 2008.

After the migration, you will need to delete the disk resource that was migrated and add the disk that you copied the folders to in the previous steps to the File Server group. Replace the dependency for this disk in the File Server properties.
Here is where your downtime will begin - Offline the File Share group on the Windows Server 2003 Cluster.
Online the File Server group on the Windows Server 2008 Cluster.
- If Kerberos was being used, it may take two onlines for the Network name to successfully come online. After it fails, online it once again and it should come online without any issue.
After verifying that all of the shares and network name came online successfully, switch to the Windows Server 2008 R2 cluster.
Run the Cluster Migration Wizard from the Windows Server 2008 R2 cluster.
Step through the wizard specifying the 2008 cluster as the source server and migrate the File Server group.
After the migration is complete, you may now offline the File Server group on the Windows Server 2008 cluster.
On the R2 cluster, Add the disk resource that contains the actual folders/files for the shares to the File Server group in the Failover Cluster Manager. Again, you may need to delete the old disk resource that was migrated.
Set the disk resource as a dependency for the File Server.
Again, bring everything online and verify that the file shares have come online successfully.
Repeat this process for any other File Share groups that you wish to migrate.

For more information on this issue, see the article:

983840 Not all resources are migrated from a Windows Server 2003-based failover cluster to a Windows Server 2008 R2-based failover cluster
http://support.microsoft.com/default.aspx?scid=kb;EN-US;983840

Deciphering Performance Data–CSV Volumes

2011-02-11T07:10:00Z

I always feel so redundant when saying “CSV Volume”. It’s like saying “Cluster Shared Volume Volume”, but ah well, here we go.

Gathering performance data when it comes to CSV volumes can be confusing at first glance. The big issue that we have to understand is that multiple nodes and multiple VMs are hitting the same CSV disks from every possible direction. If looking at only a single node in your cluster, you will only get one small piece of the puzzle.

Today’s entry will begin the discussion very basic as we try to help explain what is going on and what you can do to see the kind of activity that’s happening on your CSV... umm Volumes. :)

The Tools

For this demo, I am going to be sticking with Performance Monitor to keep things very simple.

In Perfmon, we will be looking primarily at the Physical Disk Object, Disk Transfers/sec which will allow us to simply see the total of Disk Reads and Writes that are going on. Break down to Disk Reads/sec and Disk Write/sec for specifics, but as said, keeping this one simple.

My setup is as follows:

2 Hyper-V Cluster Nodes running Windows Server 2008 R2 Service Pack 1
1 CSV Volume (on iSCSI for simplicity’s sake)
4 Network connections, iSCSI, Client/Management, Cluster/CSV, Live Migration (no matter how many NICs you have on these HAVM clusters, it’s never enough!)
2 VMs each running Windows Server 2008 R2 ‘doing stuff’
1 stand-alone Windows Server 2008 R2 hosting my iSCSI targets

The Test

First off, thank you to Cristian Edwards for opening my eyes to the scenario we are about to see!

In the first test, we are going to load up the Disk Transfer/sec counter in Perfmon on each cluster node and on the storage server. We will then move one of my VMs to each of the two cluster nodes. When ready, the VMs one at-a-time will start doing work. We will then check to see what the disk counter is telling us on the cluster nodes and compare it to what we see on the storage server machine. Logic would tell us that adding up the I/O of the two cluster nodes should be close to equal to what we see on the storage server.

In the first two images, you can see where I kicked off some work inside of VM1, and near the same time, I kick off work inside of VM2. VM1 is hitting around 867 IOPs with VM2 sitting around 715.

Moving down to our storage server, in this case my Microsoft iSCSI software target, we see that the disk is getting 1505 IOPs per second. Pretty much the sum total of these two VMs hitting the disk at the same time.

Suffice it to say, if you are looking at only one node in your Hyper-V cluster, you are only going to see what is happening on that single node and miss whatever any other cluster node is doing to the storage. If you have storage performance tools, that is likely the best way to see the actual performance of the underlying storage.

Now lets try something different. I am going to enable ‘redirected I/O’ on the CSV and see what happens when we restart the test. The CSV volume is currently hosted by cluster node 1 which means that all of the IO from node 2 should be getting routed through node 1 during this test.

This is very interesting! I started doing work on VM1, and as expected, we see the I/O shooting up. I then started doing the same work as before in VM2, but we do not see any I/O happening on the second node of the cluster. Interesting though, it appears that the I/O registering on the first cluster node is showing an increase from the first test. It is now registering I/O from both cluster nodes where cluster node 2 does not show anything at all.

On the storage server, we are able to confirm the above findings as the I/O is matching cluster node 1.

As can be seen, collecting data for CSV volumes is a puzzle. Without having all of the pieces it can be confusing to tell what you are looking at. In this very controlled environment, it is easy to see. When we are dealing with many more nodes hitting a SAN that is also being used by any number of other servers in a large enterprise environment, and then slap on top the fact that we have VMs spread out everywhere doing any range of work on the CSV volumes, this can become more complicated. Hopefully, in this blog and future ones, I will be able to help in keeping track of your I/O!

Keep in mind that there are CSV specific counters that can also be used to view ‘redirected’ I/O, but will have to save that for next time.

More tk…

HAVM Cluster–Configuration Recommendations on Windows Server 2008 R2

2011-02-10T05:34:00Z

Whether connecting to virtual machines or physical ones, the thing that matters most to us is that the servers are available and responsive. To keep VMs available, Failover Clustering enters the stage providing a number of features to add redundancy and recoverability into Hyper-V VMs.

Live Migration – Allows us to have zero down-time with our VMs to conduct hardware and software maintenance to the Hyper-V Hosts (aka Parent Partition)
CSV – The feature that allows every cluster disk to be online all the time on every cluster node. Also provides us with additional redundancy to connect to the storage.
Health Monitoring – Cluster can monitor the health of the VMs and take immediate action to bring them back to life if they are not responding.

All of these features are working together to make your VMs Highly Available. To maintain this HA, we must ensure that best practices are in place. When done properly, the net result is keeping those VMs online, responsive and available more often.

Validate

First off, many of you have heard of Validate or perhaps been using it for years. Windows Server 2008 marked the beginning of a new trend of “best practice” type tools being built right into the OS to check for proper configuration of various components. The Cluster Validation Wizard is a fully functional tool that will test the h/w, drivers and other components that are required to support a Failover Cluster system. It is designed to be run prior to or after deployment and requires that all h/w, storage connections, networking and Failover Cluster Feature are in place before running.

Cluster Validate is among one of the most useful tools that have been put into the hands of our customers that truly incorporates the kind of testing that our h/w vendors have been doing for over a decade to get their Failover Cluster solutions listed on the Microsoft HCL. Once Validate is complete, you are greeted with a detailed report of the testing that was done on your cluster system. When designing your new Hyper-V Failover Cluster, this will be among the first in the line of action items.

Green check marks means that everything is configured per Microsoft best practices and/or is working properly.
Yellow warnings indicate that there is something wrong that you should consider addressing, but whatever it is will not prevent you from building your cluster. An example of this could be if you only had a single network connection for your cluster nodes to communicate. You can technically build a cluster with only one network, but to be fully supported, Microsoft requires a second network. Another example could be if you ran Validate post-production and your Quorum configuration was not set properly for the number of nodes present in the cluster.
Red X’s are bad. If you have any red ‘X’ in your report, cluster will not function on your setup and you must address the problem before you can continue. An example of this could be if your storage did not support the SCSI-3 PRs or other hardware features that we require.

Validate should be run after creating your Hyper-V Virtual Networks. It will identify many common configuration issues and most importantly, any problems with storage.

More info on Validate: http://technet.microsoft.com/en-us/library/cc732035(WS.10).aspx

Live Migration

Live Migration, as mentioned, is the feature that allows us to move a VM between cluster nodes with ZERO downtime. We want Live Migration to just work, right? Plug the nodes in, create a few VMs, and voila, we can Live Migrate our VMs around with no problem. For this to succeed however, there are some specific configurations that must be in place. To begin with, the processing that the VM was doing on the original host must be able to continue to run on the new host. If the processors are not the same manufacturer or even at a compatible level, then Live Migration will fail. If the processors are not compatible, we can enable ‘processor compatibility’ on the VMs that will essentially bring the physical procs down to the lowest functional denominator. However, we do not support migrating to nodes with different processor manufacturers.

If the ‘virtual switches’ are not configured identically, we may fail. Ensure that every setting even down to the naming of the actual networks are identical from node to node.
If any changes have been made to a VM through Hyper-V Manager, the Virtual Machine Configuration may be inconsistent and require a ‘refresh’ in Failover Cluster Manager.
Be sure to specify the network that you want the VMs to use for Live Migration. Also, set a secondary network if you have one as a backup in case the preferred network fails.
IPSec is fine to use on all of the cluster networks, however, for performance reasons, we recommend keeping IPSec off of the Live Migration network. Unless required for security, keep it off the LM network.

Cluster Shared Volume

The CSV network will be the one that has the highest priority in the cluster. You will determine this by looking at the metric that is set in the network properties. The lowest metric value indicates the ‘highest’ priority. If the network that you intend to use for CSV traffic is not already set to the highest priority, you will want to run a PowerShell command to modify the metric value.

To determine the current metric for the cluster networks:
- Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role
To modify the metric value of one of the cluster networks:
- ( Get-ClusterNetwork “Cluster Network” ).Metric=900

Additional Details at: http://technet.microsoft.com/en-us/library/ff182335(WS.10).aspx

NTLM must be enabled on all nodes
Ensure identical HBA/Storage/Drivers/Firmware across all nodes
CSV feature utilizes SMB and thus Client for Microsoft Networks and File and Printer Sharing must be enabled
- This is a little different for those of you that are used to disabling these on the Heartbeat networks in the past.
Be aware of available space on the CSV volumes, VMs will stop or be unable to start if there is no free space.
- Alternatively, use fixed disks
- http://blogs.msdn.com/b/clustering/archive/2010/06/19/10027366.aspx

Networking

Networking is absolutely critical in a Hyper-V cluster system. Depending on how many network adapters you have in your nodes, you have some options as to how to configure them. Although it can be done with fewer, the most ideal situation is to have 4 total networks.

Virtual Machine Access
- The first network will be used for your Virtual Machines to actually get out on your client network.
Management
- This network will be set aside for your management – Hyper-V Management, SCVMM, Failover Cluster, etc.
Cluster and CSV
- Inter-node communications/heartbeating and CSV traffic will happen via the 3rd network. This network will be used to access the CSV volumes in re-directed access mode or when we are doing other maintenance to CSV volumes including backup operations. As mentioned, this will be your highest priority network.
Live Migration
- This network will be used to transfer the memory pages from VM states during the Live Migration process. If possible, always provide a dedicated network for use with Live Migration.
iSCSI
- A fifth network would be in place if we were also using iSCSI.

If you do not have a network to dedicate to Live Migration, you will need to configure a QoS policy to limit the bandwidth used for Live Migration purposes.

More details on networking can be found at: http://technet.microsoft.com/en-us/library/ff428137(WS.10).aspx

Load Balancing

One of the biggest issues that I see people run into when it comes to Failover Clustering is properly distributing the load of VMs across the cluster nodes. Special consideration needs to take place to ensure that each node can handle the pressure of VMs that may be hosted at one time. With the release of SP1 and the new Dynamic Memory feature, administration will be made simpler, but performance baselining should be completed to ensure the cluster nodes are not over-extended. SCVMM may be used to assist with proper load balancing of the VMs in conjunction with SCOM.

In my next blog, we will look at the issues of, and how to overcome, collecting performance data from CSV volumes. Thank you! –Kaitlin

Introduction!

2011-02-09T22:30:00Z

Hello there,

I am Kaitlin Goddard and just like most women, I love candle-lit dinners, long strolls on beaches, and really beefy Windows servers! I work for Microsoft as a PFE (Premier Field Engineer) and have been blowing things up real good since 1999. My work for the past >10 years has revolved around Failover Clustering and the services and applications that we make 'highly available.'

This is my initial entry into my new blog here on Technet and my first blogging experience ever, so go easy on me :). My objective for this blog is to provide a useful source of information for Windows Server pertaining to Failover Clustering and Hyper-V. I'm sure this will often spill over into other related products/components as well.

Thank you for visiting,

~Kaitlin