Ask the Core Team

Cluster Migration Wizard, is this the only way to migrate shares on a Cluster?

As everyone should know by now, the way to get your Windows 2003 Cluster Server to Windows 2008 and 2008 R2 Failover Clustering is by doing a migration to the new cluster.  So let’s say we have a Windows 2003 Cluster running as a File Server and we are going to be using new hardware (machines, SAN, etc).  In order to migrate the shares, you can use the Cluster Migration Wizard in Failover Cluster Management.  

The Cluster Migration Wizard does not copy data, so in the case of moving to a different SAN for your new Cluster, alternate means of moving the data would also need to be done.  Another difference you may notice is that in the Windows 2003 Cluster Server, you may have either a single or multiple file share resources.  Once it has been migrated over to the Windows 2008 and 2008R2 Clusters, the end result will be one File Server resource.

The purpose of this blog is to let you know of another option to migrate the shares.  With the File Server Migration TooIkit, you can migrate the shares over that also would include the data, NTFS permissions, and share permissions.  I am not going to go over any pros, cons or comparisons between the two, just another alternative.  You should test out each for your particular scenarios fully and form your own opinion on which to use.

Here is my File Share Group that currently resides on the Windows 2003 Cluster.

Disk X:              (Physical Disk)

IP Address           (IP Address)

Network-Name         (Network Name)

Home-Shares          (File Share)(Share Subdirectories)

Acct-Share           (File Share)(No Share Subdirectories)

Mgmt-Share           (File Share)(No Share Subdirectories)

Below are the shares in this group that I have and wish to migrate over.  Both the Acct-Share and the Mgmt-Share do not have the option of Share Subdirectories selected while the Home-Shares do.  I have modified the share permissions from the default recommended of Everyone Full Control.  I did go into the NTFS permissions and set them accordingly to Domain Admins FULL and the specific users of the folders to FULL.  In my case, the Cluster Service account is a part of the Domain Admins Group, so I did not specifically list it.  So the directory structure and share permissions in Cluster Administrator would look like this.

Home-Shares    

x:\home             <<-- DomainAdmins FULL, DomainUsers FULL

       x:\home\a-d  <<-- DomainAdmins FULL, DomainUsers FULL

       x:\home\e-j  <<-- DomainAdmins FULL, DomainUsers FULL

       x:\home\k-p  <<-- DomainAdmins FULL, DomainUsers FULL

       x:\home\q-x  <<-- DomainAdmins FULL, DomainUsers FULL

       x:\home\y-z  <<-- DomainAdmins FULL, DomainUsers FULL

Acct-Share

       X:\acct             <<-- DomainAdmins FULL, AcctUsers FULL

Mgmt-Share

       X:\mgmt             <<-- DomainAdmins FULL, MgmtUsers FULL

One thing to note before beginning this process,  In Windows 2003 Cluster, the Cluster Service account needed to have at least READ permissions from a share and NTFS level.  In Windows 2008 and 2008R2 Clusters, there is no longer a Cluster Service account.  Because we more rely on the local SYSTEM account, this account must be on the Share and NTFS level for Windows 2008 and 2008R2 in order to properly do its checks.  When using the Cluster Migration Wizard, we will add it automatically for the share permissions.  For the File Share Migration Toolkit, you need to add those permissions before accomplishing the migration.  Otherwise, once moved over to the Windows 2008 or 2008R2 Cluster, the share will not be created.

On the 2008 or 2008R2 Cluster, you would first want to create your File Server Group within Failover Cluster Management with just the IP Address, Network Name, and Disk you will be using.  Do not yet create any shares.  One thing you must keep in mind is that the group on both the 2003 and the 2008 or 2008R2 Cluster must be online for the file share migration.  So when creating the group on the 2008 or 2008R2 Cluster, you would need to use a different IP Address and Network Name (or Client Access Point).  Otherwise, you will receive a duplicate IP Address or duplicate name error and the resource will fail to come online.  You can create it with a different Client Access Point now, and change it after everything has been migrated over with the group on the 2003 Cluster File Share Group offline.

Now you want to run the File Server Migration Wizard to get this all started.   At the start, choose a new project and run through the wizard.

At one point in the wizard, you need to uncheck the DFS options if the source and target are not going to participate as DFS Shares.  For the Default Location, select the drive that you will send everything to and it will be referenced from the proper group.  If you are running FSMT from a 2008 Cluster node that does not own the File Server Group you created, the drive will not be available to choose.  Move the group over to this node so that it can locally get to the drive, if need be.

The next screen below you will want to select the Source Server which will be the Network Name in the File Server Group on the Windows 2003 Cluster.   Keep in mind and remember how Windows 2003 Cluster is in regards to shares.  It will display all shares on the machine, regardless of if it is a part of the same group or locally to the machine.  So ensure that you select only the shares you want to migrate.  You also want to make sure you select the “Copy security settings” as this is what will get you the NTFS and Share permissions.

When you choose to continue and view the report, it will give you a warning about this being a Cluster, but this can be safely ignored as long as you have everything properly configured as mentioned previously.  If you look at the View Report button to the right, you would see this:

Once you continue and get to the finalization, please note the warning.  Therefore, ensure you are doing this during downtime.

If you are running FSMT from a node that does not own the File Server Group (example, there was a failover or a moving of the group, the Finalization Process will fail.  The other thing to mention is that if you already have a share by the same name, you get an error. 

If you look at the Report, you would see this.

Here, you can see that it cannot find the path specified, object already exists, etc.  Again, the “warning” about the name being a Cluster can be safely ignored.  Another note here is to notice the path it is talking about.  On the Windows 2003 Cluster, the paths were:

Home-Shares    

x:\home              

       x:\home\a-d    

       x:\home\e-j    

       x:\home\k-p    

       x:\home\q-x    

       x:\home\y-z    

Acct-Share

       X:\acct              

Mgmt-Share

       X:\mgmt              

With FSMT, it copies the data (folders and files) and will put them in a folder under the network name from the previous machine.  So your folder structure would actually be this when completed.

Home-Shares    

x:\network-name\home              

       x:\network-name\home\a-d    

       x:\network-name\home\e-j    

       x:\network-name\home\k-p    

       x:\network-name\home\q-x    

       x:\network-name\home\y-z    

Acct-Share

       X:\network-name\acct              

Mgmt-Share

       X:\network-name\mgmt

Once you fix the errors from above and continue, it will copy all of the folders, the data, and the permissions.  Part of this process is to create a share which, in turn, creates the File Server resource in the group you specified.  So now in Failover Cluster Management, you see this.

Notice also that the original network name from the 2003 Cluster is tagged at the end of the share name.  So the new folder structure and the new share name needs to be taken into consideration.  This is something that the FSMT does and is not a way of changing it for the migration.  It can only be done afterwards.  So you would either need to change the share name itself in Failover Cluster Management; or, if you have scripts run from the domain (or other locations), you will need to modify those scripts.  You also will want to go ahead and take the group on the 2003 Cluster offline and change the Client Access Point (IP Address and Network Name) in the 2008 or 2008R2 Cluster to what you want it if they were to be the same.

As with all migrations, please test this process out before and after so that you can work out any kinks or issues found in your environment if this is the method you will be using.  Also keep in mind that 2008 and 2008R2 Clusters “scope” the shares to the Network Name as explained here.  Once everything appears as solid, then roll it out in your production environments.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Be Sure to Plan Carefully When Virtualizing Your Infrastructure

There is a lot of excitement around Microsoft virtualization technologies these days and rightfully so.  One of the ‘hottest’ areas right now appears to be making virtual machines highly available using Windows Server 2008 R2 Failover Clusters so end users can take maximum advantage of Live Migration and Cluster Shared Volumes (CSV).  This configuration not only saves a lot of money but also provides business continuity in the event of an unforeseen failure in  the environment.

While I could spend time extolling the virtues of our virtualization technologies, I am really here to discuss what can happen if one were to get too ‘overzealous’ and not use common sense and a sound plan for implementing the solution correctly.  As with many of the blogs you read here on the CORE blog site, they have been written because of experiences we have had with our customers.  This one is no different.

So, what happens when a customer decides they love Microsoft virtualization and high availability technologies so much, they want to virtualize their entire infrastructure?  And, suppose they want to be sure it’s highly available so they create a multi-node Failover Cluster to host the virtual machines.  When the customer completes the project, they are so very proud of what they have done because now they can retire their old hardware and save tons of money on power and cooling costs in their datacenter.  Everyone is happy and celebrations abound. And, then it happens…..someone decides that they need to shutdown the cluster(s), for whatever reason, it does not matter, and, after awhile, when they decide it is OK to bring the cluster(s) back online…they cannot.  Oh, and one more thing…..the clusters are running on Windows Server 2008 R2 CORE.  Trust me…this is a true story and has already happened more than once, hence the impetus behind this blog.

If the predicament is not immediately obvious, and it should be for cluster veterans, I will tell you that the cluster service will fail to start because it cannot contact a Domain Controller somewhere in Active Directory.  And, this is because all of the Domain Controllers and DNS servers (critical infrastructure servers) have been virtualized and are, in fact,  virtual machines currently supported by the cluster that is trying to start up.  Clearly, this is a case of having ones eggs all in one basket – not good.

How did we fix this?  It was not a quick fix.  In a nutshell, what the Support Engineer did was have the customer determine which storage LUN was hosting the VM files for one of the virtualized Domain Controller\DNS servers.  Then, the LUN was mapped to a standalone server so the VHD file could be copied off to another standalone Hyper-V server so a new VM could be created and placed in service.  Once this was accomplished, the cluster could be started.

How can this type of scenario be avoided? 

1.        Develop a solid, well thought out migration plan.  Ensure  the planning team includes people who understand how all the technologies function in a virtualized environment.

Note:  Please review KB 888794: Considerations when hosting Active Directory domain controller in a virtual hosting environments

2.       Have at least one physical Domain Controller\DNS server available in the environment.

3.       If #2 is not an option, distribute the virtualized infrastructure servers across multiple hyper-v clusters and hope they will not all be Offline at the same time.

4.       Plan to have one of more Hyper-V servers running in a WORKGROUP configuration.  Hyper-V servers do not have to be joined to a Active Directory domain.  Then distribute some of the virtualized infrastructure servers across these servers.

As always, we hope this has been informative for you.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Deleting a Cluster resource? Do it the supported way

The purpose of this posting is to explain the supportability of removing resources on a Cluster Server.  We have seen an increase lately with users manually deleting resources from the Cluster registry and I wanted to say that this is unsupported by Microsoft.  Doing this can cause issues with your Clusters and I wanted to bring up the issues as well as how to get out of the predicament that you can get in.

First, the ONLY supported ways of deleting a resource is either through Cluster Administrator (Windows 2003), Failover Cluster Management (2008 and 2008 R2), CLUSTER.EXE, and Powershell (2008 R2).  Reasons that have been given for manually deleting the resource from the registry is that they cannot get into the UI.  This is where CLUSTER.EXE or Powershell comes in.  For example, say I have a resource called Johns Resource and I want to delete it.  The command I would be using to do this would be:

CLUSTER.EXE

Cluster res “Johns Resource” /delete

Powershell

Remove-ClusterResource “Johns Resource”

Using the command will remove the resource from all entries on all nodes, including the quorum. 

To break this down further, a resource in the Cluster will be in several locations in the Cluster Hive and it is referenced by a guid.

HKEY_LOCAL_MACHINE

Cluster

Resources

C8d32427-7daa-4a94-ba85-850f5a920382       <<-- Johns Resource

HKEY_LOCAL_MACHINE

Cluster

Groups

28baec47-2589-49a9-aa7c-cc32b57e1875     <<-- the group name

Contains             <<-- all resources in the group here

What users have been doing is simply deleting the guid under the Resources key only.  However, the resource is still listed under the group.   When they do this, they also manually delete it on all nodes as well as the quorum drive.  Sometimes, it takes a restart of the Cluster Service everywhere before it finally is no longer there.  CLUSTER.EXE would have done it right then and there and no restarts necessary.

In Windows 2003 Cluster, when you start the Cluster Service, we see this in the Cluster Log:

[FM] Group 28baec47-2589-49a9-aa7c-cc32b57e1875 contains Resource C8d32427-7daa-4a94-ba85-850f5a920382.
[FM] Creating resource C8d32427-7daa-4a94-ba85-850f5a920382
[FM] Initializing resource C8d32427-7daa-4a94-ba85-850f5a920382 from the registry.
[FM] Unable to open resource key C8d32427-7daa-4a94-ba85-850f5a920382, 2
[FM] DestroyResource: destroying C8d32427-7daa-4a94-ba85-850f5a920382
[DM] Deleting object C8d32427-7daa-4a94-ba85-850f5a920382
[FM] Failed to find resource C8d32427-7daa-4a94-ba85-850f5a920382 for group 28baec47-2589-49a9-aa7c-cc32b57e1875

When you go to open Cluster Administrator, there are no initial errors.  However, if you have multiple resources that are like this in the same group, you could receive an Error 1130 (Not enough Server Storage) and you are unable to create any more resources in the group.

In Windows Server 2008 (and R2) Clusters, the results are much different.  The Cluster Service will show as started; however, the cluster will not form.  In the System Event Log, you will see these errors:

Event ID: 7024
Source: Service Control Manager
Description: The Cluster Service terminated with service-specific error 2 (0x2).

Event ID: 1092
Source: FailoverClustering
Description: Failed to form Cluster ‘clustername’ with error code 2. Failover cluster will not be available.

In the Windows 2008 Cluster Log, you will see this:

WARN [DM] Key \Registry\Machine\Cluster does not appear to be loaded (status STATUS_OBJECT_NAME_NOT_FOUND(c0000034)
INFO  [DM] Loading Hive, Key Cluster, FilePath C:\Windows\Cluster\CLUSDB
ERR   [CORE] Node 1: exception caught ERROR_FILE_NOT_FOUND(2)' because of 'OpenSubKey failed.'
ERR   Exception in the InstallState is fatal (status = 2)
ERR   FatalError is Calling Exit Process.

These are the things that you can run into by manually removing or “hacking” a resource out of the registry and not remove it from all the locations in the hive.  This is also one of the reasons why this is an unsupported method for removing a resource in a Cluster.  The whole reasoning for Failover Clusters is high availability. By attempting the unsupported methods above, you can cause downtime which gets away from high availability.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

NTFS MetaFiles

They sit there, hiding in the root directory…metafiles.  The shell hides them from the user, but they are still there…lurking.  Microsoft does a pretty good job hiding these files so you don’t accidentally damage them.  But what are these files and how does NTFS use them?

Before we have a look at them I’d like to issue a warning…

WARNING!!!  Do NOT try to alter or delete these files.  Doing so can and will cause permanent damage to your file system.  And more than likely CHKDSK won’t be able to save you.  You will lose all your data if you ignore this warning. 

Hopefully you are now sufficiently scared.

If you haven’t already done so I recommend you read my blog entitled “The Four Stages for File Growth” to give you a better idea of how files are stored on your hard drive.  It isn’t required for understanding this blog but it would help.

Now let’s have a look at these elusive files.

File 0 - $MFT:  Not to be confused with the actual MFT (Master File Table), the $MFT tells us where all the pieces of MFT are.  The MFT is part of the $MFT file.  And the $MFT file is contained within the MFT.  It’s this whole ‘chicken and the egg’ thing.

$MFT – A file in the Master File Table (MFT)

MFT – The table that contains all file records

What makes it confusing is that the entire MFT is in the $MFT file and the file record for $MFT is found in the MFT.  They are separate structures but each one contains the other.

File 1 - $MFTMirr:  This file tells us the location of a backup of the first few files in the MFT.  In data recovery situations, where the beginning of the MFT is damaged, this mirror can help save the day.  I’ve used it a number of times myself.

File 2 - $LogFile:  This is simply a journal of the NTFS’s metadata transaction.  Like most metafiles, it is not human readable and not meant for use by the user.  Corruption of this file can cause you not to be able to mount the file system.  This can be easily fixed by simply resizing the file.  These two commands can assist with that….

Chkdsk <drive:> /L

(to find out the current size for $LogFile)

Chkdsk <drive:> /f /L:<new size>

(to resize$LogFile)

File 3 - $Volume:  This file keeps record of the NTFS version, volume information, and the volume label.  So if you name your volume ‘DAVE’, this is the file that stores that information

File 4 - $AttrDef:  The $AttrDef file defines the different attributes that the file system can have.  Here is a list of the attribute types available:

NOTE:  Do not confuse file attributes like $DATA and $FILE_NAME with attributes like READ-ONLY, SYSTEM, or HIDDEN (which are actually just flags).

File 5 – (.):  The dot (.) is the root directory for the volume.  So when you do a ‘dir’ of c:\, you are looking at the dot (.). 

File 6 - $Bitmap:  This file keeps track of all the clusters of the volume and whether or not each cluster is currently in use.  That’s how we can quickly determine how much free space you have.  We just ask $Bitmap.

File 7 - $Boot:  Contains boot sector and the boot strap (the first 16 sectors of the volume).  The boot sector contains the location of the $MFT and $MFTMirr.   Otherwise we wouldn’t know where to look for them.

In the image above, all parts of the $BOOT file are shown in RED.  The file starts in the MFT and points back to the beginning of the volume for its $DATA attribute, which contains the boot strap.  It is this boot strap code that tells us what boot loader we are using (NTLDR for Windows XP/Windows 2003 and BOOTMGR for Vista/Windows 2008).  Also the boot sector tells us the location of the MFT.  This is part of how Windows is able to locate files during the early stages of bootup, before the NTFS.SYS driver actually loads.

File 8 - $BadClus:  Keeps a record of the clusters on your volume that contain physically bad sectors.  We mark them bad so we don’t try to use them.  If you ever run CHKDSK with a /r switch, then you are telling CHKDSK to update $BadClus with any new bad sectors that are found.

File 9 - $Secure:  Contains security information.  For obvious reasons, I’m not going to tell you how it works. 

File 10 - $UpCase:  This file contains the casing table.

Trivia – For the young folks that don’t know, the terms upper case and lower case came about with the early printing presses that kept the capitol letters in the upper drawer or case, while the more often used small letters were stored in the closer, lower case.

File 11 - $Extend:  A directory that can house files used for optional extensions. 

That’s about it.  Microsoft reserves space in case we want to add any additional files.  So you won’t start seeing normal files until File 17.

Robert Mitchell
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Demystifying NTFS as much as I’m allowed to

Windows Server 2008 R2 Live Migration – “The devil may be in the networking details.”

Windows Server 2008 R2 has been publicly available now for only a short period of time, but we are already seeing a good adoption rate for the new Live Migration functionality as well as the new Cluster Shared Volumes (CSV) feature. I personally have worked enough issues now where Live Migration is failing that I felt a short blog on what process I have followed to work through these may have some value.

It is important to mention right up front that there is information publicly available on the Microsoft TechNet site that discusses Live Migration and Cluster Shared Volumes. This content also includes some troubleshooting information. I acknowledge that a lot of people do not like to sit in front of a computer monitor and read a lot of text to try and figure out how to resolve an issue. I am one of those people. Having said that, let’s dive in.

It has been my experience thus far that issues that prevent Live Migration from succeeding have to do with proper network configuration. In this blog, I will address the main network related configuration items that need to be reviewed in order to be sure Live Migration has the best chance of succeeding. I begin with an initial set of assumptions which include the R2 Hyper-V Failover Cluster has been properly configured and all validation tests have passed without failure, the highly available VM(s) have been created using cluster shared storage, and the virtual machine(s) are able to start on at least one node in the cluster.

I start off by identifying the virtual machines that will not Live Migrate between nodes in the cluster. While it should not be necessary in Windows Server 2008 R2, I recommend first running a ‘refresh’ process on each virtual machine experiencing an issue with Live Migration. I say it should not be necessary because a lot of work was done by the Product Group to more tightly integrate the Failover Cluster Management interface with Hyper-V. Beginning with R2, virtual machine configuration and management can be done using the Failover Cluster Management interface. Here is a sample of some of the actions that can be executed using the Actions Pane in Failover Cluster Manager.

If virtual machine configuration and management is accomplished using the Failover Cluster Management interface, any configuration changes made to a virtual machine should be automatically synchronized across all nodes in the cluster. To ensure this has happened, I begin by selecting each virtual machine resource individually and executing a Refresh virtual machine configuration process as shown here –

The process generates a report when it completes. The desired result is shown here –

If the process completes with a Warning or Failure, examine the contents of the report and fix the issue(s) that was reported and run the process again until it successfully completes.

If the refresh process completes without Failure, try to Quick Migrate the virtual machine to each node in the cluster to see if it succeeds.

If a Quick Migration completes successfully, that confirms the Hyper-V Virtual Networks are configured correctly on each node and the processors in the Hyper-V servers themselves are compatible. The most common problem with the Hyper-V Virtual Network configuration is that the naming convention used is not the same on every node in the cluster. To determine this, open the Hyper-V Management snap-in, select the Virtual Network Manager in the Actions pane and examine the settings.

The information shown below (as seen in my cluster) must be the same across all the nodes in the cluster (which means each node must be checked). This includes not only spelling but ‘case’ as well (i.e. PUBLIC is not the same as Public) –

It is important to be able to successfully Quick Migrate all virtual machines that cannot be Live Migrated before moving forward in this process. If the virtual machine can Quick Migrate between all nodes in the cluster, we can begin taking a closer look at the networking piece.

Start verifying the network configuration on each node in the cluster by first making sure the network card binding order is correct. In each cluster node, the Network Interface Card (NIC) supporting access to the largest routable network should be listed first. The binding order can be accessed using the Network and Sharing Center, Change adapter settings. In the Menu bar, select Advanced and from the drop down list choose Advanced Settings. An example from one of my cluster nodes is shown here where the NIC (PUBLIC-HYPERV) that has access to the largest routable network is listed first.

Note: You may also want to review all the network connections that are listed and Disable those that are not being used by either the Hyper-V server itself or the virtual machines.

On each NIC in the cluster, ensure Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks is enabled (i.e. checked). This is a requirement for CSV which requires SMB (Server Message Block).

Note: Here is where people get into trouble usually because they are familiar with clusters and have been working with them for a very long time, maybe even as far back at NT 4.0 days. Because of that, they have developed a habit for configuring cluster networking which basically is outlined in KB 258750. This article does not apply to Windows Server 2008.

Note: If CSV is configured, all cluster nodes must reside on the same non-routable network. CSV (specifically for re-directed I/O) is not supported if cluster nodes reside on separate, routed networks.

Next, verify the local security policy and ensure NTLM security is not being restricted by a local or domain level policy. This can be determined by Start > Run > gpedit.msc > Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options. The default settings are shown here –

In the virtual machine resource properties in the Failover Cluster Management snap-in, set the Network for Live Migration ordering such that the highest speed network that is enabled for cluster communications and is not a Public network is listed first. Here is an example from my cluster. I have three networks defined in my cluster –

The Public network is used for client access, management for the cluster, and for cluster communications. It is configure with a Default Gateway and has the highest metric defined in the cluster for a network the cluster is allowed to use for its own internal communications. In this example, since I am also using ISCSI, the ISCSI network has been excluded from cluster use. The corresponding listing on the virtual machine resource in the Network for live migration tab looks like this –

Here, I have unchecked the iSCSI network as I do not want Live Migration traffic being sent over the same network that is supporting the storage connection. The Cluster network is totally dedicated to cluster communications only so I have moved that to the top as I want that to be my primary Live Migration network.

Note: Once the live migration network priorities have been set on one virtual machine, they will apply to all virtual machines in the cluster (i.e. it is a Global setting).

Once all the configuration checks have been verified and changes made on all nodes in the cluster, execute a Live Migration and see if it completes successfully.

Bonus material:

There are configurations that can be put in place that can help live migrations run faster and CSV to perform better. One thing that can be done, is to Disable NetBIOS on the NIC that will be supporting the primary network used by CSV for re-directed I/O. This should be a dedicated network and should not be supporting any other traffic other than internal cluster communications, redirected I/O for CSV and\or live migration traffic.

Additionally, on the same network interface supporting live migration, you can enable larger packet sizes to be transmitted between all the connected nodes in the cluster.

If, after making all the changes discussed here, live migration is still not succeeding, then perhaps it is time to open a case with one of our support engineers.

Thanks again fro you time, and I hope you have found this information useful. Come back again.

Additional resources:

Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2

High Availability Product Team Blog

Hyper-V and Virtualization on Microsoft TechNet

Windows Server 2008 R2 Hyper-V Forum

Windows Server 2008 R2 High Availability Forum

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Windows 7 Useful Links

Common questions we get asked in the Setup/Core team is where can I find some useful links regarding Deployment (with MDT 2010 and SCCM), USMT 4.0 (User State Migration Toolkit), Activation (Changes and functionality differences in Vista and Greater O/S’es), Application Compatibility with Windows 7 and/or Vista, and last but not least, what is an easy way to configure Boot to VHD for testing purposes?

So I have compiled a list of useful links and resources that are Microsoft related to answer and help with some of these questions and concerns:

Please Note: That from time to time, these links can change but can be a good starting point for your Windows 7 Deployment planning in your environment.

General Links

77 Windows 7 Tips – This is a good spot to learn what is new in Windows 7, and I want to mention one of the least talked about but one of the best features (Problem Steps Recorder) or PSR for short. This will allow you to reproduce what you are seeing and record screen shots, and even make comments when trying to show other users, or even technical support problems that you are having.

New Shortcut Keys in Windows 7 - Ever wonder how some people can work just as fast with a keyboard without even using a mouse? Well here are some that we added to the existing shortcut keys in Vista, to speed up your day to day work.

Existing Shortcut Keys from Vista – Just in case you didn’t have them all memorized, here is the Shortcut keys that we had from Windows Vista.

Deployment (Using MDT and also good Deployment Blog Sites to watch – Even set them up as RSS Feeds)

Windows 7 Deployment – Don’t know where to start with common questions like: Will my custom applications work with Windows 7? How do I upgrade? Can I upgrade? Etc… Here is a good starting point for you.

Windows Automated Installation Kit for Windows 7 – This is a MUST have when deploying Windows Vista and Later, creating Corporate Images, Doing Offline Servicing of hotfixes in your images? Etc…

Windows Automated Installation Kit for Windows 7 Direct Download Link - Be Prepared to download a 1+ gb ISO file that you will need to burn to a DVD.

MDT Solution Accelerator Home Page – Don’t think your environment is big enough to use System Center for deployment of Windows Machines, here is a Free Tool that will help you automate deployment of Custom Images from USB, WDS, DVD, Network Points. Not only will it help with deploying O/S’es it will help with deploying Applications as well. Note: This is considered a Lite Touch Environment, where user interaction will be needed, you can customize it to be very small.

MDT 2010 Direct Download Link – This does require that you have Windows Automated Installation Kit installed on the System.

Windows Deployment Services Getting Started Guide - We have made some changes in Windows 2008 R2 WDS Service, here is a link to how to get started using WDS for Network deployment of your Operating Systems.  Note: You can use a bootable image made from MDT to deploy using both MDT and WDS

Blog and Forum Sites to Keep an Eye On:

Microsoft Support Core Team Blog

The Deployment Guys Blog

Michael Niehaus Blog

Tim Mintner’s Blog

Microsoft Deployment Toolkit Technet Forum

System Center Useful Links

Operating System Deployment in Configuration Manager 2007 – Want to put SCCM 2007 in your environment, but don’t know what the Prerequisites, What all it does, How to configure it for your environment? If you answered yes to any of those questions, here is a good link for you.

System Center Configuration Manager 2007 OSD Comparision Matrix – Ever wonder how I can deploy Operating Systems? Well here is a chart from ADS 1.1 to ConfigMgr 2007 and the features/functionalities that they provide.

Operating System Deployment Task Sequence Variables – Variables, Variables, and More Variables… If you can’t keep track of every one of them, here is a cheat sheet for you.

The Configuration Manager Support Team Blog : A step by step for using ... – “Although the steps required to deploy an OS using Configuration Manager 2007 are available in many places, I decided to create a simple, concise step by step procedure for OSD deployment using SCCM that will hopefully come in handy if you are trying to use the OSD feature in SCCM for the first time.”

Troubleshooting Operating System Deployment – Ever wonder, why your deployment is having problems? What logs do I look at ? Here are common logs for when you are deploying with ConfigMgr 2007, that might help you out.

Blog and Forum Sites to Keep an Eye On:

Configuration Manager Operating System Deployment Forum

The Configuration Manager Support Team Blog

USMT 4.0

User State Migration Tool 4.0 User’s Guide - What would a Useful links post be without at least 1 User Guide? This user guide is the go-to place for anything to do with USMT 4.0, and what you can do with it.

What Does USMT Migrate? – Without little user interaction, what does USMT Migrate? How can I modify this information? Here is the link for you. You will find the Default Migration Scripts along with Supported Applications and several other goodies at here.

USMT Best Practices – Microsoft’s Best Practices for general and security related uses of USMT 4.0.

How to Use Hard Links for User State Migration – What is Hard Link Migration you ask? Well, it’s a way that you can maintain the user state data on the computer, while the old O/S is being removed and upgraded. This drastically reduces deployment time.

Step-by-Step: Basic Windows Migration using USMT for IT Professionals - “This step-by-step guide to Windows migration for IT pros provides a basic example of how to migrate files and settings from Windows XP to Windows 7 using USMT 4.0. (You can also migrate files and settings from a computer running Windows Vista®.)”

Activation

Windows Volume Activation – “Learn about the concepts, capabilities, and recommended best practices that can help you manage the activation of Windows Vista, Windows 7, Windows Server 2008, and Windows Server 2008 R2 in an enterprise environment.”

Volume License Key Management Portal - Portal to manage your Volume Activation Keys

KMS 2008 R2 Update – KB968912 – Have an existing 2008 KMS Server in your environment and want to start using Windows 7 or Windows 2008 R2, make sure you have this installed on the KMS Servers, so that it will accept your new keys.

KMS 2003 1.2 Update – KB968915 - Have an existing 2003 KMS Server in your environment and want to start using Windows 7 or Windows 2008 R2, make sure you have this installed on the KMS Servers, so that it will accept your new keys.

Boot to VHD

Add Native-Boot Virtual Hard Disk to the Boot Menu – “The following procedures describe how to add a native-boot virtual hard disk (VHD) to the boot menu using the BCDedit tool. If you are adding the VHD to a computer that already has a Windows® 7 installation, you will need to add a boot entry to the menu. If you are adding the VHD to a computer running an older version of Windows, for example Windows Server® 2008, you will need to update the system partition using the BCDboot tool and then edit the boot menu using the BCDedit tool. “

Step-By-Step Guide (Tested with Windows 7 Enterprise, for a “Corp” image and a “Personal” image)

Open Diskpart on the Windows 7 Enterprise Machine, you can also do this from within Disk Management (diskmgmt.msc) but the steps below are 100% Diskpart and Command line driven for automation purposes.

***This cannot be done in Windows 7 (Any Home Version, Professional) or any previous operating systems, as they do not support Boot to VHD

CREATE VDISK FILE=C:\VHD\Win7Corp.vhd MAXIMUM=75000 TYPE=EXPANDABLE

***Note the 75000 will create a Dynamically Expanding 75gb VHD file, it is recommended if you do this to go ahead and create a FIXED Size instead of Dynamically expanding so that you don’t accidently over commit the Physical disk. If you remove the TYPE=EXPANDABLE, it will default to FIXED.

SELECT VDISK FILE=C:\VHD\Win7Corp.vhd

ATTACH VDISK

SELECT VDISK FILE=C:\VHD\Win7Corp.vhd

CREATE PARTITION PRIMARY

FORMAT FS=NTFS LABEL="Win 7 Corp VHD" QUICK

ASSIGN LETTER=Y:

*** Must Have Windows Automated Installation Toolkit Installed on the system

Open Deployment Tools Command Prompt - Elevated

IMAGEX /Apply z:\win7x64\sources\install.wim 1 Y:\

SELECT VDISK FILE=C:\VHD\Win7Corp.vhd

DETACH VDISK

Modify the BCD Database via Elevated Command Prompt

bcdedit /copy {default} /D "Win 7 Corp Load"

bcdedit /set <GUID> device vhd=[C:]\VHD\Win7Corp.vhd

*** You will get the <GUID> during the bcdedit /copy step

bcdedit /set <GUID> osdevice vhd=[C:]\VHD\Win7Corp.vhd

Reboot the System and you will see "Windows 7" and "Win 7 Corp Load" on the Boot Manager Window

Application Compatibility

Application Compatibility Toolkit 5.5 – “The Microsoft Application Compatibility Toolkit (ACT) version 5.5 contains the necessary tools and documentation to evaluate and mitigate application compatibility issues before deploying Windows 7, Windows Vista®, a Windows Update, or a new version of Windows® Internet Explorer® in your environment.”

Windows 7 Application Compatibility Demo Video - Great Video on how to use the Application Compatibility Toolkit and how to create a “Shim”

Stock Viewer / Application Compatibility – Tool that you can use to play around with the Application Compatibility Toolkit

Managing Shims in an Enterprise – How to deploy the Shims you create with the Application Compatibility Toolkit.

Assessment and Planning Toolkit – “The Microsoft Assessment and Planning Toolkit makes it easy to assess your current IT infrastructure for a variety of technology migration projects. This Solution Accelerator provides a powerful inventory, assessment, and reporting tool to simplify the migration planning process.”

Special thanks to the following people for their contributions:

Scott McArthur (Sr. Support Escalation Engineer)

Kevin Ledman (Sr. Support Escalation Engineer)

Scott Goad (Support Escalation Engineer)

Clint Koenig (Support Escalation Engineer)

Author:

Tanner Slayton
Sr. Support Escalation Engineer
Microsoft Corporation

Using Pass-Through Disks in Conjunction with Clustered Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clusters

This blog is essentially an update, or follow-up if you will, to the original blog I wrote for Windows Server 2008 Failover Clustering.  With the release of Windows Server 2008 R2 comes the ability to Live Migrate highly available virtual machines between nodes in a Failover Cluster.  As if that were not enough to get customers excited about the product, we also include a feature called Clustered Shared Volumes (CSV) which is designed to work in conjunction with making virtual machines highly available in Windows Server 2008 R2 Failover Clustering using the Live Migration functionality.  There are users out there that are a little confused about CSV and whether  or not they must use CSV with Live Migration or can they still take advantage of pass-through disks in a virtual machine configuration.  I am here to tell you that you can use both simultaneously if desired and that is what we will discuss here.

The ultimate goal is to arrive at the configuration shown here where a highly available virtual machine is using a CSV volume for storing configuration files and the virtual hard disk (VHD) supporting the base OS while still taking advantage of a pass-through disk for data storage.

We start off with the assumption that a virtual machine has already been made highly available using a CSV volume to store the configuration file and the base OS vhd and Live Migration has already been tested and is working properly.

Next, we prepare a new LUN that has been presented to the cluster to be used as a pass-through disk.  In the Disk Management interface we can see the LUN has been presented and is Offline.

Since this is a LUN that the cluster has never seen before, we need to bring the disk Online first.

Once the disk is Online, we need to initialize the disk which will write a signature so the operating system can identify it.

Note:  The cluster uses the disk signature as one attribute for uniquely identifying storage that it controls.  How do we know cluster has control of a disk? 

Bonus material –  The disk shows as Reserved when a cluster has control of it.

Once the disk has been initialized, take the disk Offline.  There is no need to partition the drive as that will be done by the OS running in the virtual machine.

In order for the disk to be used by a highly available virtual machine, it must be under control of the cluster.  In the Failover Cluster Management snap-in, add the disk to the cluster.

Select the disk that is Offline.

Once added to the cluster, the new storage is placed in the Available Storage group and is brought Online.

Bonus material – Can a pass-through disk be used in the CSV namespace? 

If we try to add the disk to the CSV namespace –

The process will not complete –

Reviewing the error –

So, the answer is pass-through disks cannot be added to a CSV namespace because the drive must be Offline in preparation for being configured in a VM.  If the disk is Offline, the partition information cannot be read and CSV has a requirement for an NTFS partition. 

If you were to try and configure the virtual machine to use a disk that is not under the control of the cluster, you would see this pop-up when the virtual machine configuration is refreshed by the cluster.

Viewing the Details, the error is –

The reason is also explained –

With the disk added as a cluster resource and Online in the Available Storage group, access the settings for the running virtual machine by executing a Right-click on the virtual machine resource  and selecting Settings (or selecting Settings in the lower right-hand Actions Pane).

In R2, we can Hot-Add a hard disk to a pre-existing SCSI controller (if you wanted to use and IDE Controller, the virtual machine would have to be shutdown).  Execute the task by selecting the SCSI Controller and then select Hard Drive and Add.

Make sure to select the correct disk from the drop down list.

The refresh of the virtual machine should complete successfully and the new disk will be added to the group containing the Virtual Machine.

An examination of the resource dependencies for the virtual machine resource now includes the new disk that was added.

At this point, test Live Migration of the group to ensure all resources will come Online on other nodes in the cluster.

When Live Migration completes, we have achieved the desired configuration.

The final step is to prepare the new storage in the VM itself.

That wraps it up for this blog.  I hope you have found this information useful.  Come back and see us.

Additional references:

Configuring Pass-through Disks in Hyper-V

Microsoft Cluster Team Blog

Virtualization TechNet Center

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Creating a Virtual Machine to be stored on a Cluster Shared Volume

Cluster Shared Volumes is a new feature of Windows Server 2008 R2 Failover Clustering intended to enhance the capabilities of Highly Available Virtual Machines. It is recommended to utilize Cluster Shared Volumes if you intend to use the Live Migration feature of Windows Server 2008 R2 Hyper-V. In this blog I would like to cover a simple yet common oversight when creating a Virtual Machine to be used with Cluster Shared Volumes.

How to enable Cluster Shared Volumes

To enable Cluster Shared Volumes, on which you will place your Virtual Machines, we can use Failover Cluster Manager or the following Failover Cluster PowerShell CmdLets.

Using Failover Cluster Manager, simply select the cluster and from the Configure pane select Enable Cluster Shared Volumes.

Figure 1: Enabling Cluster Shared Volumes

Using PowerShell, ensure that the Failover Cluster module is loaded and run the following command:

PS C:\> get-cluster “your cluster name” | %{$_.EnableSharedVolumes="Enabled"}

Read the exception returned by the above command and then enable CSV following the instructions.

We must stop here for the fine print:     Cluster Shared Volumes Support for Hyper-V
In Windows Server 2008 R2, the Cluster Shared Volumes feature included in failover clustering is only supported for use with the Hyper-V server role. The creation, reproduction, and storage of files on Cluster Shared Volumes that were not created for the Hyper-V role, including any user or application data stored under the ClusterStorage directory of the system drive on every node, are not supported and may result in unpredictable behavior, including data corruption or data loss on these shared volumes. An example of a file type that is created for the Hyper-V role is a virtual hard disk (.vhd) file.

Once enabled, you can add one or more available storage clustered disks as a Cluster Shared Volume:

Figure 2: Adding disks to be used as Cluster Shared Volumes

Now that we enabled and added disks as Cluster Shared Volumes, we will have the following folder structure on the %SystemDrive%, which will typically be C:\

C:\ClusterStorage\Volume1
C:\ClusterStorage\Volume2
C:\ClusterStorage\Volume3

VolumeN above represents an entire clustered disk. These folders can be renamed for administrative purposes; however the C:\ClusterStorage folder must remain as it is.

For our example here I will rename C:\ClusterStorage\Volume1 to C:\ClusterStorage\AccntsDB-VMs.

For a more detailed walk-through of using Cluster Shared Volumes for virtual machine live migration see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

Now for the main purpose of this blog…

Creating the Virtual Machine on the Cluster Shared Volume:

For a Highly Available VM to function correctly all configuration and data files for the virtual machine must reside on a cluster shared disk that will be accessible to which ever node currently owns the VM. When using Cluster Shared Volumes this means all files must reside in the same folder, the folder where the Cluster Shared Volume disk is mounted, in our example C:\ClusterStorage\AccntsDB-VMs.

We create the Virtual Machine using Hyper-V Manager, as you would any VM. You can create a new VHD to install into or utilize an existing VHD. Below is an example of each. The critical point in this process is where to locate the entire Virtual Machine, not only the VHD file. I have seen many people copy the VHD file to the correct CSV folder but miss this step, resulting in all of the configuration files being placed into the default location, which most likely will not be available to all nodes in the cluster resulting in failures when attempting to make highly available or move the VMs between cluster nodes.

While creating a New Virtual Machine, choose a location:

Figure 3: Change the VM location to the desired CSV folder

Specifying a VHD file for the VM:

Figure 4: Specify the location and name for the VHD

Or use an existing VHD… (Of course we would need to have copied the file to this location previously)

                                                    Figure 5: Specify the location of an existing VHD

Once you have located the VM and VHD(s) onto the Cluster Shared Volume properly, proceed with creating the virtual machine as you normally would. In addition, we still need to create the Highly Available virtual machine resource using Failover Cluster Manager.

For a more detailed walk-through of creating virtual machines for live migration please see the following blog from one of our Clustering & High-Availability Program Managers, Symon Perriman:

Deploying Cluster Shared Volumes (CSV) in Windows Server 2008 R2 Failover Clustering

For more information please reference:

Understanding Cluster Shared Volumes in a Failover Cluster

Understanding Hyper-V and Virtual Machines in the Context of a Cluster

Errors you may encounter if not done properly include:

The operation has failed. None of the virtual machine configurations chosen was successfully made highly available.

Report Errors:

“Disk path 'C:\ProgramData\Microsoft\Windows\Hyper-V' is not a path to storage in the cluster or to storage that can be added to the cluster. You must ensure this storage is available to every node in the cluster to make this virtual machine highly available.”

There was a failure bringing the virtual machine configuration resource 'Virtual Machine Configuration New Virtual Machine' online.

Chris Allen
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Resource Hosting Subsystem (RHS) In Windows Server 2008 Failover Clusters

In this blog, I would like to explore some of the inner-workings of the Resource Host Subsystem (RHS) which is responsible for monitoring the health of the various cluster resources being provided as part of highly available services in a Failover cluster.   A Windows Server 2008 Failover Cluster is capable of providing high availability services using a variety of resources some of which are included as part of the Failover Cluster feature and others are as part of ’cluster-aware’ applications like SQL and Exchange. Resources are designed to work together and are typically organized in Resource Groups (Figure 1).  For example, a group of resources supporting a highly available File Server may consist of one or more of the following types of resources –   Client Access Point (IP Address(s) + Network Name resource), Physical Disk (Storage), and a File Server.  A highly available SQL Instance could contain the following resources -   Client Access Point (IP Address + Network Name resource), Physical Disk (Storage), SQL Server and SQL Server Agent.  Cluster resources are supported by special ‘plugins’ or resource Data Link Libraries (DLLs) that include coding to allow them to properly integrate\interoperate with the cluster service.

Figure 1

A Windows Server 2008 Failover Cluster is capable of hosting an unlimited number of resources.  The management of these resources is the responsibility of the Resource Control Manager (RCM) and the Resource Host Subsystem (RHS) which provide this functionality as part of the Cluster Service itself (Figure 2). 

Figure 2

The Resource Control Manager (RCM) is part of the overall cluster architecture and is responsible for implementing failover mechanisms and policies for the cluster service as well as establishing and maintaining the dependency tree (Figure 3) for each resource (e.g. a File Server resource requires a dependency on a Client Access Point and a Storage resource). 

Figure 3

The Resource Control Manager maintains the state for individual resources (Online, Offline, Failed, Online Pending, and Offline Pending) as well as for Resource Groups (Online, Offline, Partial Online, and Failed).  The Resource Control Manager can execute the following actions on a group of resources – Move, Failover and Failback.  Which action is executed depends on several factors including the current ‘health’ of resources in the group, administrative actions taken on the group (e.g. Move Group), or the current policies in effect for the group.  Here is an example (Figure 4) of Failover and Failback Group Policies –

Figure 4

Individual resources have policies (Figure 5) that apply to them as well.

Figure 5

The Resource Hosting Subsystem (RHS) is responsible for initially hosting all resources that come Online in the cluster in one default process – rhs.exe (Resource Host Monitoring process) (Figure 6).

Figure 6

Note:  The rhs.exe *32 process supports  32-bit resource DLLs running in the cluster.

 In previous versions of Microsoft clustering, this was called the resource monitor process (resrcmon.exe) (Figure 7).

Figure 7

There is one exception to this rule which has been implemented in the Windows Server 2008 R2 Failover Clustering feature.  In Windows Server 2008 R2, the Cluster Group which consists of the Cluster Network Name resource, one or more associated IP address resources and a ‘witness’ resource and the Available Storage group are considered to be ‘critical’ cluster resource groupings and are hosted in an rhs.exe process separate from all the other cluster resources.

The Resource Hosting Subsystem (RHS) conducts periodic health checks of all cluster resources to ensure they are functioning properly.  This is accomplished by executing   IsAlive and LooksAlive processes which are specific to the type of resource.  Examples of these are documented in the following KB article –

KB 914458 -   Behavior of the LooksAlive and IsAlive functions for the resources that are included in the Windows server Clustering component of Windows Server 2003.

How often health checks are conducted is determined by the specific resource DLL or by a policy set by the cluster administrator.  An example of this policy is shown in Figure 5.  Should a resource fail to respond to a low-level LooksAlive check, a more in-depth IsAlive check is conducted.  If a resource fails an IsAlive check, additional policies are executed until such time it is determined that a resource cannot run on a particular node in the cluster.  When that point has been reached, RHS notifies the Resource Control Manager which will report the resource as Failed to the cluster service and a Failover is executed to move the Resource Group to another node in the cluster provided the default policy (Figure 8) is in effect.

Figure 8

There are times when a cluster administrator will choose not to implement the default policy shown in Figure 8 for specific ‘non-critical’ resources.  This reduces instability in the cluster which could adversely impact clients connected to highly available service(s). 

The IsAlive and LooksAlive health monitoring function is but a small part of what can be done with cluster resources.  Figure 9 shows a listing of additional Resource DLL Entry-Point functions. 

Figure 9

Note:  Information on the Failover Cluster APIs can be found on MSDN.

Failure of an IsAlive call into a resource is but one way resources can become unavailable in the cluster.  Other ways include:

• Deadlocks in a resource DLL
• Crashes in a resource DLL
• RHS process itself terminates in the cluster
• Cluster service fails on the node
• Operating system failures (e.g. resource exhaustion)

Most of us who have been working with clusters for a long period of time understand what happens if a resource fails a critical health check.  I want to spend a little time discussing resource deadlocks. 

What is a resource ‘deadlock’?  Basically, there are two common reasons for instability within a resource DLL.  The resource DLL itself crashes (e.g. access violation in the resource DLL) or the resource fails to respond to a command in a timely fashion.  Every time a call is made into a resource, a timer is started.  If a response is not received within a specific period of time (configurable), the resource is considered to be deadlocked and  the RHS process hosting that resource will be terminated and the resource will be placed in a newly created RHS process thereby isolating it from all the other resources running in the default rhs.exe process.   When a deadlock happens, the Failover Cluster service registers an event in the cluster log.  Here is an example of a deadlock occurring in the ‘Cluster Name’ resource –

 Figure 10

Entries are also made in the Windows System Event Log.  Here is an example –

MDT 2010: Incorrect wimgapi.dll version causing WIM mounting issues

Today’s blog will cover an issue we have seen with Microsoft Deployment Toolkit 2010.  You may see one or more of the following error messages when generating or updating boot images and other actions in MDT that involve the Lite Touch Images:

• Unable to mount the WIM, so the update process cannot continue.
• Unable to load DLL 'wimgapi.dll'
• Mount did not succeed

You can also run into errors when Windows System Image Manager tries to catalog an image.  This issue can occur because we are finding an incorrect version of WIMGAPI.DLL first in the path

Steps to Resolve

At a command run the following command

Where wimgapi.dll

For the first location listed in the page verify that the .DLL Version is correct.  If MDT 2010 is installed on Windows 7 or Windows Server 2008 R2 you should see wimgapi.dll in two locations because it ships with the operating system and with the Windows Automated Installation Kit (WAIK). 

C:\windows\system32\wimgapi.dll
C:\program files\windows imaging\wimgapi.dll

Both versions should be 6.1.7600.16385.  The version could be later than this if any updates have shipped that replace this file at a later date

Notes: 

• This issue can occur if you installed a beta version of the WAIK or System Center Configuration Manager (SCCM)
• You may find other versions in locations like Program Files(X86) or other directories.
• If MDT is installed on Windows XP or Windows Server 2003 wimgapi.dll should only be found in C:\Program Files\Windows Imaging
• The version of WIMGAPI.DLL may be updated in later releases of the Windows AIK or updates to Windows. 

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Unable to select an attached VHD as a Shadow Copy Storage location

You may notice that you cannot choose to store Shadow Copies on an attached VHD, and that when configuring Shadow Copy protection on an attached VHD, there are no other locations available to store the copies on, other than the protected VHD volume.

This behavior is by design. Illustrated below is the behavior you may see while working with attached VHDs and Shadow Copies. I’ll provide some additional information at the end of this post that will explain why this behavior is occurring.

To get into a configuration where we’ll see the behavior I’m describing:

We create our VHD in Fig.1

(Fig.1)

Remember the size of the destination volume for shadow copies must be at least 300mb before you can store a single shadow copy on the volume

(Fig.2)

Here, in Fig.2, I’m making it 350mb so that there wouldn’t be any concern about the volume being too small and thereby omitted from the list of possible locations.

Below in Fig.3, we’re going through the steps of initializing the new disk

(Fig.3)

Now, in Fig.4, we’re putting a simple volume on it

(Fig.4)

(Fig.5) Enabling Shadow Copies on the C:\ volume

In Fig.5 above, I’ve pulled up the properties of the C:\ in preparation to enable Volume Shadow Copies. You can see here, that Disk 3 (F:\) is present in the list of volumes that we can enable the feature on. F:\ is the attached VHD we created in Fig.2.

In Fig.6, let’s enable Shadow Copies on C:\ and then choose to put the copies on our F:\ drive…

(Fig.6)

Whoa… where’d the F:\ drive go?

As you can see from the list, the attached F: (VHD file) is not listed as a valid target for Shadow Copy Storage.

Also, while you can configure the attached VHD to be protected with Shadow Copies, you cannot store the Shadow Copies on any other volume but itself.

Why is this happening?

A virtual volume cannot be used as the target volume for the snapshot of another volume, and the volume can only hold shadow copies associated with its own snapshots. VSS will constrain the shadow copy storage area for attached VHDs to only allow the volume to host its own shadow copies. This behavior will ensure that the volume is self consistent, and therefore, maintain its portability. Portability is one of the bigger design goals behind the ability to create and mount VHDs under Windows 7 and 2008/R2. The end result of this behavior is that snapshots of the volume will travel with the volume as it is deployed and moved around your environment so there’s never a need to worry about a volume missing when it comes time to restore a previous version of a file.

For additional information, be sure to visit these links:

Frequently Asked Questions: Virtual Hard Disks in Windows 7

http://technet.microsoft.com/en-us/library/dd440865(WS.10).aspx#one

Understanding Virtual Hard Disks with Native Boot

http://technet.microsoft.com/en-us/library/dd799282(WS.10).aspx

Thanks for taking the time to read this. I hope it’s helped you understand one of the caveats that can be seen when using Native VHD support!

Sean Dwyer
Support Escalation Engineer
Microsoft Enterprise Platforms Support

The Four Stages of NTFS File Growth

In my quest to better understand the interworking of how NTFS stores information on disk, I have been researching what happens to a file as it grows in size and complexity.  The reason I’m after this knowledge is so I can better troubleshoot certain storage issues. 

Recently, I realized that I’d stuffed my head with enough information to make a pretty good blog.  Read along as I explain what I call ‘the four stages of file growth’.

Before we can address file growth, we need to first look at how NTFS works under the covers. 

Let’s start out with some basics.

When NTFS stores a file, it starts by creating a small 1KB file record segment that we will call the base record.  Every file starts like this, including the special hidden files such as $MFT, $LOGFILE, $VOLUME and so on.  In fact when we refer to the MFT (master file table), what we are talking about is the entire list of base record segments and child record segments (explained later) for all files in the volume.

For today, we are just going to talk about some simple text files.  You will see it getting complex enough without us doing anything fancy.  Here are three base records for three text files.

Before going any farther, it is important to clear up a common misconception on what a file really is.  We tend to think of the data in our file as the file itself.

The truth is that data is just one attribute of a file.

Every file record starts with a header, and then has various attributes, each attribute having its own header.  For small files, it is common to find the data attribute last.

Do not confuse these attributes with file attributes like Read-only, Hidden, or System (which are actually just flags).  Think of attributes as structures within the file that define things about the file.  Common attributes are $STANDARD_INFORMATION, $FILE_NAME, and of course $DATA.

Any space left over in the 1KB record is unused until one of the attributes needs it or a new attribute is added.

Now let’s watch our file grow….

Stage one – Completely resident

I created a small text file with just one line of text in it.  This file was so small that it was able to fit all parts of the file into its base record.  We call this being resident, as the data for the file resides in the base record segment.  This also means that the entire file exists in the MFT.  No need to look elsewhere.  Everything we need is in that 1KB record.

The diagram shows our 1KB base record segment for the file File1.txt.  Inside you can see the data attribute and the file data within it.  The file data, also known as the stream for this attribute, is what we as computer users tend to think of when we think about a file.  We don’t think about all the structures involved in storing the stream.

Along with the data that we put in the file, you can also see that we have lots of room still left in the 1KB base record segment. 

To make the file grow, I just pasted the same line of text into it a few more times.  Soon I had the file looking like this....

This was about as big as I could get the file before it was too big to fit into the 1k range of the base record segment.  Any bigger and we go to stage two.

Stage two – Nonresident Data

Once the data starts to push out toward the end of the 1KB base record segment, the data will be shipped outside and stored elsewhere on the disk.  To keep track of where it is, we maintain a mapping pair that tells us the location and length of the now nonresident data.  The new location is outside the MFT and is simply an allocated range of clusters.

NOTE:  At his stage the file data is nonresident, but the attribute record is in the base record segment.

As the file continues to grow we will either increase the length defined by the mapping pair, or if we can’t store the data contiguously, we create more mapping pairs. 

Eventually, the file starts to look like this….

Stage three – Nonresident Attribute

When an attribute grows to the point that the list of mapping pairs no longer fits into the base record segment, it too is shipped out but this time it is housed in a new child record.  To keep track of this child record a new structure is created in the base record.  We call this new structure an attribute list or $ATTRIBUTE_LIST.

Each entry in the attribute list points to the file record where each attribute instance can be found.  There will be an attribute list entry for nearly every attribute that the file has.  The exception being that there isn’t a list entry for the attribute list.  For the attributes still resident (like the $FILE_NAME attribute), their respective list entry will simply point back to the base record segment.  The diagram above shows only the one entry that corresponds to the $DATA attribute.  The other entries are left out of the diagram to keep it readable.

After even more data is stuffed in the file it branches out and creates more child records as needed.  Each child record has an entry in the attribute list that points to it.

This is somewhat different than what we did when we moved file data outside the base record segment.  When the file data was moved, the new location on disk contained no attribute information.  It just had data.  If viewed in a sector editor, it would just show lines and lines of file data.

The child records are just that, records.  They contain elements common to those found in a base record segment.  It will have a MULTI_SECTOR_HEADER and one or more attribute records….along with some mapping pairs.  The pairs themselves will point to the allocated clusters that contain actual file data.

The information dumped out of the file gets more complex at each stage.  But fear not, it’s almost over.

Stage four – Nonresident Attribute List

The final stage of file growth occurs when attribute list contains so many entries that the list itself no longer fits in the base record segment.  When we reach that point, the attribute list is shipped outside the record into an allocated cluster range and an attribute list record is left behind to track the location of said cluster range.  The new location of the attribute list is outside the MFT and is similar to how we are storing the chunks of data that make up the $DATA:”” stream(shown in the red boxes) in that it is not an actual child record.

The dotted line shows the entire stream as it would be virtually.  Logically these chunks of data will be found all over the storage device. 

Unlike the child records and the data instances, a file can only have one attribute list and the $ATTRIBUTE_LIST record must reside in the base record even though the list is nonresident.

In review

Stage one- Completely resident

A file starts out simple, storing file data locally.

Stage two- Nonresident data

When the data will no longer fit in the 1KB range, it is moved to another part of the disk.

This process can result in multiple mapping pairs.

Stage 3-Nonresident attribute

When the mapping pairs are too numerous, they are moved out to form their own child record. 

An attribute list entry is created for each child record.  Multiple attribute list entries will mean multiple child records.

Stage 4-Nonresident attribute list

Lastly, when the list of attribute entries is too large to be stored inside the base record segment, the attribute list itself becomes nonresident and moves outside the MFT.

The greater the complexity used to store the file, the greater the performance hit will be to your computer when retrieving and storing the file.  Things like compression, file size, number of files, and fragmentation all can greatly affect this complexity and therefore affect your computer experience. 

Robert Mitchell
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

KMS Host Client Count not Increasing Due to Duplicate CMID'S

Hello, my name is Scott McArthur.  I am a Senior Support Escalation Engineer in the Windows group and today’s blog will cover an issue involving KMS activation and deployment of images.    This issue seems to be more prevalent today due to the various tools used to clone images, create images based on a templates, Physical to Physical (P2V) tools, etc... 

Generally the symptom you will see is that the count on your KMS host will not show the correct number of clients or it will not increase.  There are a number of reasons why this can occur but a common reason is that sysprep was not used when preparing images for deployment. 

Any time you use imaging as your deployment method it is required that you run sysprep to prepare the image for deployment. 

This policy is outlined in the following KB article:  http://support.microsoft.com/default.aspx?scid=kb;EN-US;162001

To determine if you are encountering this you can use the Key Management Service Log do the following:

1. On your KMS host open Event Viewer
2. Right click the Key Management Service Log and choose “Save all events as”
3. Change the Save as type to Text(Tab Delimited)(*.txt)
4. Save the file as KMS.TXT
5. Close out of the Event Viewer completely
6. Open Excel
7. Click File, Open, and browse to KMS.TXT
8. You should see the Text Import Wizard. Choose the following options
    Delimited
    Start Import at Row: 8
    Delimiters: Comma
9. When complete the data may look all messed up. Don’t worry we will correct that
10. Click the upper left of the spreadsheet to select the entire spreadsheet
11. Click Data, Sort, In the Sort By selection choose “Column D”
12. When complete you should see the data sorted in columns.

The Client Machine ID (CMID) is how we uniquely identify a KMS client.  When sysprep is run one of its jobs is to generalize this GUID so when the image is deployed every machine has a unique CMID.  Here is an example output

From this output you can see that multiple computernames have the same CMID. Each computer should have a unique CMID. This means that sysprep /generalize was not used to prepare these computers for deployment. So to KMS those 4 machines appear as one. That what be why the count would not be increasing or not reflect the true number of machines deployed.

While it is possible to run slmgr.vbs /rearm to reset the machines CMID that does not leave the machine in a supported state. Images deployed without using Sysprep to prepare the image are not supported by Microsoft. Sysprep executes ~30 sysprep providers. These providers are written to correct issues with various components when you duplicate the installation. By not running sysprep it is unknown what types of issues you could encounter and many components will be in a broken state. The supported solution is to rebuild the image using the Sysprep /generalize switch and redeploy the systems.

Thanks for your time. Stay tuned to our blog for more Activation and Deployment Topics.

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

How to run a Sysprep and Capture Task Sequence From MDT 2010

Hello, my name is Kevin Ledman. I am a Support Escalation Engineer in the Windows group and today’s blog will cover how to run the new Sysprep and Capture Task Sequence included with MDT 2010.

If you choose to deploy an operating system manually or need to make customizations outside of the MDT task sequencer, you can still use MDT to automatically sysprep and capture the image for future use.

To configure the task sequence, launch the MDT 2010 deployment workbench and create a new task sequence using the sysprep and capture template.  Answer the remaining wizard items, making sure to choose an OS source that matches the OS you are going to be capturing.

Update your deployment points and switch to the reference computer to start the task sequence.  **A common mistake at this point is to boot the reference computer from your LiteTouch image and start this task sequence.  The sysprep and capture task sequence is designed to be run from the desktop of the reference machine similar to a post OS installation task sequence.  To launch this, you will need to establish connectivity to the deployment share and launch LiteTouch.WSF manually.  Because you are logged in to the reference machine as a local administrator and not joined to a workgroup, be sure to establish the session under the same security context that will be used for the task sequence:

net use * \\mdtserver\DeploymentShare$ /user:domain\username

Once the connection is established, execute LiteTouch.WSF:

cscript \\mdtserver\DeploymentShare$\Scripts\LiteTouch.WSF

The MDT Wizard Screens will launch and prompt for the information required to complete this task sequence.  **Note – we will still process customsettings.ini for this task sequence.  If you have modified customsettings.ini to skip wizard screens, those settings will be honored with this task sequence as well.

Choose the task sequence you have created.

Choose the capture option and supply the location and file name.

Supply the credentials that LiteTouch will use to connect to the deployment share.

View the summary and click ‘Begin’ to start the task sequence.  If you receive error “A connection to the distribution share could not be made” see the following blog: 

http://blogs.technet.com/msdeployment/archive/2009/09/18/fix-for-multiple-connections-to-a-server-or-shared-resource-by-the-same-user-using-more-than-one-user-name-are-not-allowed-problem-with-mdt-2010.aspx

MDT will copy the necessary files to the reference computer, launch sysprep, apply the LiteTouch Image and reboot the machine

.

LiteTouch boots and begins the capture of the image.  Depending on the size of the installation, this may take a significant amount of time.

Once the capture has completed, you can now import the captured image as a custom image file in MDT and use it for future task sequences.

Add new operating system and choose custom image file.

Point to the “Captures” path and move it to the to the deployment share.

Include the setup files for the OS which you are importing and complete the wizards.

The operating system is now ready for use with new task sequences.

Kevin Ledman
Support Escalation Engineer
Microsoft Enterprise Platforms Support

Invalid Product Key Error Specifying MAK key in unattend.xml

Hello, my name is Scott McArthur.  I am a Senior Support Escalation Engineer in the Windows group and today’s blog will cover an issue involving specifying MAK Product Keys during setup of Windows 7 and Windows Server 2008 R2. 

When deploying a Volume License (VL) version of Windows 7 or Windows Server 2008 R2 you may encounter the following error message:

The unattend answer file contains an invalid product key.  Either remove the invalid key or provide a valid product key in the unattend answer file to proceed with Windows Installation

The setuperr.log will log the following

2009-10-01 12:52:38, Error      [0x060551] IBS    Callback_Productkey_Validate: EditionID for product key was NULL.
2009-10-01 12:52:38, Error      [0x060554] IBS    Callback_Productkey_Validate: An error occurred writing the product key data to the blackboard.
2009-10-01 12:52:38, Error      [0x06011a] IBS    Callback_Productkey_Validate_Unattend:Product key did not successfully validate.[gle=0x00000490]
2009-10-01 12:52:38, Error      [0x0603c7] IBS    Callback_Productkey_Validate_Unattend:Did not pass validation; halting Setup.[gle=0x00000490]
2009-10-01 12:52:38, Error      [0x060120] IBS    Callback_Productkey_Validate_Unattend: An error occurred preventing setup from being able to validate the product key; hr = 0x80300006[gle=0x00000490]

This error can occur if you have specified a Multiple Activation Key (MAK) in your answer file in the WindowsPE phase of setup.  VL versions do not prompt for a ProductKey so they do not need a ProductKey during the WindowsPE phase of setup.  The ProductKey can be specified by clicking “Change Product Key”, SLMGR.VBS /IPK, or specifying it in the answer file. 

The ProductKey entry is found in 2 places in Windows System Manager:

Microsoft-Windows-Setup in WindowsPe phase

The above entry would be used when using retail media. 

Microsoft-Windows-Shell-Setup in Specialize phase

To resolve this issue use the Microsoft-Windows-Shell-Setup component in the Specialize phase.  In order to get the ProductKey entry you must right click the Microsoft-Windows-Shell-Setup component under Windows Image and add it to the Specialize phase. 

ProductKey will not show up under the component(On the Answer File pane) until you actually add it.  The unattend.xml will look like this

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="specialize">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <ProductKey>xxxxx-xxxxx-xxxxx-xxxxx-xxxxx</ProductKey>
        </component>
    </settings>
    <cpi:offlineImage cpi:source="catalog://server/catalogs/win7/enterprise/x86/install_windows 7 enterprise.clg" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

Note:  If you are using Microsoft Deployment Toolkit 2010 for your deployments you can enter the MAK key during the ProductKey prompt during the Lite Touch Wizard.  Hope this helps with your deployments and keep an eye on our blog for other activation issues. 

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support