Ask the Core Team

Exchange DAGs and Cluster Networks

Jeff Hughes [MSFT] — Mon, 21 May 2012 12:41:08 GMT

One of my colleagues here at Microsoft wrote a couple blogs on managing Exchange networks on a cluster so I thought I’d post the links since they are tied into behavior you would see in a clustered environment.

Exchange 2010: Mapping DAG network states to cluster network states…

Exchange 2010: Collapsing DAG Networks

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

MDT 2010 / 2012 – My deployment failed. What logs should I review and where are they located?

Jeff Hughes [MSFT] — Fri, 18 May 2012 11:51:59 GMT

One of the most common questions I get is “What logs should I look at if my deployment fails?” So here is a little summary of the logs you will be most concerned with when troubleshooting a failed Windows 7 or Windows Server 2008 R2 install that are being deployed via Microsoft Deployment Toolkit 2010 or 2012.

First thing to note is that the location of the logs move around depending on what portion of setup we are talking about. Here is a breakdown of the log locations during a MDT install task sequence:

Before the Image is applied to the machine:
X:\MININT\SMSOSD\OSDLOGS

After the system drive has been formatted:
C:\MININT\SMSOSD\OSDLOGS

After Deployment:
%WINDIR%\TEMP\DeploymentLogs

The logs of most interest for troubleshooting a failed install will be:

BDD.LOG – This is an aggregated log of all the MDT Logs.
SMSTS.LOG – This log would be used to troubleshoot Task Sequence errors.

Also note that each MDT script creates its own log files during execution (example: ZTIGather.log, ZTIDiskpart.log, ZTIDrivers.log, etc)

Next Question is: How do I read the logs?

Although you can view the logs with Notepad, it can be hard to make sense of the info in a regular text editor. The MDT logs are most easily read by Trace32 (which is part of the Microsoft SCCM 2007 Toolkit.

Here is an excerpt of a BDD.log in Notepad:

Here is that same portion of that bdd.log viewed in SMS Trace:
Notice warnings are highlighted in yellow, and errors in red.

Now that you know how to locate more detailed error information in the logs, here are some locations that you can use to search to help find solutions to your issues.

The “Ask The Core Team” Blogs on TechNet:
“The Deployment Guys” Blogs on TechNet:
The MDT Social Forums on TechNet:
The Microsoft Deployment Toolkit homepage on TechNet:
The MDT Help Files also have a lot of information and serve as a good reference for finding proper syntax and examples.

Hopefully, this information will help aid your MDT troubleshooting. Being armed with the proper tools and knowing where to look for the answers is the key to troubleshooting success.

Bill Spears
Microsoft Enterprise Platforms Support
Senior Support Escalation Engineer / Premier Field Engineer

Requirements to save Bitlocker Recovery Key to AD using MDT

Jeff Hughes [MSFT] — Wed, 16 May 2012 13:45:43 GMT

My name is Naziya Shaik and I am a Support Escalation Engineer with Windows Core team @ Microsoft. I would like to share information about enabling BitLocker while deploying operating system via MDT and the group policies that are required to be configured in AD DS.

If you use Microsoft Deployment Toolkit to deploy Windows one of the prompts during Lite Touch is the prompt to enable Bitlocker Drive Encryption and you are given the option to save the bitlocker recovery key “in active directory”. See Figure 1

Figure 1. Bitlocker prompt during Lite Touch

You can also enable the step manually in the task sequence. See Figure 2

Figure 2. Bitlocker step in TS

The issue we have seen is that some customers believe that this is all that is required to save the recovery key to active directory.

In order for MDT to save the recovery key to active directory you must at a minimum enable the following group policy:

Deploying Windows 7 with Windows Server 2003 or Windows Server 2008 DC

Path to GP: Computer Configuration\Administrative templates\Windows Components\Bitlocker Drive Encryption

Policy: Turn on Bitlocker backup to Active Directory Domain Services

Deploying Windows 7 with Windows Server 2008 R2 DC

Path to GP: Computer Configuration\Administrative templates\Windows Components\Bitlocker Drive Encryption\Operating System Drives

Policy: Choose how Bitlocker-protected operating system drives can be recovered

Additional configuration may be required also depending on your scenario. For more information see “Backing Up BitLocker and TPM Recovery Information to AD DS”

The following blog refers to a script to back up the recovery information to AD DS.

How to backup recovery information in AD after BitLocker is turned ON in Windows 7

Naziya Shaik

Support Escalation Engineer

Microsoft Enterprise Platforms Support

Windows 8 and .Net Framework 3.5

John Marlin [MSFT] — Mon, 14 May 2012 15:03:00 GMT

With each new release of Windows, it's a challenge to balance keeping the OS footprint small with maintaining forward compatibility for applications and devices. The .Net Framework is at the heart of this challenge. In Windows 8, Microsoft decided to take a new approach to the way that such features are installed. In this post, I will describe the new approach, how the changes might affect you, and how you can prepare for them to ensure a smooth transition to Windows 8.

About the .Net Framework

The .Net Framework provides a class library and a Common Language Runtime (CLR) that allow software developers to create rich, secure applications. Additionally, the Framework provides the functionality required for such applications to run. A program developed using a particular version of the Framework typically requires that version, or a compatible one, to be installed on computers where it will run.

Windows 7 and Windows Server 2008 R2 included .Net Framework 3.5. Additionally many applications have been written using .Net 4.0, so that version is often installed using a redistributable package from Microsoft. Windows 8 and Windows Server 2012 include .Net 4.5, which supports building and running the next generation of applications and web services, including Metro-style apps. .Net 4.5 supports applications written for 4.0, so there is no need to install .Net 4.0 on Windows 8.

Features on Demand (FoD)

"Features on Demand (FoD)" is a new concept in Windows 8 that allows administrators and image builders to reduce the amount of space used by the component store by adding only the payload for optional components they need to a system image. "Payload" refers to the binaries and other files associated with a feature. Features on Demand also allows for the addition of roles and features to an image at any time they are needed .

In Windows 8,.Net Framework 3.5 is now a Feature on Demand. And to simplify the installation of common legacy versions of the .Net Framework, .Net 3.0 and 2.0 have been included in the same feature package as 3.5. That means if any of those three versions need to be installed, all the administrator needs to do is enable the single .Net Framework 3.5 feature in Windows 8.

While we encourage developers to create or upgrade their applications using .Net 4.5, we realize that many commonly-used apps exist that depend on older versions of .Net, and that it takes time for developers to upgrade their code and for customers to upgrade to the new apps. For these reasons we have provided a variety of methods by which customers can enable the legacy versions of .Net in Windows 8.

Installation Sources

The .Net Framework 3.5 payload can be obtained from any of the following sources:

· Windows Update (WU)

· A Windows Image file (.wim) to which the payload has been added

· The \sources\sxs folder on the installation media

There are unique advantages to using each. The source can be specified for the environment using a new Group Policy setting. It can also be specified when installing .Net 3.5 manually on an individual machine or image.

The simplest scenario is one in which WU is accessible to both the machine and the user, and the machine is not configured to obtain updates from Windows Server Update Services (WSUS). In this case, when the feature is enabled, the user will be prompted for permission to download the update. If permitted, Windows will download the payload directly from Windows Update and install the feature. Done!

In more controlled environments, administrators might want to redirect such download requests to an alternate source such as a Windows Image file (.wim) to which the payload was added, or the\sources\sxs folder from the installation media. There might also be network , proxy, or security configurations that prevent users from directly accessing Windows Update. Additionally, WSUS does not currently support the payloads for Features on Demand, although it does support the subsequent patching of the features. So in environments where machines are configured to obtain updates from WSUS, administrators will need to configure the source for initial FoD installations.

To allow administrators to manage these scenarios, a new Group Policy setting was introduced in Windows 8 / Windows Server 2012: "Specify settings for optional component installation and component repair”, located under Computer Configuration\Administrative Templates\System:

This policy allows the administrator to configure the installation of Features on Demand and feature store repair operations to use only authorized locations.

When this policy is enabled, a network location (for example, a file server) can be specified for both repair of the feature store, and enabling features whose payloads were not originally added. The Alternate source file path can point to a \sources\sxs folder or a Windows image (WIM) file using the WIM: prefix. The repair WIM can be different than the initial WIM file used for installation. You can specify multiple paths by using ";" between the paths. Valid syntax is "wim:<path to wim>:<index>". Or "<path to sxs folder>".

Examples:

\\server\Win8Media\sources\sxs

wim:\\server\sourcewim\install.wim:3

If you select Never attempt to download payload from Windows Update, WU is not contacted during an installation or repair operation.

If you select Contact Windows Update directly to download repair content instead of Windows Server Update Services (WSUS), attempts to add features (for example, .NET Framework 3.5) or repair the feature file store use Windows Update to download files. Target computers require Internet and WU access for this option. Normal servicing operations continue to use WSUS if it has been configured as a source.

The advantage of a WIM file is that it can be kept current with updates. This may be important because files associated with features that have post-RTM updates will be repaired to their current versions if the specified source contains the current updates. It is important to note however, that the initial installation of FoDs from an updated WIM will still use RTM versions of files, with subsequent updates being applied using standard patching sources such as WU or WSUS. I'm not going to cover how to create and maintain a WIM file here. For more information on how to do that see Deployment Image Servicing and Management (DISM) Technical Reference .

The\sources\sxs folder on the installation media can provide a quick and easy local source for initial installations. In addition to referencing the sxs folder in the policy setting above, you can specify it as the source for manually installation on individual machines. To make the source files available on the internal network, use the XCOPY command to copy the source files to a network share, and then reference the share as a mapped drive or a UNC where applicable:

xcopy d:\sources\sxs\*.* x:\Win8Media\sources\sxs\ /s

Installation Methods

Now that we have covered the various sources, let's talk about how to enable the feature. There are three basic ways .Net 3.5 can be enabled, in order of preference: proactively, manually, or automatically.

By far, the most hassle-free way to install any FoD to more than a very few machines is to add the payload and enable the feature on images you will be deploying in your environment. Couple that with configuring the Group Policy setting mentioned earlier to point to the right source for one-off scenarios, and about the only administrative work you will have beyond that will be maintaining an updated WIM file for feature repair. Again, I am not going into details here about how to create images for broad deployment. For more information on how to do that see Deployment Image Servicing and Management (DISM) Technical Reference.

The simplest way to enable .Net 3.5 on a small number of Windows 8 client machine is through Turn Windows features on or off:

1. On the Start Screen begin typing “turn windows features on or off”, select Settings in the search pane, and click on Turn Windows features on or off.

2. Check the box next to .Net Framework 3.5 (includes .NET 2.0 and 3.0)

3. The wizard will search for required files and then prompt you to download files from Windows Update.

4. Select Download files from Windows Update.

5. After the wizard completes, click Finish.

The process on Windows Server 2012 is similar but is accomplished using the Add Roles and Features Wizard:

1. In Server Manager, click Manage, then select Add Roles and Features.

2. Click Next at the Before you begin screen.

3. At the Select installation type screen, select Role-based or feature-based installation and click Next.

4. On the Select destination server screen, select the target server and click Next.

5. On the Select server roles screen, click Next.

6. On the Select features screen, check the box next to .Net Framework 3.5 Features and click Next. If you expand the tree it looks like this:

7. On the Confirm installation selections screen, a warning will be displayed asking "Do you need to specify an alternate source path?...". If the target machine does not have access to Windows Update, click the Specify an alternate source path link to specify the path to the \sources\sxs folder and click OK:

8. After you have specified the alternate source, or if the target machine has access to Windows Update, click Install.

Note that the Add Roles and Features Wizard gives the administrator the option to specify an alternate resource, whereas the Turn Windows features on or off dialog does not. In most environments, administrators will deploy standard images of client machines that include the necessary features, or they will configure Group Policy to specify the installation source for Features on Demand.

Administrators can also use Deployment Image Servicing and Management (DISM) or Powershell cmdlets to enable the feature. DISM can be used to install the feature on individual computers, or to install it on an image that will be deployed to multiple computers.

Here are some sample DISM commands to enable and get the status of the .Net Framework 3.5 feature:

Dism /online /get-featureinfo /featurename:NetFx3

Dism /online /enable-feature /featurename:NetFx3 /All

Dism /online /enable-feature /featurename:NetFx3 /All /LimitAccess /Source:x:\sources\sxs

Use /All to enable all parent features of the specified feature

Use /LimitAccess to prevent DISM from contacting WU/WSUS

Use /Source to specify the location of the files needed to restore the feature.

(x: is the drive letter of the installation media or mapped network share that contains a copy of the installation files)

To install using Powershell, use Server Manager cmdlets on servers to install features. (It should be noted that the Get-WindowsFeature cmdlet may not show a feature as installed unless all of its parent features are also installed):

Get-WindowsFeature –name NET-Framework-Core

Install-WindowsFeature –name NET-Framework-Core

Install-WindowsFeature –name NET-Framework-Core –source x:\sources\sxs

Similar DISM Powershell cmdlets can be used on either client or server, but note again that the Server manager cmdlets above are recommended on server..

Get-WindowsOptionalFeature –Online –FeatureName NetFx3

Enable-WindowsOptionalFeature –Online –FeatureName NetFx3 –All

Enable-WindowsOptionalFeature –Online –FeatureName NetFx3 –All -LimitAccess -Source x:\sources\sxs

Windows features that require .Net 3.5

As of this writing, the only Windows feature I am aware of that requires .Net 3.5 (beyond those that are specific to .Net 3.5 and ASP.Net 3.5) is Powershell 2.0. If you need it for script compatibility, you will need to install .Net 3.5 to get Powershell 2.0 functionality. On Windows 8 client, Windows PowerShell 2.0 is already enabled and you simply need to enable .Net 3.5 to use it. On server, you will need to install the Windows PowerShell 2.0 Engine, which will prompt you to also install .Net 3.5.

Applications

The automatic installation trigger I alluded to earlier requires a discussion about installing applications.

As mentioned before, many applications rely on older versions of the .Net Framework. In fact, some current versions of Microsoft products including Microsoft SQL and some System Center products use older versions.

Many applications perform checks during their installation to verify that the required version of the .Net Framework is installed. If not, they install the necessary version, often using a redistributable package available from Microsoft. However, the methods that application installers use to verify and install .Net can vary. The experience of installing apps in Windows 8 might have unexpected results because of how external attempts to install .Net are now intercepted and handled by Windows.

Installation of the new.Net Framework 3.5 FoD package can be triggered automatically in Windows 8 in the following scenarios:

· You attempt to install .Net 2.0, 3.0, or 3.5 using a redistributable package available for download from Microsoft.

· An application attempts to install one of the redistributable packages for a required version during its own installation process.

· An application that requires a legacy version, executes without preinstalling the required version.

In each of these cases, an application shim in Windows 8 intercepts the attempt and invokes the installation of the new .Net 3.5 feature. Once triggered, the installation should proceed as if it was initiated from the UI, DISM, or Powershell.

The classic "Your mileage may vary" disclaimer applies here because we really don't know how all applications will react to the shim intercepting the installation attempt. Also, some apps look for the existence of certain files to verify the installation of the desired .Net version. Such app installations may fail if .Net 3.5 was preinstalled because some files previously present on the older versions have been deprecated in Windows 8. We are however testing many frequently used apps, and in some cases introducing pre-app shims if they are detecting .NET inappropriately.

To avoid problems with applications that need it, it is best to enable the new .Net 3.5 feature before installing your app.

Recommendations

As with anything else, a little planning can go a long way. Here are the steps I recommend you consider in order of importance:

A. Understand and evaluate the requirements of your applications. Don't just assume that every computer will require .Net 3.5 and enable it on all of your images. Doing this will defeat the potential benefits of having the payload removed. By the same token, if you deploy Windows 8 with the intention of dealing with the users and computers that need .Net 3.5 as they come up, you are setting yourself and your helpdesk up for a lot of unnecessary hassle. Install all of your common LOB and infrastructure apps one at a time in a test environment without .Net 3.5 and observer whether they attempt to install an older version of the Framework, and in general how they function. Then enable .Net 3.5 and make sure all apps function as expected.

B. Include .Net 3.5 in images that will be deployed to users that will likely be running apps that require it.

C. Copy the files from the installation media to a network share to so they can be referenced by manual installation attempts and optionally by the new Group Policy setting.

D. Regardless of whether your environment includes WSUS servers, treat configuring the new Group Policy setting as a requirement. You should determine how you want to handle attempts to download not only .Net 3.5, but also other Features on demand. Additionally, as the name implies, the policy also tells Windows where to obtain files to repair the feature store.

E. Create and maintain a Windows Image that can be referenced by the Group Policy setting for ongoing FoD and feature store repair downloads.

F. Experiment with the DISM Powershell cmdlets to be prepared to deploy Features on Demand remotely to machines as needed.

Misc. Considerations

Upgrading from Windows 7 / Windows Server 2008 R2

When a PC running Windows 7 (which includes .NET Framework 3.5.1 by default) is upgraded to Windows 8, or when a server running Windows Server 2008 R2 (which has .NET Framework 3.5.1 feature installed) is upgraded to Windows Server 2012, .NET Framework 3.5 is enabled automatically using the files on the installation media. The purpose of this is to increase the chances that all apps will work as they did before the upgrade.

Multilingual images

For images that will support more than one language, it is best to add the .NET Framework 3.5 binaries before adding any language packs. This order ensures that .NET Framework 3.5 language resources are installed correctly within the reference image and available to users and applications.

Common Problems

Here, I've included three issues that you might run into when installing .Net 3.5. The hex error code is the most recognizable part and uniquely identifies each error. I've also included variations on text that might accompany the error code to increase the discoverability of this post. The text will vary, depending on whether it occurs on Windows 8 or Windows Server 2012, whether you tried to install it using the UI, DISM, or PowerShell, and on the build (Developer Preview, Consumer Preview, Release Preview, etc.):

0x800F0906 - CBS_E_DOWNLOAD_FAILURE

Windows couldn't complete the requested changes.

Windows couldn't connect to the Internet to download necessary files. Make sure you are connected to the Internet, and press 'Retry' to try again.

Windows couldn't connect to the Internet to download necessary files. Make sure you're connected to the Internet, and press 'Retry' to try again.

Error code: 0x800F0906

The request to add or remove features on the specified server failed.

Installation of one or more roles, roles services, or features failed.

The source files could not be downloaded.

The source files could not be found.

Use the /source option to specify the location of the files that are required to restore the feature. The file location should be either the root directory of a mounted image or a component store that has the Windows Side-by-Side directory as an immediate subfolder.

Use the "source" option to specify the location of the files that are required to restore the feature. For more information on specifying a source location, see http://go.microsoft.com/fwlink/?LinkId=243077.

Try installing the roles, role services, or features again in a new Add Roles and Features Wizard session, and on the Confirmation page of the wizard, click "Specify an alternate source path" to specify a valid location of the source files that are required for the installation. The location must be accessible by the computer account of the destination server.

This is the most common issue that occurs when attempting to install .Net 3.5. It stems from the machine or user not having access to a configured payload source.

To resolve this issue, make sure the machine and user have access to Windows Update, or that you have configured an alternate source to which the user and machine have access in the Group Policy setting. If machines in your environment are configured to use WSUS, make sure you have configured the Group Policy setting to direct download requests to WU or an alternate source. If it continues to fail, use the DISM command or Powershell cmdlet to install the feature, pointing to a local installation source.

0x800F081F - CBS_E_SOURCE_MISSING

(The text for this message is often similar to that of the 0x800F0906 error above.)

This can occur when trying to install from an installation source that is corrupt, incomplete, or invalid for the installed build (for example trying to use a Windows 8 Developer Preview source for a Consumer Preview or RTM installation). Also, make sure the full path is correct (x:\sources\sxs) and that the user has at least READ access to the location. Try to access the source directly as the installing user from the affected machine.

0x800F0907 - CBS_E_GROUPPOLICY_DISALLOWED

DISM failed. No operation was performed. For more information, review the log file. The DISM log file can be found at %WINDIR%\logs\DISM\dism.log.

Due to network policy settings, Windows couldn't connect to the Internet to download files required to complete the requested changes. Please contact your network administrator for more information.

This might occur if Group Policy has been configured to prevent installing .NET Framework 3.5 from Windows Update. To resolve this, configure an alternate source in the new Group Policy setting.

References

How to Enable or Disable Windows Features

(How to) Configure a Windows Repair Source

Microsoft .Net

.Net Framework Developer Center

.NET Framework 3.5.1 Features Overview

Deployment Image Servicing and Management (DISM) Technical Reference

How to Mount and Modify an Image

How the Windows Update client determines which proxy server to use to connect to the Windows Update Web site

Jim Martin

Senior Support Escalation Engineer

Microsoft Platforms Core Support

EVENT ID 55: When Good Bits Go Bad

Jeff Hughes [MSFT] — Wed, 09 May 2012 14:58:41 GMT

My name is William Effinger, and I am a Senior Support Escalation Engineer with the Windows Core team at Microsoft. Some of the most common questions we get here at the storage team center around CHKDSK. If you have ever come across any event ID 55s in your system event log, this blog is for you.

What is an event ID 55? Let’s start by looking at the error:

Event Type: Error
Event Source: NTFS
Event ID: 55
Description:
The file system structure on disk is corrupt and unusable. Please run the chkdsk utility on the volume "Drive_letter:"

While the error description is direct and self-explanatory, we often receive calls from customers who want to better understand how this occurred and their subsequent options.

How did we get into this situation to begin with?

The file system does not proactively check each write (doing so would hugely impact performance, as you can imagine). NTFS instead has a logic that checks some reads for congruence. As a preventative measure, we can only suggest best practices such as keeping up to date with drivers and firmware.

We do know, however, that the corruption that we see in the file system is due one of two things: either a hardware problem or an issue with the file system driver.

The majority of the issues we see revolve around problems with hardware. As a rule of thumb, hardware tends to corrupt unpredictably.

If you suspect hardware, the following techniques will likely help you identify the culprit:

Install the latest SCSIPORT/RAIDPORT update(s).
Remove or update filter drivers.
Update 3rd party storage drivers /firmware.
Try switching to different type of driver (raidport vs. monolithic).
P2V the machine and try using a different storage method (virtualized SCSI or ATAPI).
If all else fails, rearrange hardware into various combinations.

While extremely rare, NTFS issues can certainly happen. Unlike hardware, NTFS will leave a trail of very specific clues in the corruption leading straight to the offending code. Due to the exceptionally large install base of NTFS, opportunities for corruption have already been identified and resolved. If a software concern still remains, updating the latest version of NTFS would be a prudent course of action.

The only option left for Microsoft support is to look over your server and provide an analysis of the storage stack health. To do that, we look for filter and out of date storage drivers. Then we attempt to eliminate multi-pathing and/or contact the storage vendor suggesting updates as necessary.

Sometimes identifying the root cause of the corruption is less important than resolving the corruption itself. Event ID 55 has alerted you to the fact that there is corruption on the volume, and the only tool capable of resolving the file system corruption is CHKDSK. Unfortunately, customers are unable to use the corrupted volume while CHKDSK is repairing problems with the file system and estimating downtime is impossible. My colleague summed this up in his blog “Questions you shouldn't call Microsoft for...”

If you have any event ID 55s in your system event log, it’s time to run CHKDSK. The longer you wait the worse it is likely to get.

William Effinger
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

MDT 2010 & 2012 – My deployment failed. What and where are logs I should review?

John Marlin [MSFT] — Tue, 08 May 2012 14:01:50 GMT

First thing to note is that the location of the logs move around depending on what portion of setup we are talking about. Here is a breakdown of the log locations during a MDT install task sequence:

Before the Image is applied to the machine:

X:\MININT\SMSOSD\OSDLOGS

After the system drive has been formatted:

C:\MININT\SMSOSD\OSDLOGS

After Deployment:

%WINDIR%\TEMP\DeploymentLogs

The logs of most interest for troubleshooting a failed install will be:

BDD.LOG – This is an aggregated log of all the MDT Logs.

SMSTS.LOG – This log would be used to troubleshoot Task Sequence errors.

Also note that each MDT script creates its own log files during execution (example: ZTIGather.log, ZTIDiskpart.log, ZTIDrivers.log, etc)

Next Question is: How do I read the logs?

Microsoft SCCM 2007 Toolkit
http://www.microsoft.com/download/en/details.aspx?id=9257

Here is an excerpt of a BDD.log in Notepad:

Here is that same portion of that bdd.log viewed in SMS Trace:

Notice warnings are highlighted in yellow, and errors in red.

Now that you know how to locate more detailed error information in the logs, here are some locations that you can use to search to help find solutions to your issues.

The “Ask The Core Team” Blogs on TechNet:
http://blogs.technet.com/b/askcore/

“The Deployment Guys” Blogs on TechNet:
http://blogs.technet.com/b/deploymentguys/

The MDT Social Forums on TechNet:
http://social.technet.microsoft.com/Forums/en/mdt/threads

The Microsoft Deployment Toolkit homepage on TechNet:
http://technet.microsoft.com/en-us/solutionaccelerators/dd407791

The MDT Help Files also have a lot of information and serve as a good reference for finding proper syntax and examples.

Hopefully, this information will help aid your MDT troubleshooting. Being armed with the proper tools and knowing where to look for the answers is the key to troubleshooting success.

Bill Spears
Microsoft Corporation
Senior Support Escalation Engineer / Premier Field Engineer

How to: VSS Tracing

John Marlin [MSFT] — Sun, 29 Apr 2012 16:24:00 GMT

How VSS Tracing Works!

Where to start? Well this depends on your current level of knowledge on the topic. So let’s start with a basic understanding of what VSS is. If you are just starting to work with VSS I suggest reading the articles below.

The key points to focus on in this article are:

Volume Shadow Copy Service Components
Copy-on-Write (Differential Copy)
Hardware-based Providers
Software-based Providers
How Shadow Copies Are Created

Below are a couple of TechNet articles as a reference.

How Volume Shadow Copy Service Works
http://technet.microsoft.com/en-us/library/cc785914(WS.10).aspx

Volume Shadow Copy Service
http://technet.microsoft.com/en-us/library/ee923636.aspx

I like to think about VSS as being camera that takes pictures of your data. Like a picture a VSS snapshot is a point in time view of a volume. The overall concept is rather simple. We hold writes to a disk so we can take a picture of the blocks on that disk. We then release the hold and are left with a view of the data at the time we took the picture. In the case of a client-accessible snapshot we then use COW or Copy-on-Write to make a copy of any blocks that change. This allows you to build a view of your data from the time of the snapshot. Since we have any a copy of the changed blocks we can use them to restore our view of the volume. Backup applications use this same type of process only they use different types of snapshot depending on the type of backup being run.

The Tools

Now that you have a basic understanding of how VSS works we can look at what to do when you run into errors. We have several tools at our disposal. VSSTrace.exe, Logman.exe, and TraceLog.exe are all tools that we can leverage to capture traces. Tracing VSS will provide you with a log file that contains what function was called and the result. From this we can see where VSS is failing and the error being returned.

One of the more common ways to capture a VSS trace is with Vsstrace.exe. Using this tool and a few other tools from the VSS SDK, like Vssagent.exe we can capture additional diagnostics data that helps provide a better understanding of the type of snapshot being ran and details about the error we encountered. VSSAgent collects disk, volume, shadowcopy storage, events log entries and other critical information necessary to provide context to the trace you are capturing.

One of the most important things to know when reviewing a VSS trace is the context. Case in point, are you looking at the actions of a software provider or hardware provider? Or, maybe the trace is simply a command being run in which case we would not be expecting the see the same function calls as we would with a complete snapshot. For example,

vssadmin list writers

Knowing the intention of the requestor at the time of the snapshot is important to understanding what to look for in the trace.

Tracing Options Explained

Let’s spend some time and talk about the options available when capturing tracing of VSS. For this I am going to focus on VSSTRace.exe. VSSTRace.exe allows us to use the various tracing levels and options available to us from the VSS service. For example:

vsstrace.exe -o trace.txt

This will generate a trace file in plain text format. Where this command:

vsstrace.exe -etl trace.etl

This would create an ETL formatted file which you would not be able to read with a text editor.

So let’s say you were having an issue where you wanted to only capture tracing from action performed by the writers. You could run the following command:

vsstrace.exe -f 0 +WRITER

In this command the “-f” would indicate the enabling of flags and the “0” would indicate no modules are going to be traced. Now that we have told tracing to not capture anything we have to add back the stuff we want. That is where the “+WRITER” comes into play. This enables tracing for just the writers.

There is a great chart on MSDN that provides you with all the various modules and flags that can be turned on (http://msdn.microsoft.com/en-us/library/windows/desktop/dd765233(v=vs.85).aspx). To find out more about Vsstrace.exe, please see:

Using VSS Diagnostics
http://msdn.microsoft.com/en-us/library/windows/desktop/aa384631(v=vs.85).aspx

Level is the next thing we should cover. There are 21 different log levels for VSS tracing. The most commonly used is 170. This is the default log level for Vsstrace.exe; however, this can be adjusted by appending to the command line we use to start the trace. Here is the command you would use with VSSTrace.exe to capture all events for only the VSS service components.

vsstrace.exe +f 0xffff -COORD -l 255

This chart has all the log levels listed for you. This can also be found in greater detail on the above mentioned links to MSDN.

Level

Information included in trace output

000

None

020

Fatal errors

030

Unhandled exceptions

040

Errors

050

Assertions

060

Warnings

080

Exception handling

100

Event Log activity

120

General information

140

Code flow

160

Function enter and exit

170

Function return values

180

Function parameters (terse)

190

Function parameters (verbose)

200

Verbose information level 1

210

Verbose information level 2

220

Verbose information level 3

230

Fast Code Level 1

240

Fast Code Level 2

250

Fast Code Level 3

255

All

Taking a Trace

Now it’s time for us to do some tracing. Earlier we talked about the options available to us, now let’s put them practice. We will start with a basic trace that will capture all modules and write the trace data to a plain text file on C:\TEMP.

vsstrace.exe +f 0xffff -o C:\temp\trace.txt

Now say you need a trace of only the Windows built-in in software provider. You would run the following command:

vsstrace.exe -f 0 +SWPRV -o C:\trace.txt

Here we see that the “-f 0” indicates no modules to be traced, and then we add the “+SWPRV” to indicate that we will be only tracing the built-in software provider to a text file located at “C:\Temp\Trace.txt”. This kind of granularity is useful when trying to isolate a particular module.

If you need to capture a greater level of detail within the trace. We can add the “-l” flag to set the level of logging to capture. By default we use 170 from the above chart. Let’s turn that up to the maximum setting of 255. The command would now look like this:

vsstrace.exe -l 255 –o C:\trace.exe

This setting will get you every possible message from all modules. That being said, keep in mind that this comes at the cost of space. The trace file will grow in size quickly depending on the flags set. In contrast to this by limiting logging to specific modules and trace levels you can limit the overall size on lengthy captures.

Reading the Trace

Now that we understand the options and tracing levels and have captured our trace, it’s time to figure out what’s wrong. The best way to do this is to use a text analysis tool like Notepad++ or some other tool with a filtering option. You are going to want to create filters on the following key words.

Failure
Warning
abort
0x8004
0x8007
0x8000f

These key words should help point out where the trace is failing. Please remember that there are all kinds of patterns and failures that will be shown in these logs. Understanding them comes in time so don’t get discouraged if you cannot figure these out easily.

Here is an example of a VSS Trace.

---Trace Snip ---

12402 [0480230921,0x001e34:0x1864:0x3bc9015a] modules\writers\evtlogwriter.cxx(0460): CEventLogWriter::BackupLogs: Failed to backup event log Microsoft-Windows-EventCollector/Operational to C:\WINDOWS\Repair\Backup\ServiceState\EventLogs\Microsoft-Windows-EventCollector/Operational.evt. [0x0000007b]
12403 [0480230937,0x001e34:0x1864:0x3bc9015a] modules\writers\evtlogwriter.cxx(0460): CEventLogWriter::BackupLogs: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x00000000
12404 [0480230937,0x001e34:0x2438:0x3bc9015a] coord\src\async.cxx(0509): CVssAsync::QueryStatus: Returning *pHrResult: 0x00042309
12466 [0480231203,0x001e34:0x1864:0x36bde5db] modules\registry\registry.cxx(1277): CVssDiag::RecordGenericEvent: Event name: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT (SetCurrentState)

---End Snip---

In the example, you can see a failure to access a particular resource during the snapshot process. Let’s break this down. The first line in example 1 shows that VSS was running the CEventLogWriter::BackupLogs: function and its result was:

Failed to backup event log Microsoft-Windows-EventCollector/Operational to C:\WINDOWS\Repair\Backup\ServiceState\EventLogs\Microsoft-Windows-EventCollector/Operational.evt. [0x0000007b]

Looking deeper at the results, we see that the failure code was 0x0000007b.

To see what the this error code is indicative of, you can run the following at an administrative command prompt:

C:\> slui.exe 0x2a 0x0000007b

This will give you a dialog box that gives the description:

You can also use a tool such as Err.exe.

Taking what we just learned about the error code we can look at the path to the object that we were trying to access. Notice that the path is invalid as the path contains a “/”. From this we can now investigate that issue with that path.

Now that we have taken tracing from A-Z, you should be well armed to get out there and collect some traces. Remember it takes time to get the hang of what's going on in a trace.

Good luck and successful backups!

Randy Monteleone
Sr. Support Escalation Engineer
Microsoft

Troubleshooting Boot Issues due to missing Driver signature (x64)

John Marlin [MSFT] — Sun, 15 Apr 2012 14:18:00 GMT

Today, I am going to discuss how to troubleshoot a scenario where the system does not boot in normal mode; however, boots up fine upon selecting F8 -> Disable Driver Signature Enforcement.

In 64-bit operating systems starting with Windows Vista, Windows will load a kernel-mode driver only if the driver is signed. You might get different fatal errors during the boot process depending on the driver that was blocked from loading and how it impacted the further processes. While some of the fatal system errors reference the driver on the blue screen, some may not.

Here is an example of how we can diagnose the problem and fix it. When booting normally, you may receive this error and the machine halts:

This article explains this type of stop error.

Bug Check 0xC000021A: STATUS_SYSTEM_PROCESS_TERMINATED
http://msdn.microsoft.com/en-us/library/windows/hardware/ff560177(v=vs.85).aspx

In decoding the stop error, you should look at the second parameter.

Stop 0XC000021A FATAL SYSTEM ERROR
(0X00000000 0XC0000428 0X00100588) <--- second parameter indicates error code

To translate the hex code, you can use the inbox SLUI.EXE tool or look up the error code from ntstatus.h file in the Windows SDK.

Command Usage:

slui.exe 0x2a 0xC0000428

Restart your computer and start pressing the <F8> key on your keyboard. On a computer that is configured for booting to multiple operating systems, you can press the <F8> key when the Boot Menu appears. On the Advanced Boot Options menu, select Disable Driver Signature Enforcement press <ENTER>.

If the server boots up after selecting this mode, we now know that it is some driver or module which is unsigned or being detected as unsigned that is preventing the system from booting up properly.

Our next task is to find out the module name and devise a remedy. You would need to open Event Viewer and go to Applications and Services Logs -> Microsoft -> Windows -> CodeIntegrity -> Operational.

Note: Check if you can see events like the ones depicted below.
In case you get “access denied” while accessing the “operational” log create a folder on the root of the C drive and give Everyone full rights.
Redirect the ETL file path to the newly created folder. Disable and enable the logging again.

Once logging is enabled, you will find the unsigned driver name from the events.

Remediation...

The driver can be a Microsoft inbox driver or a 3rd party driver. For drivers that have newer versions available and can be updated, that is the first thing to try. In most cases, this will fix the problem.

However, you might encounter a scenario when there is no update available. Here are the steps to find out the required catalog file and place it accordingly.

Files can be copied from another working server with similar file versions and extracted from the update that was the source of the file version.

We can use the SIGCHECK.EXE tool to find the catalog file for a file.

Sigcheck
http://technet.microsoft.com/en-us/sysinternals/bb897441

Step -1 : Verify if the driver is really Unsigned:

Note: The full path of the driver file should be passed as a command line parameter

You can run the command as described in the screenshot above to make sure the driver is indeed being detected as unsigned.

Step – 2 : Get the Signed driver from a working server:

The actual file and the catalog can be copied from a working server and placed in respective locations as depicted above. Once the copy completes, use the SIGCHECK tool to verify and make sure that the file is detected as signed and displays the name of the catalog. Reboot the server and ensure it comes up fine in the normal mode.

Appendix - A

Note:

To find the catalog for the file, you can also enable verbose logging for CodeIntegrity on the working server. This will log the name of the CAT file where the file hash was found during the verification.

Steps to enable CodeIntegrity verbose logging:

i. Open Event Viewer
ii. Select the View drop down menu and choose Show Analytic and Debug Logs from the list
iii. Navigate to Applications and Services Logs -> Microsoft -> Windows -> CodeIntegrity -> Verbose
iv. Right – click and select enable log

More information on the Code Integrity can be found here:

Code Integrity Diagnostic System Log Events
http://msdn.microsoft.com/en-us/library/windows/hardware/ff539911(v=vs.85).aspx

Appendix - B

Note: The SIGNTOOL application can be used to get similar information.

Signtool
http://msdn.microsoft.com/en-us/library/windows/desktop/aa387764(v=vs.85).aspx

Mentioned below is an example of this:

C:\WinDDK\7600.16385.1\bin\amd64>SignTool.exe verify /a /v c:\windows\system32\win32k.sys

Verifying: c:\windows\system32\win32k.sys
File is signed in catalog: C:\Windows\system32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Package_7_for_KB2661001~31bf3856ad364e35~amd64~~6.1.1.0.cat
Hash of file (sha1): DEDF7D339D8355A9875661559BD582FA46008AE5

Signing Certificate Chain:
    Issued to: Microsoft Root Certificate Authority
    Issued by: Microsoft Root Certificate Authority
    Expires:   Mon May 10 04:58:13 2021
    SHA1 hash: CDD4EEAE6000AC7F40C3802C171E30148030C072
        Issued to: Microsoft Windows Verification PCA
        Issued by: Microsoft Root Certificate Authority
        Expires:   Wed Mar 16 03:35:41 2016
        SHA1 hash: 5DF0D7571B0780783960C68B78571FFD7EDAF021
            Issued to: Microsoft Windows
            Issued by: Microsoft Windows Verification PCA
            Expires:   Tue May 15 02:41:44 2012
            SHA1 hash: 5C616DC011E309DFCD15C0EA32494186654A2CDC

The signature is timestamped: Mon Jan 16 15:24:23 2012
Timestamp Verified by:
    Issued to: Microsoft Root Certificate Authority
    Issued by: Microsoft Root Certificate Authority
    Expires:   Mon May 10 04:58:13 2021

SHA1 hash: CDD4EEAE6000AC7F40C3802C171E30148030C072
        Issued to: Microsoft Time-Stamp PCA
        Issued by: Microsoft Root Certificate Authority
        Expires:   Sat Apr 03 18:33:09 2021
        SHA1 hash: 375FCB825C3DC3752A02E34EB70993B4997191EF

            Issued to: Microsoft Time-Stamp Service
            Issued by: Microsoft Time-Stamp PCA
            Expires:   Fri Oct 26 02:12:17 2012
            SHA1 hash: FC33104FAE31FB538749D5F2D17FA0ECB819EAE5

Successfully verified: c:\windows\system32\win32k.sys
Number of files successfully Verified: 1
Number of warnings: 0
Number of errors: 0

Below are articles you should review that discuss more details about this subject.

Driver Signing Requirements for Windows
http://msdn.microsoft.com/en-us/windows/hardware/gg487317

Driver Signing Guidelines for ISVs
http://msdn.microsoft.com/en-us/windows/hardware/gg463036

Kernel-Mode Code Signing Walkthrough
http://msdn.microsoft.com/en-us/windows/hardware/gg487328

Installing an Unsigned Driver during Development and Test
http://msdn.microsoft.com/en-us/library/windows/hardware/ff547565(v=vs.85).aspx

Hope this helps and until next time, Take care.

Parthiv Seth
Support Escalation Engineer
Microsoft Windows Server Core Team

Why is the CNO in a Failed State?

Jeff Hughes [MSFT] — Tue, 27 Mar 2012 17:15:25 GMT

Hello, my name is Steven Graves and I am a Support Escalation Engineer (SEE) in the Platforms Support group here at Microsoft. One of the technologies I support is Windows Server Failover Clustering. I’d like to take a couple of minutes to share some information on an issue I previously worked on. The customer wanted to create an Exchange 2010 DAG, which would be the first Windows Server 2008 R2 cluster in their environment and they were having issues bringing the CNO online after the cluster was created. The customer domain was originally 2003 and they had to add a 2008 R2 DC and update the schema in order to install Exchange 2010 DAG.

For starters, since I knew the CNO was not coming online after creating the cluster I had the customer destroy the cluster, pre-staged a new computer object for the CNO then created a new cluster based on the name of the new CNO. After the cluster was created I noticed that the computer object was still disabled in AD and the following error message in the cluster log.

00000e80.00000d5c::2012/03/14-16:07:33.149 INFO [RES] Network Name <Cluster Name>: Trying to find computer account W2K8R2Cluster object GUID(cae8b3dcc60aa040bbcef250634427bb) on any available domain controller.
00000e80.00000d5c::2012/03/14-16:07:33.306 WARN [RES] Network Name <Cluster Name>: Search for existing computer account failed. status 8007052E
00000e80.00000d5c::2012/03/14-16:07:33.352 WARN [RES] Network Name <Cluster Name>: Couldn't get information from DC \\Info-dc3.infoimage.com. status 5
00000e80.00000d5c::2012/03/14-16:07:33.352 INFO [RES] Network Name <Cluster Name>: Trying to find object cae8b3dcc60aa040bbcef250634427bb on a PDC.
00000e80.00000d5c::2012/03/14-16:07:33.462 WARN [RES] Network Name <Cluster Name>: Couldn't get information about PDC. status 5
00000e80.00000d5c::2012/03/14-16:07:33.462 INFO [RES] Network Name <Cluster Name>: Unable to find object cae8b3dcc60aa040bbcef250634427bb on a PDC.
00000e80.00000d5c::2012/03/14-16:07:33.462 INFO [RES] Network Name <Cluster Name>: GetComputerObjectViaGUIDEx() failed, Status 8007052E.

‘Access is denied’

In the System Event log you will see an ID 1207 that should be in synch with the time in the cluster log. The main thing to focus on is the “Unable to get the Computer Object using GUID”.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 3/14/2012 9:07:10 AM
Event ID: 1207
Task Category: Network Name Resource
Level: Error
Keywords:
User: SYSTEM
Computer: W2K8R2Cluster.Corp.com
Description:
Cluster network name resource 'Cluster Name' cannot be brought online. The computer object associated with the resource could not be updated in domain 'infoimage.com' for the following reason:
Unable to get Computer Object using GUID.
The text for the associated error code is: Logon failure: unknown user name or bad password.

At this point, I’m pretty convinced there are some issues with the GPOs on the domain controllers but I still need to do my due diligence in troubleshooting the issue with the Cluster Network Name in a failed state.

Since I pre-staged the CNO and it was still disabled after creating a new cluster, this gave me more evidence indicating an issue with the DC. I created a new OU and blocked inheritance in order to prevent any GPOs from being applied to the Node(s). I refreshed the GPO’s on the Node(s), confirmed there are no GPOs applied by running Gpresult /V from an Administrative CMD Prompt, but the Cluster Network Name still fails to come online. I’m convinced there is some issue with GPO’s on the DC but I’m not sure where to start looking.

Next, I verified the permissions in AD on the CNO and, to be on the safe, I granted the CNO Full Control to the object and also confirmed that the CNO has the correct permissions to the OU(READ permissions on the OU should be sufficient rights to access the OU and get to the computer object). Despite this, the Cluster Network Name failed to come online.

I moved on to check the DNS Host A record for the CNO not really thinking this is the issue but more or less making sure everything is in order. I came to find out a Host A record was not created for the Cluster Network Name because they do not have Dynamic Updates enable for DNS. I created the Host A record and checked off “Allow any authenticated user to update the DNS records with the same owner name.” I already knew the node was able to resolve the DC from the warnings in the cluster log but couldn’t get information from DC \\W2K8R2-DC.Corp.com. So it was not a name resolution issue trying to access the DC.

At this point, I have gone through all the normal troubleshooting steps that generally resolve the ID 1207 and the CNO in a failed state from the cluster perspective. Now it’s time to engage Directory Services to take a deeper look at the DC configuration. After some time reviewing the Domain Controller configuration and GPOs the DS engineer narrowed it down to permission issues in the “Access this computer from the network” policy. The default permissions are pictured below.

Access this computer from the network - This user right determines which users and groups are allowed to connect to the computer over the network. Since "Everyone" and "Authenticated Users" were missing from the settings, this meant that no computer would be able to access the domain controller.

Picture above shows the default permissions for the Access this computer from the network policy

The DS engineer modified the “Access this computer from the network” policy in the Default Domain Controllers policy by adding Authenticated Users, refreshed GPOs by running GPUpdate /force, ran RSOP.msc to confirm the GPO is applied, and the CNO came online.

Steven Graves
Support Escalation Engineer
Microsoft Enterprise Platforms Support

FIXED: Cluster Shared Volumes (CSV) in redirected access mode after installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1

John Marlin [MSFT] — Sun, 18 Mar 2012 17:47:00 GMT

A fix for the above titled problem has been released. If you are running into this problem, please downlaod and install the following fix on all Clusters running McAfee and wanting the updates they provide.

2674551
Redirected mode is enabled unexpectedly in a Cluster Shared Volume when you are running a third-party application in a Windows Server 2008 R2-based cluster
http://support.microsoft.com/default.aspx?scid=kb;EN-US;2674551

===============

Below is the information from the original post:

There is an issue with Cluster Shared Volumes and McAfee VirusScan Enterprise that I wanted to pass along. When installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1, the CSV drives will go into redirected mode and will not go out of it.

The reason for this is that the McAfee filter driver (mfehidk.sys) is using decimal points in the altitude to help in identifying upgrade scenarios for their product. The Cluster CSV filter only accepts whole numbers and puts the drives in redirected access mode when it sees this decimal value.

When seeing this, if you run FLTMC from an administrative command prompt, you may see something similar too:

C:\> fltmc

Filter Name   Num Instances   Altitude   Frame
------------------------------------------------------
CSVFilter          2          404900       0
mfehidk                       329998.99 <Legacy>
mfehidk            2          321300.00    0

If you were to generate a Cluster Log, you would see the below identifying that it cannot read the altitude value properly.

INFO [DCM] FsFilterCanUseDirectIO is called for \\?\Volume{188c44f1-9cd0-11df-926b-a4ca2baf36ff}\
ERR mscs::FilterSnooper::CanUseDirectIO: BadFormat(5917)' because of 'non-digit found'
INFO [DCM] PostOnline. CanUseDirectIO for C2V1 => false

McAfee has released the following document giving a temporary workaround.

Cluster Shared Volumes (CSV) status becomes Online (Redirected access)
https://kc.mcafee.com/corporate/index?page=content&id=KB73596

Microsoft is aware of the problem and currently working on a fix. When this fix is available, this will be updated and a new KB Article will be created with the fix.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Windows Performance Monitor Disk Counters Explained

Jeff Hughes [MSFT] — Fri, 16 Mar 2012 14:53:15 GMT

My name is Flavio Muratore and I am a Sr. Support Escalation Engineer with the Windows Core team at Microsoft. If you ever find yourself analyzing storage performance with Performance Monitor, this post is for you. We will go beyond very brief descriptions provided in Perfmon and describe how we calculate the data for the Physical and Logical disk counters.

Why the Performance Monitor?
When it comes to the subject of disk performance in Windows, the majority of questions can be quickly answered by Performance Monitor alone. Performance Monitor is very low overhead, does a great job with averages and can also capture and store data over long periods of time. It is an excellent choice to record a performance baseline and to troubleshoot.
For short in this text, we are going to call the Windows Performance Monitor by its nickname: Perfmon. The nickname comes from its executable file located at %systemroot%\system32\Perfmon.exe.

There are some things Perfmon will not be able to tell us. For advanced analysis, Windows provides us with xPerf, enabling state of the art performance data capture through Event Tracing for Windows (ETW). There is an excellent bog on the subject by Robert Smith (Sr. PFE/SDE). “Analyzing Storage Performance using the Windows Performance Analysis ToolKit (WPT)”.

What is the difference between the Physical Disk vs. Logical Disk performance objects in Perfmon?

Perfmon has two objects directly related to disk performance, namely Physical Disk and Logical Disk. Their counters are calculated in the same way but their scope is different.

The Physical Disk performance object monitors disk drives on the computer. It identifies the instances representing the physical hardware, and the counters are the sum of the access to all partitions on the physical instance.

The Logical Disk Performance object monitors logical partitions. Performance monitor identifies logical disks by their drive letter or mount point. If a physical disk contains multiple partitions, this counter will report the values just for the partition selected and not for the entire disk. On the other hand, when using Dynamic Disks the logical volumes may span more than one physical disk, in this scenario the counter values will include the access to the logical disk in all the physical disks it spans.

Disk Counters Explained.

%Disk Time (% Disk Read Time, % Disk Write Time)
The “% Disk Time” counter is nothing more than the “Avg. Disk Queue Length” counter multiplied by 100. It is the same value displayed in a different scale.
If the Avg. Disk queue length is equal to 1, the %Disk Time will equal 100. If the Avg. Disk Queue Length is 0.37, then the %Disk Time will be 37.
This is the reason why you can see the % Disk Time being greater than 100%, all it takes is the Avg. Disk Queue length value being greater than 1.
The same logic applies to the % Disk Read Time and the % Disk Write Time. Their data comes from the Avg. Disk Read Queue Length and Avg. Disk Write Queue Length, respectively.

Avg. Disk Queue Length (Avg. Disks Read Queue Length, Avg. Disk Write Queue Length)
Avg. Disk Queue Length is equal to the (Disk Transfers/sec) *( Disk sec/Transfer). This is based on “Little’s Law” from the mathematical theory of queues. It is important to note this is a derived value and not a direct measurement, I recommend reading this article from Mark Friedman, the information still applies to Windows 2008 R2.
As you would expect, the Avg. Disk Read Queue Length is equal to the “(Disk Reads/sec) * (Disk sec/Read)” and Avg. Disk Write Queue Length is equal to the “(Disk Writes/sec) * (Disk sec/Write)”.

Current Disk Queue Length
Current Disk Queue Length is a direct measurement of the disk queue present at the time of the sampling.

% Idle Time
This counter provides a very precise measurement of how much time the disk remained in idle state, meaning all the requests from the operating system to the disk have been completed and there is zero pending requests.
This is how it’s calculated, the system timestamps an event when the disk goes idle, then timestamps another event when the disk receives a new request. At the end of the capture interval, we calculate the percentage of the time spent in idle. This counter ranges from 100 (meaning always Idle) to 0 (meaning always busy).

Disk Transfers/sec (Disk Reads/sec, Disk Writes/sec)
Perfmon captures the total number of individual disk IO requests completed over a period of one second. If the Perfmon capture interval is set for anything greater than one second, the average of the values captured is presented.
Disk Reads/sec and Disk Writes/sec are calculated in the same way, but break down the results in read requests only or write requests only, respectively.

Disk Bytes/sec (Disk Read Bytes/sec, Disk Write Bytes/sec)
Perfmon captures the total number of bytes sent to the disk (write) and retrieved from the disk (read) over a period of one second. If the Perfmon capture interval is set for anything greater than one second, the average of the values captured is presented.
The Disk Read Bytes/sec and the Disk Write Bytes/sec counters break down the results displaying only read bytes or only write bytes, respectively.

Avg. Disk Bytes/Transfer (Avg. Disk Bytes/Read, Avg. Disk Bytes/Write)
Displays the average size of the individual disk requests (IO size) in bytes, for the capture interval. Example: If the system had ninety nine IO requests of 8K and one IO request of 2048K, the average will be 28.4K. Calculation = (8k*99) + (1*2048k) / 100
The Avg. Disk Bytes/Read and Avg. Disk Bytes/Write counters break down the results showing the average size for only read requests or only write requests, respectively.

Avg. Disk sec/Transfer (Avg. Disk sec/Read, Avg. Disk sec/Write)
Displays the average time the disk transfers took to complete, in seconds. Although the scale is seconds, the counter has millisecond precision, meaning a value of 0.004 indicates the average time for disk transfers to complete was 4 milliseconds.
This is the counter in Perfmon used to measure IO latency.
I wrote a blog specifically about measuring latency with Perfmon. For details got to “Measuring Disk Latency with Windows Performance Monitor”.

Split IO/Sec
Measures the rate of IO split due to file fragmentation. This happens if the IO request touches data on non-contiguous file segments. For an explanation about file segments see this blog from Robert Mitchell - The Four Stages of NTFS File Growth.

Logical Disk Counters Exclusive Counters
The Logical Disk performance object has all the same counters as the physical disk, and except for the fact they are reported per logical unit instead of physical device, they are calculated in the same way.
Because the physical disk counter does not understand volumes, the following counters are exclusive to the Logical Disk Object.

% Free Space
Display the percentage of the total usable space on the selected logical disk that was free.

Free Megabytes
Displays the unallocated space, in megabytes, on the volume.
How can we quickly tell how much free space is available in the volume? Check this blog from Robert Mitchell – NTFS Metafiles.

A few words about performance monitor counters averaging and rounding:
Perfmon is really good at averaging results and rounding numbers, this enables us to have relatively small log files and extract useful the information from the data captured. Although the numbers displayed to the user during a live capture and the numbers saved in the log files are rounded, the numbers used in the internal calculations are more precise.
When reading the description for some counters in this blog, you probably noticed Perfmon has to calculate an average of averages, this leads to small imprecisions on the final numbers. Also, when we combine this with instances that do further rounding and averaging, like the “ _Total instance”, you will see some results are close but do not add up exactly. For example, if you get the “Disk Transfers/sec” over a period of time and subtract both the “Disk Reads/sec” and the “Disk Writes/sec” the resulting number may not be exactly zero.

This is expected and does not pose a problem to the performance analysis at this level. If you can’t tolerate these small imprecisions you will need to use xPerf. xPerf does event tracing and all data is kept with no averaging or rounding. The downside is the resulting log files with xPerf are much bigger than the ones Perfmon creates.

Conclusion:
The Windows Performance Monitor is a very powerful diagnostic tool and is capable of answering most questions about the state of disks on the fly. Perfmon uses averaging and rounding to keep only meaningful data in its log files, thus allowing captures over a long period of time.

I must thank a bunch of Microsoft fellows for helping me with this blog. Big thanks to Bruce Worthington (Principal Development Lead), without your knowledge I would not be able to finish this blog. Thanks also to Mark Licata (Principal SE), Robert Smith (Sr. PFE), Clint Huffman (Sr. PFE), John Rodriguez (Principal PFE), Steven Andress (Sr. SEE) and the Storage performance discussion group at Microsoft. It seems so simple now, but it took a lot of sweat to get the exact data to make sure this information is accurate.

Y’all have fun with Perfmon.

Flavio Muratore
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

How to manage Out-of-Box Drivers with the use of Selection Profiles in Microsoft Deployment Toolkit 2010 Update 1

Jeff Hughes [MSFT] — Wed, 07 Mar 2012 18:38:51 GMT

Hello. My name is Bill Spears and I am a Senior Support Escalation Engineer / Premier Field Engineer in the Windows/Platforms group at Microsoft. In today’s blog I will discuss the approach that I use to manage Out-Of-Box drivers within the deployment process of MDT (Microsoft Deployment Toolkit 2010 Update 1).

If you are used to deployments of legacy Operating Systems (such as Windows XP), Out-of-Box driver management sometimes became a very confusing task, and many folks ended up having very long driver paths in their answer files (OEMPnPDriversPath) and ended up having a large folder structure of drivers on each machine that contained drivers for any and all hardware that was to be deployed in that environment and in turn, this was a nightmare to manage. The good news is that managing your Out-of-Box drivers is now much easier and cleaner when deploying Windows 7 via Microsoft Deployment Toolkit 2010 Update 1. Now you can import all of the Out-of-Box drivers that will be needed into your MDT Deployment Workbench, and then those drivers can be injected offline into your install WIM depending on which hardware you are deploying the image to.

Driver injection works as follows: When a machine is booted via the Litetouch media, PNPEnum.exe collects a hardware inventory of the devices in that machine to determine which drivers will need to be injected in to the image before the image is installed on the target machine. By default the inject driver step in the task sequence will query for driver matches in the “All Drivers” Selection Profile. So by default, you could get away with dumping all of your drivers in to the Out-Of-Box Drivers node of MDT and have most deployment scenarios install with the correct driver. In reality, this is a horrible way to approach driver management. The reason I say that is because sometimes you will have multiple drivers that state via their INF file that they will work for a particular device (PNPID), when in reality, that’s not always true. This could be the result of a poorly written driver, or it could be that even though you have a driver that is a match based on the INF, you may need to force the use of a different version of that driver for a particular model of hardware. For Best Practice, it makes more sense to create a folder structure under the Out-of-Box Driver store to better manage how you add drivers to your MDT Workbench. I prefer to create a subfolder for Each Operating System, then each architecture type (x86,x64), then each hardware model, as shown in the screen shot below:

In the screenshot above, notice the folder structure created under the Out-of-Box Drivers Node. Import the drivers for each specific model into the respective folder.

This also makes it easier when it comes time to add updated drivers for existing hardware types, or add a new folder for a new hardware type when you start getting newer model machines. Having this well organized folder structure now also lets you be very granular of what drivers you make available during your deployment by making use of Selection Profiles. Consider the scenario of where PNPEnum.exe detects a piece of hardware with a certain PNPID, but where we have multiple drivers in our Out-of-Box drivers node that claim to be a match for this hardware based on the driver’s INF file information. Without the use of Selection Profiles or DriverGroups, you would not be able to force which driver gets installed. The driver who wins the built in driver ranking process would end up getting installed. By using Selection Profiles, you can force the Driver Injection step of the task sequence to only look in a specific folder for its choice of drivers. Therefore you are in control of what specific drivers are available to what specific machines. This is possible by adding multiple Inject Drivers steps to your task sequence, and then have each one using a specific Selection Profile, based on what hardware we are installing to. We would then control which Inject Driver task sequence step actually gets used by adding conditional statements to each Inject Drivers task sequence properties and we can use built in the built in Task Sequence Variable %model% to control this. Below are screenshots of this configuration:

In the screenshot above, notice I have created a selection profile correlating to each driver folder in my Out-of-Box Drivers node.

In the screenshot above, notice in the properties of a Selection Profile, I have only selected the folder where I have imported the specific drivers for that model.

Now that we have our drivers imported into our folder structure in the Out-of-Box Drivers node, and we also have our Selection Profiles created for each model that we will be deploying to, it’s time to configure the Inject Drivers steps in our task sequence. The approach for this will be to have multiple Inject Drivers steps in our task sequence, each one pointing to a specific Selection Profile for a particular hardware model. We can then use conditional statements and task sequence variables to control which Inject Drivers step actually gets executed. This is outlined in the screenshots below:

First, to find the model name of your machine, you could use one of the following methods:

In the screenshot above, notice I’m running the command: wmic computersystem get model to obtain the model name. Alternatively, you could obtain this info via MSinfo32 as shown below:

The screenshot above is a portion of the output from the MSInfo32 command. Notice the System Model name. Next we will configure the task sequence.

In the screenshot above, notice that I have 4 Inject Drivers steps in my task sequence, because I plan to deploy this image to 4 different hardware models. Notice that I’m choosing to “Install only matching drivers from the selection profile” and I’m pointing to the specific Selection Profile that was created in the previous step.

In the screenshot above, we are now looking at the Options tab of the first Inject Drivers step of our Task Sequence. I am using a conditional statement, “If all conditions are true” and then using one of the built in task sequence variables that is collected during the gather phase to add the logic that if %model% equals the hardware model specified, then it will run this step. You would then do the same for each of your Inject Drivers steps and have each one pointing to a specific Selection Profile and have the conditional statement control if this step will actually execute or not based on the value of the %model% variable. Now you can rest assured that when you run this task sequence on a particular model machine, you know exactly where it will be looking to find the drivers based on what hardware is detected in that machine.

If new drivers become available from the OEM for a particular model, you would simply need to replace the new drivers in the proper folder of your Out-of-Box Drivers node. If you will be deploying this task sequence to a new piece of hardware, you would simply create a new folder in your Out-of-Box Drivers node, then import the new drivers into this folder, then create a selection profile pointing to the new folder, then add an additional Inject Drivers step to your task sequence and then you will be ready to deploy to the new hardware.

One thing to note about Driver injections is that the driver must have an INF and SYS file in order for us to install the driver this way. If a driver installs via an EXE and there is no way to extract that driver to reveal the actual INF and SYS file, then you would be forced to add that driver EXE package as an application and install it as an application. Note that in that situation, you would need to handle the Network Adapter Drivers and Mass Storage Device drivers differently to ensure that the Operating System could communicate with the storage device and the network adapter in order to complete the install, and then the Vendor’s EXE program could be launched as an application to install the other drivers.

Note that there are multiple approaches to handle Out-of-Box Driver Management with your deployments, and this is just one approach and it is the approach that works best for me. Another approach would be to utilize task sequence variables in your CustomSettings.ini for DriverSelectionProfile and DriverGroup00x as explained in the following blog:
http://blogs.technet.com/b/mniehaus/archive/2009/09/09/mdt-2010-new-feature-19-improved-driver-management.aspx
Which method you use will be dependent on what works best for you in your particular situation, as there is not always a one size fits all solution to the design process of building and managing your images. The key benefit I find from using this method is that you know exactly where your drivers are coming from and once you have the framework setup, it’s easy to add updated drivers for existing models, and easy to add new models to your deployments, and most of all, it is very organized.

Another thing to note is that when using this procedure to deploy Lenovo machines, other special considerations may need to be made. This is due to the fact the Lenovo reports back a model string that frequently changes. This is explained further in the following blog which also has a recommended solution for dealing with this scenario:
http://blogs.technet.com/b/mniehaus/archive/2009/07/17/querying-the-mdt-database-using-a-custom-model.aspx

I hope that this information is helpful for your design strategy of how to manage Out-of-Box drivers.

Bill Spears
Microsoft Corporation
Senior Support Escalation Engineer / Premier Field Engineer

Understanding the DiskRunChkdsk parameter in Windows 2008 and 2008R2 Failover Clusters

Jeff Hughes [MSFT] — Mon, 05 Mar 2012 16:49:00 GMT

My name is Sean Dwyer and I am a Support Escalation Engineer with the Microsoft CORE team.

I’d like to share a quick tip for handling Windows Server Cluster administrators.

There may come a time, for whatever reason, that a Cluster managed volume is flagged as dirty and you will see an event ID message indicating that CHKDSK needs to run against the volume. Just for a little background, the NTFS File System is monitoring the drive/partition at all times. If it detects corruption, it will flip a bit on the volume and mark it as dirty. During the online process of a Clustered drive, it will check for the existance of this bit and spawn CHKDSK if it sees it. You can check, at any time, to see if a volume it is dirty with the CHKNTFS command.

C:\> chkntfs z:

The type of the file system is NTFS.

Z: is not dirty.

C:\> chkntfs z:

The type of the file system is NTFS.

Z: is dirty.

In a best case scenario, you can take the volume out of production, run CHKDSK on the volume if needed (refer to: http://technet.microsoft.com/en-us/library/cc772587.aspx, and then put the volume back into production.

In most situations though, the volume that needs attention is a heavily utilized production volume and will be extremely disruptive to have the volume offline for any length of time.

For example, a recent case I was involved with had a 14Tb* (see note 1 below) volume that was being flagged for CHKDSK to run on it about once a month. The volume had about 9tb of data on it. Apart from the concern of why the volume was continually being flagged as corrupt, the length of time that CHKDSK took to run on the volume was extremely painful for the customer’s business. When it ran initially, it took roughly 80 hours to complete a run on the volume.

It may be necessary to temporarily configure a problem volume to block CHKDSK from running against it while troubleshooting continues to determine why the volume is being flagged for CHKDSK to run.

I stress the word temporary here.

Turning off the health monitoring tool for the file system as a permanent solution could only lead to more downtime in the future. You may end up on the phone with one of the File Systems experts on my team, such as Robert Mitchell.

Ok – so let’s talk specifics about temporarily blocking CHKDSK from doing work on a Cluster volume.

Say we have determined that we need to suspend CHKDSK from running on a problem volume. For you old school Cluster admins, the first command parameter that probably jumps to mind is SKIPCHKDSK.

This works just fine for Windows 2003 Server Clusters, but will NOT work for Windows 2008 and 2008R2 Failover Clusters.

If SKIPCHKDSK is used for a Clustered volume, it will be ignored when the disk is next brought online and CHKDSK will be run. In a situation where the volume is 18tb, the volume will remain unavailable for use until CHKDSK finishes* (See note 2 below).

The correct way to configure a volume to block CHKDSK from running on it, is to use the DiskRunChkdsk parameter. Keep in mind that these two parameters we are discussing only apply to the Cluster environment. If the machine is restarted, the OS may prompt for CHKDSK to run on the affected volumes.

For information on how to configure the OS to ignore the dirty bit, refer to:

KB158675

How to Cancel CHKDSK After It Has Been Scheduled

http://support.microsoft.com/default.aspx?scid=kb;EN-US;158675

Before walking through an example of setting the DiskRunChkdsk parameter, I first must expain what the values mean. In Windows 2003 Server Clusters, the SKIPCHKDSK parameter was either 0x0 (disabled) or 0x1 (enabled). In Windows 2008 and 2008R2 Failover Clusters, there are different settings and what it is checking varies.

DiskRunChkDsk <0x0>: This is the default setting for all Failover Clusters. This policy will check the volume to see if the dirty bit is set and it will perform a Normal check of the file system. The Normal check is similar to running the DIR command at the root. If the dirty bit is set or if the Normal check returns a STATUS_FILE_CORRUPT_ERROR or STATUS_DISK_CORRUPT_ERROR, CHKDSK with be started in Verbose mode (Chkdsk /x /f).

DiskRunChkDsk <0x1>: This setting will check the volume to see if the dirty bit is set and it will perform a Verbose check. A verbose check will scan the volume by traversing from the volume root and checking all the files) of the file system. If the dirty bit is set or if the Verbose check returns a STATUS_FILE_CORRUPT_ERROR, CHKDSK with be started in normal mode (Chkdsk /x /f).

DiskRunChkDsk <0x2>: This setting will run CHKDSK in Verbose mode (Chkdsk /x /f) on the volume every time it is mounted.

DiskRunChkDsk <0x3>: This setting will check the volume to see if the dirty bit is set and it will perform a Normal check of the file system. The Normal check is similar to running the DIR command at the root. If the dirty bit is set or if the Normal check returns a STATUS_DISK_CORRUPT_ERROR, CHKDSK will be started in Verbose mode (Chkdsk /x /f), otherwise CHKDSK will be started in read only mode (Chkdsk without any switches).

DiskRunChkDsk <0x4>: This setting doesn’t perform any checks at all.

DiskRunChkDsk <0x5>: This setting will check the volume to see if the dirty bit is set and it will perform a Verbose check (scan the volume by traversing from the volume root and checking all the files) of the file system. If a problem is found, CHKDSK will not be started and the volume will not be brought online.

So now that we know what the varies switches do, to have CHKDSK never run during an online operation of the disk, we want to set DiskRunChkdsk to 0x4.

Here are the steps you can run through to accomplish this task.

Step 1: Determine the resource name as seen by Cluster

Step 2: Open either an Administrative command prompt or Windows Powershell Modules and run the command:

C:\> cluster res "Cluster Disk 8" /priv DiskRunChkdsk=4

PS C:\> Get-ClusterResource "Cluster Disk 8" | Set-ClusterParameter DiskRunChkdsk 4

Note: For the setting to WORK, the disk must be brought offline and back online. Otherwise, it is simply stored until the next time it is taken offline and back online.

Step 4: Bring the disk offline, then online again.

Step 5: Verify the setting is applied

PS C:\> Get-ClusterResource "Cluster Disk 8" | Get-ClusterParameter DiskRunChkdsk

Object Name Value

------ ---- -----

Cluster Disk 8 DiskRunChkDsk 4

Step 6: Actively start troubleshooting what could cause the volume to end up flagged dirty and needing CHKDSK.

Footnotes:

Note 1: It’s not suggested to run with volumes this large. In my experience once they exceed 2tb in size, they rapidly become an administrative liability, especially in a situation where CHKDSK has to run against the volume. We strongly suggest that mount points be used to carve up larger volumes like this, into more administratively friendly chunks. CHKDSK runs against mount points just fine, too.

Note 2: While it’s not recommended to interrupt CHKDSK while it’s running, an admin is not locked into having to let CHKDSK finish once it starts. The process can be terminated if absolutely required. However, we cannot guarantee that the end result will be positive. If the process is interrupted during the “magic moment” when CHKDSK is making changes, the results may be worse than the initial reason for the volume being flagged as corrupt.

Additional reading material related to the components and tools mentioned in this post:

KB947021

How to configure volume mount points on a server cluster in Windows Server 2008

http://support.microsoft.com/default.aspx?scid=kb;EN-US;947021

The shared disk on Windows Server 2008 cluster fails to come online

http://support.microsoft.com/default.aspx?scid=kb;en-US;2517696

FSUTIL utility; marking a volume dirty for testing

http://technet.microsoft.com/en-us/library/bb490641.aspx

In summary; try to keep your production volumes’ size under control, be aware that command line switches may not persist through all versions of a product, and continue being successful with Windows Server 2008!

I hope this post has been helpful!

Sean Dwyer

Support Escalation Engineer

Windows CORE Team

Windows Server 8 beta now available !!!

John Marlin [MSFT] — Sun, 04 Mar 2012 20:27:28 GMT

Thursday morning Bill Laing, Corporate Vice President of Server and Cloud Announced the beta release of Windows Server “8”. I think it is really important for anyone who is interested in hearing more about what is coming in this beta with Hyper-V and our Virtualization development efforts; please go read what Bill has to say.

In the coming days and months, there will be more Windows Server “8” blog posts coming from Bill Laing and his leadership team on the Windows Server Blog.

Also, stay tuned in the coming days and weeks for more deeper dives in the supporting technologies here. I have been waiting to start talking, so hope you have some time carved out to learn more about Windows Server “8” technologies.

Download Windows Server “8” Beta here

These are exciting times here and hope that everyone can download and try out the Windows 8 Server (beta) and see what everyone is talking about.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Windows 8 Consumer Preview now available !!!

John Marlin [MSFT] — Wed, 29 Feb 2012 17:16:44 GMT

Today is a big day here at Microsoft. This morning at the Mobile World Congress in Barcelona, Spain, it was announced that Windows 8 Consumer Preview (beta) is now released.

At the conference, several key points were discussed and demonstrated.

Broad range of product changes and improvements that were made between the Developer Preview and the Consumer Preview.
Windows Store with new apps. The Windows 8 Consumer Preview marks the beta opening of the Windows Store, which is filled with a variety of new Metro style apps from both third-party developers and Microsoft.
Connecting to the cloud across Windows-based PCs and Windows Phone. The Windows 8 Consumer Preview offers seamless integration with the content people care about across their devices.
Internet Explorer 10 Platform Preview 5. The best way to experience the Web on Windows is with Internet Explorer 10.
Preview of new hardware capabilities. Showcasing Windows 8 running on a wide range of new x86 and ARM-based reference hardware.

Come preview the latest version and where Windows is headed. Get involved to help shape what Windows will look like in the future.

Windows 8 Consumer Preview
http://windows.microsoft.com/en-US/windows-8/consumer-preview

Welcome to Windows 8 – The Consumer Preview
http://blogs.msdn.com/b/b8/archive/2012/02/29/welcome-to-windows-8-the-consumer-preview.aspx?ocid=soc-n-mea-loc--ZaFb

Running the Consumer Preview: system recommendations
http://blogs.msdn.com/b/b8/archive/2012/02/29/running-the-consumer-preview-system-recommendations.aspx

Windows 8 Consumer Preview Product Guide for Business
http://www.microsoft.com/download/en/details.aspx?id=28970

If you want to see what others are experiencing or get involved in the discussions on Technet, this is your place to go.

http://social.technet.microsoft.com/Forums/en-US/category/w8itpro

This is an exciting day here and hope that everyone can download and try out the Windows 8 Consumer Preview (beta) and see what everyone is talking about.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Having a problem with nodes being removed from active Failover Cluster membership?

John Marlin [MSFT] — Wed, 08 Feb 2012 15:40:00 GMT

Welcome to the AskCore blog. Today, we are going to talk about nodes being removed from active Failover Cluster membership randomly. If you are having problems with a node being removed from membership, you are seeing events like this logged in your System Event Log:

This event will be logged on all nodes in the Cluster except for the node that was removed. The reason for this event is because one of the nodes in the Cluster marked that node as down. It then notifies all of the other nodes of the event. When the nodes are notified, they discontinue and tear down their heartbeat connections to the downed node.

What caused the node to be marked down?

All nodes in a Windows 2008 or 2008 R2 Failover Cluster talk to each other over the networks that are set to Allow cluster network communication on this network. The nodes will send out heartbeat packets across these networks to all of the other nodes. These packets are supposed to be received by the other nodes and then a response is sent back. The responding node then sends out a heartbeat request of its own and waits for a response. This completes one heartbeat. The example below should help clarify this:

If any one of these packets is lost, then the specific heartbeat is considered failed. By default, Cluster nodes have a limit of 5 failures before the connection is marked down. Once all connections are marked down for a node, then the node is removed from active Failover Cluster membership and the 1135 event is logged on all other nodes. On the node that is removed from active Failover Cluster membership, the Cluster service is terminated and then started so it can try to rejoin the Cluster.

Now that we know how the heartbeat process works, what are some of the known causes for the process to fail.

1. Actual network hardware failures. If the packet is lost on the wire somewhere between the nodes, then the heartbeats will fail. A network trace from both nodes involved will reveal this.

2. The profile for your network connections could possibly be bouncing from Domain to Public and back to Domain again. During the transition of these changes, network I/O can be blocked. You can check to see if this is the case by looking at the Network Profile Operational log. You can find this log by opening the Event Viewer and navigating to: Applications and Services Logs\Microsoft\Windows\NetworkProfile\Operational. Look at the events in this log on the node that was mentioned in the Event ID: 1135 and see if the profile was changing at this time. If so, please check out the KB article “The network location profile changes from "Domain" to "Public" in Windows 7 or in Windows Server 2008 R2”.

3. You have IPv6 enabled on the servers, but have the following two rules disabled for Inbound and Outbound in the Windows Firewall:

Core Networking - Neighbor Discovery Advertisement
Core Networking - Neighbor Discovery Solicitation

4. Anti-virus software could be interfering with this process also. If you suspect this, test by disabling or uninstalling the software. Do this at your own risk because you will be unprotected from viruses at this point.

5. Latency on your network could also cause this to happen. The packets may not be lost between the nodes, but they may not get to the nodes fast enough before the timeout period expires.

6. IPv6 is the default protocol that Failover Clustering will use for its heartbeats. The heartbeat itself is a UDP unicast network packet that communicates over Port 3343. If there are switches, firewalls, or routers not configured properly to allow this traffic through, you can issues like this.

7. IPsec security policy refreshes can also cause this problem. The specific issue is that during an IPSec group policy update all IPsec Security Associations (SAs) are torn down by Windows Firewall with Advanced Security (WFAS). While this is happening, all network connectivity is blocked. When re-negotiating the Security Associations if there are delays in performing authentication with Active Directory, these delays (where all network communication is blocked) will also block cluster heartbeats from getting through and cause cluster health monitoring to detect nodes as down if they do not respond within the 5 second threshold.

These are the most common reasons that these events are logged, but there could be other reasons also. The point of this blog was to give you some insight into the process and also give ideas of what to look for. Some will raise the following values to their maximum values to try and get this problem to stop.

Parameter	Default	Range
SameSubnetDelay	1000 milliseconds	250-2000 milliseconds
CrossSubnetDelay	1000 milliseconds	250-4000 milliseconds
SameSubnetThreshold	5	3-10
CrossSubnetThreshold	5	3-10

Increasing these values to their maximum may make the event and node removal go away, it just masks the problem. It does not fix anything. The best thing to do is find out the root cause of the heartbeat failures and get it fixed. The only real need for increasing these values is in a multi-site scenario where nodes reside in different locations and network latency cannot be overcome.

I hope that this post helps you!

Thanks,
James Burrage
Senior Support Escalation Engineer
Windows High Availability Group

Measuring Disk Latency with Windows Performance Monitor (Perfmon)

Jeff Hughes [MSFT] — Tue, 07 Feb 2012 17:40:47 GMT

My name is Flavio Muratore and I am a Senior Support Escalation Engineer with the Windows Core Team. One subject we haven’t written much about in the Core team blog is “disk performance”.

Today I would like to talk a little bit about measuring Physical Disk IO Latency with Windows Performance Monitor (perfmon). Most likely you have some experience with Perfmon, it’s been around since the NT days.
You have probably heard general statements about what are acceptable disk latency measurements: “Less than 10 milliseconds is good and more than 20 milliseconds is bad”. Although these rules of thumb are used to simplify analysis, they do not apply in all cases and may lead to incorrect conclusions. Let’s check how this really works so we can understand these numbers.

Summary: The IO latency measured in perfmon includes all the time spent in the hardware layers as well as the time spent in the Microsoft Port Driver queue (Storport.sys for SCSI). If the running processes generate a large storport queue, the measured latency increases, as IO has to wait before getting dispatched to the hardware layers.

What is disk IO latency?
We can define disk IO latency as: A measure of the time delay from the time a disk IO request is created, until the time the disk IO request is completed.

What counters in Windows Performance Monitor show the physical disk latency?
“Physical disk performance object -> Avg. Disk sec/Read counter” - Shows the average read latency.
“Physical disk performance object -> Avg. Disk sec/Write counter” - Shows the average write latency.
“Physical disk performance object -> Avg. Disk sec/Transfer counter” - Shows the combined averages for both read and writes.
The “_Total” instance is an average of the latencies for all physical disks in the computer.
Each other instance represents an individual Physical Disk.

Note: Do no confuse with Avg. Disk Transfers/sec, which is a completely different counter.

Where does the performance data comes from?
For the “physical disk performance object”, the data is captured at the “Partition Manager” level in the storage stack.
Keep in mind Perfmon does not create any performance data per se; it only consumes data provided by other subsystems within Windows.

Where is the partition Manager in the Storage Stack?

A simplified explanation on the Windows Storage Stack follows.
When an application creates an IO request, it sends it to the Windows IO Subsystem (at the top of the stack). The IO will then make its way all the way down the stack (to the Hardware Disk Subsystem) and then come all the way back up. During this process, each layer will perform its function and then hand over the IO to the next layer.

So what are we really measuring with the Physical disk performance object -> Avg. Disk sec/Transfer (or /Read, or /Write) counter?
We are measuring all the time spent below the partition manager level.
When the IO request is sent by the Partition Manager down the stack we time stamp it, when it arrives back we time stamp it again and calculate the time difference. The time difference is the latency.

This means we are accounting for the time spent in the following components:

Class Driver – manages the device type, such as disks, tapes, etc.
Port Driver – manages the transport protocol, such as SCSI, FC, SATA, etc.
Device Miniport Driver – This is the device driver for the Storage Adapter. It is supplied by the vendor of the device (Raid Controller, and FC HBA).
Disk Subsystem – This includes everything below the Device Miniport Driver – This could be as simple as a cable connected to a single physical hard disk, or as complex as a Storage Area Network.

How disk queuing affects the measured latency in Perfmon?
There is only a limited number of IO a disk subsystem can accept at a given time. The excess IO gets queued until the disk can accept IO again. The time IO spends in the queues below the Partition Manager is accounted in the Perfmon physical disk latency measurements. As queues grow larger and IO has to wait longer, the measured latency also grows.

There are a multiple queues below the Partition Manager level:

Microsoft Port Driver Queue -SCSIport or Storport queue.
Vendor Supplied Device Driver Queue- OEM Device driver.
Hardware Queues - such as disk controller queue, SAN switches queue, array controller queue, hard disk queue.
Although not a queue, we also account for the time the hard disk spends actively servicing the IO and the travel time all the way back to the partition manager level to be marked as completed.

Finally, special attention to the Port Driver Queue (for SCSI Storport.sys).

The Port Driver is the last Microsoft component to touch an IO before we hand it off to the vendor supplied Device Miniport Driver.
If the Device Miniport Driver can’t accept any more IO because its queue and/or the hardware queues below are saturated, we will start accumulating IO on the Port Driver Queue. The size of the Microsoft Port Driver queue is limited only by the available system memory (RAM) and can grow very large, causing large measured latency.

In Conclusion:
The time the IO spent in queue is added to the disk latency in perfmon.

To keep the queue under control you have to tune your applications to limit the maximum number of outstanding I/O operations they generate. That’s a subject for another blog post.

Reference:
For SCSI Disks (FC/RAID) you can enable Storport tracing to measure the latency below the Port Driver level. This does not account for the time spent in the storport queue or anything above. Essentially this is the lowest level we can possibly monitor the latency inside Windows before the IO is handed over to third party components. Check this excellent blog from NTdebug team for details.
“Storport ETW Logging to Measure Requests Made to a Disk Unit”
http://blogs.msdn.com/b/ntdebugging/archive/2010/04/22/etw-storport.aspx

Flavio Muratore
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Cluster Shared Volumes (CSV) in redirected access mode after installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1

John Marlin [MSFT] — Mon, 06 Feb 2012 22:16:00 GMT

When seeing this, if you run FLTMC from an administrative command prompt, you may see something similar too:

C:\> fltmc

Filter Name    Num Instances      Altitude    Frame
------------------------------------------------------
CSVFilter            2            404900        0
mfehidk                           329998.99   <Legacy>
mfehidk              2            321300.00     0

If you were to generate a Cluster Log, you would see the below identifying that it cannot read the altitude value properly.

McAfee has released the following document giving a temporary workaround.

Cluster Shared Volumes (CSV) status becomes Online (Redirected access)
https://kc.mcafee.com/corporate/index?page=content&id=KB73596

Microsoft is aware of the problem and currently working on a fix. When this fix is available, this will be updated and a new KB Article will be created with the fix.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Better alternatives to manually merging snapshots in Windows Server 2008 R2

Jeff Hughes [MSFT] — Wed, 25 Jan 2012 17:17:01 GMT

There might be times on Windows Server 2008 R2 Hyper-V host when the snapshots will not merge, the virtual machine disappears from the Hyper-V management console or the virtual machine goes into a Critical-Saved or Paused state for what seems to be no reason at all. This blog will discuss the ways in Windows Server 2008 R2 to resolve some common issues with Hyper-V virtual machines and snapshots and avoid having to manually merge snapshots.

Common Scenarios

The virtual machine has unexpectedly disappeared from the Hyper-V Management console.

This problem can be caused by antivirus software that is installed in the parent partition if the real-time scanning component is configured to monitor the Hyper-V virtual machine files. Antivirus exclusions should be configured per KB; Virtual machines are missing in the Hyper-V Manager Console or when you create or start a virtual machine, you receive one of the following error codes: "0x800704C8", "0x80070037" or "0x800703E3" http://support.microsoft.com/default.aspx?scid=kb;EN-US;961804

If the virtual machine does not show up in the Hyper-V Manager console once you configure the antivirus exclusions, create a new virtual machine and attach the latest snapshot file. This is where the Inspect Disk wizard comes in handy in order to find the last AVHD in the chain.

Perform the following steps to create a new virtual machine and attach the latest snapshot (.AVHD) file:

Open Hyper-V Manager console
In the Actions pane click New Virtual Machine
Give the VM a new name, click Next
Assign Memory, click Next
Assign a Network, click Next
Select Attach a disk later, click Next

The new virtual machine shows up in the Hyper-V console. Now you need to point the virtual machine to use the most recent snapshot in order not to lose the most recent changes to the virtual machine.

1. In the Actions pane on the right select Inspect Disk

2. Browse to the location of the AVHDs you want to Inspect, click Open and you will be presented with the following window. You can see there is only 1 snapshot and the parent is the VHD.

3. Continue clicking the Inspect button on each AVHD until you have inspected all the AVHDs

4. Go to the Hyper-V console, right click the virtual machine and go to Settings, change the VHD to point to the most recent AVHD, click OK

Note: If you’re performing these steps on a Windows Server 2008 system, the Hyper-V Manager console doesn’t support attaching .AVHD files. To resolve this issue on a Windows Server 2008 system, you need to manually merge the snapshots files and attach the VHD which contains the merged data.

5. Start the virtual machine

The virtual machine will not start or the status in Hyper-V Management console shows the VM in a Paused state.

This issue is generally caused when the volume where the VM is stored is low on disk space. The idea way to resolve the issue is to add a new disk to the Hyper-V host that is large enough to hold all the virtual machines VHDS and associated snapshots and Export the VMs to the new disk. Once the virtual machines have been exported you can Import the virtual machine to add them back into the Hyper-V Manager console and the virtual machines will reside on the new disk you added.

1. Select the virtual machine in the Hyper-V Manager Console then click Export in the Actions Menu

2. Click Browse and select a disks with enough free space then click Export

The export process can take a few minutes to a couple of hours+ depending on the size of the VHD and number of AVHDs associated with the virtual machine

Once the Export is finished you can delete the virtual machine from the console then Import the virtual machine to add it back to the console.

1. In Hyper-V Manager console got the Actions menu and click Import Virtual Machine

2. Now browse to the location where the virtual machine was exported

3. Since the virtual machine was deleted leave it at the default of Move or restore the virtual machine (use the existing unique ID)

The OS was reinstalled on the parent and the virtual machines do not show up in the Hyper-V Management Console.

The best way to resolve this issue is to create a new VM and attach the most recent AVHD file.

Steven Graves
Support Escalation Engineer
Microsoft Enterprise Platforms Support

Free Microsoft Private Cloud Training

Jeff Hughes [MSFT] — Mon, 23 Jan 2012 14:47:37 GMT

Are you ready for the Microsoft Private Cloud?

On February 21 & 22 Microsoft Learning is running a 2-day virtual training event to help the world learn about the upcoming enhancements with the Creating & Managing a Private Cloud with System Center 2012 Jump Start.

It is 100% free and open to the public, so register now!

Event Overview

Adopting this exciting new computing paradigm provides a whole new landscape of technology and career direction for IT professionals. Microsoft Learning and the Microsoft System Center 2012 team have partnered to bring you an exciting opportunity to learn what you need to know to deploy, manage and maintain Microsoft’s private cloud solution. Leveraging the popular Jump Start virtual classroom approach, the industry’s most gifted cloud experts will show attendees why this new private cloud solution, based on System Center 2012 and Windows Server, has garnered so much attention. Presenters include Symon Perriman, Sean Christensen, Adam Hall, Kenon Owens, Prabu Rambadran & Chris Van Wesep and there will be a live Q&A during the event.

Event Agenda

Day 1: Deployment & Configuration (Feb. 21)

· Part 1: Understanding the Microsoft Private Cloud

· Part 2: Deploying the Infrastructure Components

· Part 3: Deploying the Private Cloud Infrastructure

· Part 4: Deploying the Service Layer

· Part 5: Deploying the Applications & VMs

Day 2: Management & Operations (Feb. 22)

· Part 6: Managing the Infrastructure Components

· Part 7: Managing the Private Cloud Infrastructure

· Part 8: Managing the Service Layer

· Part 9: Managing the Applications & VMs

Jump Start Overview

This accelerated Jump Start sponsored by Microsoft Learning is tailored for IT professionals familiar with Windows Server technologies, Hyper-V virtualization, and the System Center management solutions. The course is designed to provide a fast-paced and technical understanding of how and why Microsoft’s approach to the private cloud delivers scalability, security, flexibility and control. Here are few unique benefits of this course:

· Students have the opportunity to learn from and interact with the industry’s best cloud technologists!

· This high-energy, demo-rich learning experience will help IT Professionals understand why Microsoft private cloud solutions are making a splash in the industry.

· Students will see with their own eyes how Windows Server 2008 R2 and System Center 2012 work together to provide the best combination of security and scale.

· Information-packed agenda! Day one of this two-day online course will focus on designing and deploying the right solutions for your organization, while day two will provide an in-depth look at the tools available to help monitor, secure and control the operational aspects of a private cloud.

Error “Encountered errors while reading in the specified configuration file” when you use FSRM dirquota command

Jeff Hughes [MSFT] — Fri, 06 Jan 2012 13:56:09 GMT

My name is Himanshu Singh and in this blog I will talk about an issue seen on Windows Server 2003 64-bit servers running File Server Resource Manager (FSRM), where, when you run a dirquota command to modify notifications using a configuration file, you may encounter error “Encountered errors while reading in the specified configuration file”.

The TechNet article http://technet.microsoft.com/en-us/library/cc754836(WS.10).aspx explains the dirquota command and its syntax in detail.

Now, let me explain the issue, by taking an example:

Consider that you have an existing 85% quota threshold notification on the “MyStorage" directory in FSRM, which you would like to modify, and your modified configuration file looks like below:

Notification=m

To=someone@microsoft.com

Subject=[Quota Threshold]% share directory quota threshold exceeded

Message=The [Quota Path] share directory has reached [Quota Used Percent]% ([Quota Used MB] MBs) of the current [Quota Limit MB] MB quota limit. There are [Quota Free MB] MBs of free space remaining.

To make changes to an existing configuration using command line, you should run the dirquota quota modify command and specify the above configuration file, let’s call it "FSRMConfig.txt". The exact command is as below:

dirquota quota modify /Path:S:\MyStorage /modify-notification:85,m,c:\FSRMConfig.txt

If the above command is run on a Windows Server 2003 SP2 64-bit, you may get below error message:

"Encountered errors while reading in the specified configuration file: c:\FSRMConfig.txt"

This happens if the configuration text file is saved in ANSI format. In order to get the above command to work, re-create the configuration file and save it in UNICODE (UTF-16) format. Here is a MSDN article which describes UNICODE in detail:

http://msdn.microsoft.com/en-us/library/windows/desktop/dd374081(v=vs.85).aspx

http://msdn.microsoft.com/en-us/goglobal/bb688114

To save the file in UNICODE, you can either create a new file and save it in UNICODE or select File -> Save As option for the same file and in the save as dialog box, change the encoding to UNICODE, to save it in UNICODE format, see the screen-shot below:

After I saved the file as UNICODE the command worked and successfully modified the quota entry.

Hopefully, this would help you resolve your issue, in case you happen to run into it.

Himanshu Singh
Support Engineer
Microsoft Enterprise Platforms Support

One of the source volumes specified has invalid….” Error messaging running wbadmin to backup system state

Jeff Hughes [MSFT] — Fri, 16 Dec 2011 14:42:39 GMT

When creating a system state backup with Windows Server 2008 or Windows Server 2008 R2 you may encounter the following error:

“One of the source volumes specified has an invalid format and cannot be protected using Windows Backup. Only volumes formatted with NTFS can be protected.”

One possible cause of this error is having a FAT32 System Volume which has a 512 byte sector size (can be dependent on the size of the volume). There are two ways to work around this issue:

Convert the volume to NTFS
Delete the FAT32 volume and recreate it using NTFS.

Converting a FAT32 System Volume to an NTFS System Volume

Prerequisite: The FAT32 sector size must be 4096 bytes when originally created.

Creating a FAT32 Volume with a 4K Sector Size

Inside Disk Manager, on a disk that has been brought online and initialized, right-click on the unallocated graphical representation of the disk, and select “New Simple Volume”.

The New Simple Volume Wizard will open. Click “Next” to continue. The next window allows you to specify the volume size. In this example, the volume size specified is 2048MB. After entering the size of the volume in megabytes, click “Next”.

The next window allows you to assign a drive letter or path to the partition. In order to complete a convert, the partition needs to have a drive letter assigned to it. You can do this in this stage or in Disk Manager prior to performing a conversion. In this example, the default drive letter (next available) is “E”, and will be assigned to as the drive letter to the partition. Click “Next” to continue. An example of changing the drive letter after the partition has been created will be provided as well.

The Format Partition window is the next step in creating the partition. Here, you can select whether or not to format the volume, how you would like to format the volume, and whether or not you would like to enable file and folder compression. In this example, the volume will be given a FAT32 File system, an allocation unit size of 4096 (sector size of 4K), and Volume label is going to remain as the default “New Volume”. The default settings for performing a quick format and not enabling the file and folder compression will be used. Click “Next” to continue.

The final window in the wizard provides you with the summary of the create volume that has been created, with the specifications you have provided. Click “Finish” to continue.

The new volume will show up in the Disk Manager.

Adding a Drive Letter to a Volume that was created without one

An additional FAT32 volume was created without a drive letter. To add a drive letter to the volume, right-click on the graphical representation of the volume, and select “Change Drive Letter and Paths”.

Click on “Add” to add a drive letter to the volume. Choose which letter you would like assigned to the volume and click “OK”.

Your volume now has a drive letter.

To determine what sector or allocation unit of a volume, select the volume in diskpart, and type filesystem.

1. Open an elevated command prompt.

2. Type convert <volume letter>: /fs:ntfs

3. Hit Enter.

4. You will be asked to enter the current volume name for the volume you would like to convert.

In this example, the volume label is TEST. This can be viewed in Disk Management. Enter the volume label, and press Enter.

5. The conversion will process. It will be complete once the cursor returns to the directory you ran the convert command from. Here is an example of the output.

From the example above, you can see that the sector size remained 4096 bytes (“4,096 bytes in each allocation unit.”). Only the file system will change.

Converting FAT32 volumes to NTFS is an irrevocable operation. NTFS volumes cannot be converted.

Rebuilding a System Volume as NTFS and Rebuilding BCD Store

If a conversion from FAT32 to NTFS is not an option, rebuilding the system volume with an NTFS file system and then rebuilding the BCD store is the other option.

Using installation media, boot into the Windows Recovery Environment (WinRE). You will be asked to select a language, time, currency, keyboard or an input method. Once completed, click “Next”. Select “Repair Your Computer”. Select the operating system you’d like to repair, and click “Next”.

In the System Recovery Options window, select “Command Prompt”. Open a command prompt.

Locate where the \Boot directory is. The \Boot directory is rarely located on the C: partition, but on a separate system partition. Please keep in mind that OEMs are unlikely to assign a volume letter to the system partition.

If C: is not the Active System Partition

Due to being in WinRE, all volumes, including the system partition are assigned drive letters. The order of the drive letter assignment may/will not be the same as how the volume letters are assigned when Windows 7 is running. Because of this, partition C: may not be listed as the correct volume letter. In order to correct this, you need to obtain the correct active partition.

Open Diskpart: diskpart

List disks: list disk

Select Disk 0: select disk 0

List partitions on Disk 0: list partition

Select Partition 1: select partition 1

Get the details for this partition: detail partition

As per this example, this is a System partition and is Active. Repeat the above steps to locate to Boot partition. If this partition is not marked as active, look at the next partition to see if it is marked as active. Once located, this will be your active system partition.

On Windows Vista, one partition is typically used as both the System and Boot partitions and the \Boot\BCD store is located on the same volume where the \Windows directory is located. On Windows 7, separate partitions are used the System and Boot partitions to make it easier to enable Bitlocker Drive Encryption and to support computers with UEFI firmware.

Once the directory is located, the attributes on the BCD file must be changed to make it visible. In the \Boot directory, type

attrib bcd –s –h –r

to allow access to the BCD file. This step removes the system file attribute (-s), the hidden attribute (-h), and the read-only file file attribute (-r). In this example, C: is the active partition.

Command Lines to remove the BCD store, and recreate it (hit Enter after each line):

Bcdedit /export C:\BCD_Backup

Ren c:\Boot\bcd bcd.old

Bootrec /rebuildbcd

Shannon Gowan
Senior Support Engineer
Microsoft Enterprise Platforms Support

Troubleshooting Error 800B0100, TRUST_E_NOSIGNATURE, during Service Pack 1 installation on Windows 7 or Windows 2008 R2

Jeff Hughes [MSFT] — Wed, 14 Dec 2011 12:43:16 GMT

Hello,

My name is Himanshu and I work with the Windows Core Support Team. Recently, I worked on a patch installation issue known internally at Microsoft as a servicing issue. I thought this would be a great customer example as to how a specific error message can sometimes be misleading when troubleshooting based on error text alone.

For this issue, my customer explained me that he was getting an error 800B0100 when he trying to install patches using the Windows Update control panel applet. A screenshot of the error is below:

In addition to the Windows Update failure, attempting to open Server Manager on the server would also fail with the same error (0x800B0100) as seen in the screenshot below:

Now, let me first explain, why the error you see in Windows Update and Server Manager is related.

When you open the Server Manager it runs a discovery using Role-specific and Component Based Servicing APIs to query CBS and discover which roles, role services, and features are currently available and installed. If CBS is in an error condition, Server Manager will fail to function properly until the error condition is resolved.. You can always look at the %SYSTEMROOT%\Logs\ServerManager.log to find out what server manager is doing at any point of time. Because the ServerManager.log is in use on a running installation of Windows, you will need to copy the log to a different location to open it properly.

The error code 800B0100 maps to “TRUST_E_NOSIGNATURE” which means that the file was not signed or had a signature that was not valid.

A manifest is an XML format document that describes individual elements of installation for a component or package. Below is a sample manifest:

<?xml version="1.0" encoding="utf-8"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v3" manifestVersion="1.0">
<assemblyIdentity name="0e348ebbcfb1d8af965282e6002364f0" version="6.1.7601.21755" processorArchitecture="amd64" language="neutral" buildType="release" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" />
<deployment />
<dependency discoverable="false">
<dependentAssemblydependencyType="install">
<assemblyIdentity name="Microsoft-Windows-OS-Kernel" version="6.1.7601.21755" processorArchitecture="amd64" language="neutral" buildType="release" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" />
</dependentAssembly>
</dependency>
</assembly>

The error tells me that either there is no signature on the manifest file or the signature is corrupt. This can occur for a number of reasons such as the downloaded WU package are corrupt. In cases like this, we would recommend the customer attempt to re-download and re-attempt installation from a new package download. In this case, the customer had already attempted to download and install the package locally with similar failure results.

The next step in troubleshooting this issue would be to run the System Update Readiness tool (http://support.microsoft.com/kb/947821) to attempt to resolve the corruption within CBS. (The CheckSUR tool checks for inconsistencies in file data or registry data and detects incorrect manifests and files contained within its payload and attempts to replace them with the proper version. We ran the CheckSUR utility against the machine and the results turned out to be clean as seen in the log output…

Checking System Update Readiness.

Binary Version 6.1.7601.21645

Package Version 13.0

2011-08-24 09:28

Checking Windows Servicing Packages

Checking Package Manifests and Catalogs

Checking Package Watchlist

Checking Component Watchlist

Checking Packages

Checking Component Store

Summary:

Seconds executed: 757

No errors detected

Next, I looked at the CBS logs (%SYSTEMROOT%\Logs\CBS\CBS.log) in an attempt to determine the root of the problem. The CBS.log is similar to the ServerManager.log in that its in use and needs to be moved to another location to open properly. Note: You can also get around this by using notepad.exe in a command interpreter and typing in notepad.exe CBS.log to open the log...

CBS.log had several entries similar to the ones below:

2011-08-24 13:05:23, Info CBS WinVerifyTrust failed [HRESULT = 0x800b0100 -TRUST_E_NOSIGNATURE]

2011-08-24 13:05:23, Error CBS Failed to verify if catalog file \\?\C:\Windows\Servicing\Packages\Package_for_KB2556532~31bf3856ad364e35~amd64~~6.1.1.1.cat is valid. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:23, Info CBS Failed to initialize internal package [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:23, Info CBS Failed to create package. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:23, Error CBS Failed to internally open package. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:23, Info CBS Session: 30171776_51578712 initialized by client WindowsUpdateAgent.

2011-08-24 13:05:23, Info CBS Read out cached package applicability for package: Package_for_KB2556532~31bf3856ad364e35~amd64~~6.1.1.1, ApplicableState: 112, CurrentState:64

2011-08-24 13:05:26, Info CBS Session: 30171776_82377522 initialized by client WindowsUpdateAgent.

2011-08-24 13:05:26, Info CBS WinVerifyTrust failed [HRESULT = 0x800b0100 -TRUST_E_NOSIGNATURE]

2011-08-24 13:05:26, Error CBS Failed to verify if catalog file \\?\C:\Windows\Servicing\Packages\Package_for_KB2556532~31bf3856ad364e35~amd64~~6.1.1.1.cat is valid. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:26, Info CBS Failed to initialize internal package [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:26, Info CBS Failed to create package. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

2011-08-24 13:05:26, Error CBS Failed to internally open package. [HRESULT = 0x800b0100 - TRUST_E_NOSIGNATURE]

The errors above tell me that the problem is with a particular catalog file of patch KB2556532 (bolded and underlined above for clarity) and it has no signature on it. The best course of action here appeared to be to replace the catalog files with known good files from a working Windows installation.

The file which I want to replace was under C:\Windows\Servicing\Packages\ folder on which a normal user would have access. I have seen that many blogs suggest modifying permission on the C:\windows\servicing\Packages and then copy/replace the files. However, that’s not required. In fact, changing permissions to this directory may result in other errors and is not recommended.. To replace a corrupt file which you know is part of a particular patch, just follow the below steps:

Download and Install the System Update Readiness/Checksur tool from http://support.microsoft.com/kb/947821. Run the tool initially in an attempt to have the tool repair the corruption on the installation.
If the CheckSUR utility does not fix the corruption, download the .MSU patch for the update you're having issues with from the Microsoft Update catalog website or the Microsoft Support site and then put that file under the C:\Windows\CheckSur\Packages folder. NOTE: The \Packages folder does not natively exist and will need to be created
Run the System update readiness tool again, it will attempt to replace any corrupt/missing files from the downloaded package automatically.
You may want to view the %SYSTEMROOT%\Logs\CBS\Checksur.log to verify if the corrupt/missing files have been replaced.

In my case, this didn’t work as CHECKSUR didn’t replace any files.

I looked for the same file (C:\Windows\Servicing\Packages\Package_for_KB2556532~31bf3856ad364e35~amd64~~6.1.1.1.cat) on a working machine and they were exactly same in size, signatures, etc. So, the error looked misleading to me. I decided to find out where exactly we are failing.

Before, I ask my customer to run the SysInternals PROCMON tool to determine which component/registry/file we were failing on. I compared the C:\Windows\servicing\ folder with the working machine and found that the permissions on C:\windows\servicing\packages had been changed. Administrators, SYSTEM, TrustedInstaller and USERS had no permission on the folder. As mentioned previously, modifications to the default permissions can have adverse effects on a servicing operation.

Normally, the NT SERVICE\TrustedInstaller is the owner of the C:\Windows\servicing folder and has FULL Control on it. The SYSTEM, Local Administrators group and the Local USERS group has Read and Execute permission on this folder. The same permission is propagated down the folder to all subdirectories and files. Hence, the C:\windows\servicing\packages directory should have same permission as its parent C:\Windows\servicing.

Since, in this case permission were missing on the C:\windows\servicing\packages folder, we needed to fix the permissions. Icacls is one tool that can be used to reset the permissions on a directory.

The below command was run to add the NT Service\TrustedInstaller as owner of the Packages folder and any files/folders under it:

icacls C:\Windows\Servicing\Packages /setowner "NT SERVICE\TrustedInstaller" /t /c

We ran the below commands to give Read and Execute permission to the SYSTEM, Local Administrators and Local USERS account:

icacls c:\Windows\Servicing\Packages /grant SYSTEM:RX /t /c

icacls c:\Windows\Servicing\Packages /grant domain\administrators:RX /t /c

icacls c:\Windows\Servicing\Packages /grant domain\Users:RX /t /c

After the required permission was given, the installation of the previously failed updated completed successfully and Sever Manager came up fine as well.

In this blog, I have tried to explain the troubleshooting steps used in a typical servicing operation in a chronological order where installations of patch/patches may fail.

Hopefully, this would help you resolve issues similar to this if you are to run into them.

Himanshu Singh
Support Engineer
Microsoft Enterprise Platforms Support

Error message running WBAdmin to backup system state: “One of the source volumes specified has an invalid….”

Jeff Hughes [MSFT] — Wed, 07 Dec 2011 14:31:19 GMT

When creating a system state backup with Windows Server 2008 or Windows Server 2008 R2 you may encounter the following error:

“One of the source volumes specified has an invalid format and cannot be protected using Windows Backup. Only volumes formatted with NTFS can be protected.”

One possible cause of this error is having a FAT32 System Volume which has a 512 byte cluster size (can be dependent on the size of the volume). There are two ways to work around this issue:

· Convert the volume to NTFS

· Delete the FAT32 volume and recreate it using NTFS.

Converting a FAT32 System Volume to an NTFS System Volume

Prerequisite: The FAT32 sector size must be 4096 bytes when originally created.

Creating a FAT32 Volume with a 4K Sector Size

Inside Disk Manager, on a disk that has been brought online and initialized, right-click on the unallocated graphical representation of the disk, and select “New Simple Volume”.

The next window allows you to assign a drive letter or path to the partition. In order to complete a convert, the partition needs to have a drive letter assigned to it. You can do this in this stage or in Disk Manager prior to performing a conversion. In this example, the default drive letter (next available) is “E”, and will be assigned to as the drive letter to the partition. Click “Next” to continue. An example of changing the drive letter after the partition has been created will be provided as well.

The final window in the wizard provides you with the summary of the create volume that has been created, with the specifications you have provided. Click “Finish” to continue.

The new volume will show up in the Disk Manager.

Adding a Drive Letter to a Volume that was created without one

Click on “Add” to add a drive letter to the volume. Choose which letter you would like assigned to the volume and click “OK”.

Your volume now has a drive letter.

To determine what sector or allocation unit of a volume, select the volume in diskpart, and type filesystem.

1. Open an elevated command prompt.

2. Type convert< volume letter>: /fs:ntfs

3. Hit Enter.

4. You will be asked to enter the current volume name for the volume you would like to convert.

In this example, the volume lable is TEST. This can be viewed in Disk Management. Enter the volume label, and press Enter.

5. The conversion will process. It will be complete once the cursor returns to the directory you ran the convert command from. Here is an example of the output.

From the example above, you can see that the sector size remained 4096 bytes (“4,096 bytes in each allocation unit.”). Only the file system will change.

Converting FAT32 volumes to NTFS is an irrevocable operation. NTFS volumes cannot be converted.

Rebuilding a System Volume as NTFS and Rebuilding BCD Store

If a conversion from FAT32 to NTFS is not an option, rebuilding the system volume with an NTFS file system and then rebuilding the BCD store is the other option.

Using installation media, boot into the Windows Recovery Environment (WinRE). You will be asked to select a language, time, currency, keyboard or an input method. Once completed, click“Next”. Select “Repair Your Computer”. Select the operating system you’d like to repair, and click “Next”.

In the System Recovery Options window, select “Command Prompt”. Open a command prompt.

If C: is not the Active System Partition

Open Diskpart: diskpart

List disks: list disk

Select Disk 0: select disk 0

List partitions on Disk 0: list partition

Select Partition 1: select partition 1

Get the details for this partition: detail partition

Once the directory is located, the attributes on the BCD file must be changed to make it visible. In the \Boot directory, type attribbcd –s–h –r to allow access to the BCD file. This step removes the system file attribute (-s), the hidden attribute (-h), and the read-only file file attribute (-r). In this example, C: is the active partition.

Command Lines to remove the BCD store, and recreate it (hit Enter after each line):

· Bcdedit /export C:\BCD_Backup

· Ren c:\Boot\bcd bcd.old

· Bootrec /rebuildbcd

Shannon Gowen
Senior Support Engineer
Microsoft Platforms Support

Stop 0x50 on Windows 2008 R2 Failover Cluster

Jeff Hughes [MSFT] — Mon, 24 Oct 2011 16:03:30 GMT

Greetings Cluster fans,

John Marlin back for another go at it. I wanted to write something about what we have been seeing involving the use of Cluster Shared Volumes (CSV) and File Server resources on the same Cluster.

There have been multiple instances we have seen regarding a Stop 0x00000050 on Cluster Servers that point to CSVFILTER.SYS as being the culprit.

CSVFILTER.SYS is a filter driver used by Failover Clustering to filter metadata I/O writes to a Cluster Shared Volume. If there is a metadata write from a node that“owns” or is the coordinator node, it allows the direct I/O write. If the node is not the coordinator, CSVFILTER.SYS redirects the I/O over the network to the node that is the coordinator.

Since it is a filter driver, it will attach itself to all drives in the Cluster. The stop error occurs because CSVFILTER sees SMB I/O that it does not want to see.

These are three different scenarios where you can get a Stop 0x00000050 error.

1. A Cluster that has File Server resources only (no Hyper-V VMs) with Cluster Shared Volumes enabled.

2. File Server resources or shares that are located on the Cluster Shared Volumes.

3. A Cluster that has both Hyper-V VMs on Cluster Shared Volumes and File Server resources on non-CSV drives.

Scenario 1

==========

A Cluster that has File Server resources only (no Hyper-V VMs) with Cluster Shared Volumes enabled.

When you are in this scenario, there is no need for Cluster Shared Volumes to be enabled. To resolve this, you should disable CSV so that CSVFILTER.SYS is no longer in play.

To do this, run Powershell from the Administrative Tools with this command:

get-cluster | %{$_.EnableSharedVolumes="Disabled"}

This will disable Cluster Shared Volumes and you will no longer receive the stop errors. In this type of configuration, there is no need for the enabling of Cluster Shared Volumes as they are not being used anyway.

Scenario 2

==========

File Server resources or shares that are located on the Cluster Shared Volumes.

When you enable Cluster Shared volumes, you will receive this dialog box:

As it states, you do not want any kind of user or application data on these volumes. Key point in the box above is “may result in unpredictable behavior, including data corruption or data loss” and we all know that data integrity needs to be there.

So if you are keeping user or application data on a CSV drive, get it off or bad things can happen. This is not a valid or supported configuration.

Scenario 3

==========

A Cluster that has both Hyper-V VMs on Cluster Shared Volumes and File Server resources on non-CSV drives.

In this configuration, you have all the highly available virtual machines on CSV drives and separate groups for File Servers on non-CSV drives. As mentioned at the beginning of this, CSVFILTER.SYS is attaching itself to all drives, including these non-CSV drives. This is where you would need the workaround and there are two options to consider.

The first is to create a virtual machine that is the File Server resource and shares. Add this VM into the Cluster on the drive that you can convert to Cluster Shared Volume. This one would take some work and a little bit of time to do.

The second option is to detach CSVFILTER.SYS from the non-CSV drives. This one is the easiest and quickest to do, but it is a little kludgy. For example, say your non-CSV was the Z: drive. To detach it, the command would be:

Fltmc detach csvfilter z:

This would remove CSVFILTER.SYS as a filter on the drive. The caveat to this is that if you restart the Cluster Service, reboot the machine, or simply move the group to another node, CSVFILTER.SYS may attach itself again.

To get around this, you would want to create a batch file with the above command and place it on the Z: drive. You would need to create a Generic Application resource with this batch file. You would then want to have the File Server resources depend on this Generic Application Resource and the Generic Application resource depend on the Drive Z: resource. This way, no matter what happens, the disk comes online, CSVFILTER is told to attach, the File Server resources do what they do.

No more stop errors. Is it kludgy? Yes. Does it do the job? Yes.

Microsoft is looking into this further. There are no guarantees that a fix will be created at this point. For now, we must utilize the workarounds mentioned above.

Happy Clustering !!

John Marlin

Senior Support Escalation Engineer

Microsoft Enterprise Platforms Support

Level	Information included in trace output
000	None
020	Fatal errors
030	Unhandled exceptions
040	Errors
050	Assertions
060	Warnings
080	Exception handling
100	Event Log activity
120	General information
140	Code flow
160	Function enter and exit
170	Function return values
180	Function parameters (terse)
190	Function parameters (verbose)
200	Verbose information level 1
210	Verbose information level 2
220	Verbose information level 3
230	Fast Code Level 1
240	Fast Code Level 2
250	Fast Code Level 3
255	All