Ask the Core Team : Backup/Restore

Unable to select an attached VHD as a Shadow Copy Storage location

jeffhugh — Tue, 27 Oct 2009 19:27:09 GMT

You may notice that you cannot choose to store Shadow Copies on an attached VHD, and that when configuring Shadow Copy protection on an attached VHD, there are no other locations available to store the copies on, other than the protected VHD volume.

This behavior is by design. Illustrated below is the behavior you may see while working with attached VHDs and Shadow Copies. I’ll provide some additional information at the end of this post that will explain why this behavior is occurring.

To get into a configuration where we’ll see the behavior I’m describing:

We create our VHD in Fig.1

(Fig.1)

Remember the size of the destination volume for shadow copies must be at least 300mb before you can store a single shadow copy on the volume

(Fig.2)

Here, in Fig.2, I’m making it 350mb so that there wouldn’t be any concern about the volume being too small and thereby omitted from the list of possible locations.

Below in Fig.3, we’re going through the steps of initializing the new disk

(Fig.3)

Now, in Fig.4, we’re putting a simple volume on it

(Fig.4)

(Fig.5) Enabling Shadow Copies on the C:\ volume

In Fig.5 above, I’ve pulled up the properties of the C:\ in preparation to enable Volume Shadow Copies. You can see here, that Disk 3 (F:\) is present in the list of volumes that we can enable the feature on. F:\ is the attached VHD we created in Fig.2.

In Fig.6, let’s enable Shadow Copies on C:\ and then choose to put the copies on our F:\ drive…

(Fig.6)

Whoa… where’d the F:\ drive go?

As you can see from the list, the attached F: (VHD file) is not listed as a valid target for Shadow Copy Storage.

Also, while you can configure the attached VHD to be protected with Shadow Copies, you cannot store the Shadow Copies on any other volume but itself.

Why is this happening?

A virtual volume cannot be used as the target volume for the snapshot of another volume, and the volume can only hold shadow copies associated with its own snapshots. VSS will constrain the shadow copy storage area for attached VHDs to only allow the volume to host its own shadow copies. This behavior will ensure that the volume is self consistent, and therefore, maintain its portability. Portability is one of the bigger design goals behind the ability to create and mount VHDs under Windows 7 and 2008/R2. The end result of this behavior is that snapshots of the volume will travel with the volume as it is deployed and moved around your environment so there’s never a need to worry about a volume missing when it comes time to restore a previous version of a file.

For additional information, be sure to visit these links:

Frequently Asked Questions: Virtual Hard Disks in Windows 7

http://technet.microsoft.com/en-us/library/dd440865(WS.10).aspx#one

Understanding Virtual Hard Disks with Native Boot

http://technet.microsoft.com/en-us/library/dd799282(WS.10).aspx

Thanks for taking the time to read this. I hope it’s helped you understand one of the caveats that can be seen when using Native VHD support!

Sean Dwyer
Support Escalation Engineer
Microsoft Enterprise Platforms Support

DPM 2007 - Troubleshooting protection for Hyper-V

tomac — Fri, 31 Jul 2009 00:22:03 GMT

This post is about Windows Server 2008 with the Hyper-V role installed, that are being protected by System Center Data Protection Manager 2007. There may be one or many Virtual Machines on each Host/Parent Partition, and they may be running Windows 2003 and/or Windows 2008. Supposing the DPM Agent is installed only on the Host/Parent partition of the Hyper-V server, you may find that DPM jobs fail intermittently on the 2003 VM’s, but the 2008 VM’s successfully complete. The following error may be encountered:

Type: Recovery point
Status: Failed
Description: DPM encountered a retryable VSS error. (ID 30112 Details:
Unknown error (0x800423f3) (0x800423F3))
End time: 4/23/2009 3:37:22 PM
Start time: 4/23/2009 3:36:38 PM
Time elapsed: 00:00:44
Data transferred: 0 MB
Cluster node -
Recovery Point Type Express Full
Source details: \Backup Using Child Partition Snapshot\%ServerName%
Protection group: %ProtectionGroupName%

We found these jobs fail when the Volume Shadow Copy Service (VSS service) on the guest VM is in a “Stopping” state and the only way to get the service in a good condition is to kill the process or reboot the VM. If the VSS service is in this “Stopping” state the next DPM job will fail. But if you first verify the VSS service is in a correct state (running or stopped) the DPM job will run successfully. However, once the DPM job is done you may see the VSS service stuck in the “Stopping” state. This service should automatically stop after 3 minutes of idle time but intermittently it may not stop. We experienced this behavior across several Hosts and almost all VM’s in a particular environment. The behavior is random but a few VM’s experience the problem more frequently than others. We also noticed if the VM is rebooted it will likely work without issues for a few days before the problem re-occurs.

When using vssadmin Windows Server command (see http://technet.microsoft.com/en-us/library/cc754968(WS.10).aspx), it appeared the “Microsoft Hyper-V VSS Writer” on the host was in a “Failed” state with a “Retryable” Last error state when the job fails. Ordinarily the writer will show a “Stable” state, and “No error” as follows.

When the jobs fail, the above command will return:

Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {59f449f9-2413-494d-b679-965bc56129fd}
State: [8] Failed
Last error: Retryable error

After installing Service Pack 2 for Windows 2008, or hotfix KB967560 (see Resolution section below) and running another DPM job on a different VM that has the VSS service in a non-stopping state, the job will run succesfully and place the Hyper-V VSS Writer back into a “Stable” state.

Quick reference for different possible scenarios:

VM in good state + Host in good state = Good backup
VM in bad state + Host in good state = Failed backup
VM in bad state + Host in bad state = Failed job
VM in good state + Host in bad state = Good job

Another possible symptom, when a DPM job is running you may notice on the Hyper-V Management screen next to the VM it displays this message: “Creating VSS Snapshot Set…”. This will continue to be displayed and when looking at the Volume Shadow Copy Service inside the W2K3 VM you may notice the service is stopped. Additionally, when running “vssadmin list writers” on the Host a message is displayed but no writers are visible:

“Waiting for responses. These may be delayed if a shadow copy is being prepared.”

When this condition occurs you may not be able cancel the DPM job. When trying to cancel the DPM job you may get the following in the detailed pane:

Type: Recovery point
Status: Attempting to cancel
End time: -
Start time: 4/28/2009 10:00:21 AM
Time elapsed: 01:01:22
Data transferred: -
Cluster node -
Recovery Point Type Express Full
Source details: \Backup Using Child Partition Snapshot\%Servername%
Protection group: %ProtectionGroupName%

Looking in the System Event log on the W2K3 VM during the times when the DPM jobs failed it may be clean. But the Application event log may be filled with the VSS errors below:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 8193
Date: 2/24/2009
Time: 8:46:34 AM
User: N/A
Computer: %SystemName%
Description:
Volume Shadow Copy Service error: Unexpected error calling routine IEventSystem::Store. hr = 0x80040206.

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12302
Date: 2/24/2009
Time: 5:58:49 AM
User: N/A
Computer: %SystemName%
Description:
Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers. Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.

When viewing the System and Application event logs on the DPM server neither have any entries for the same times as the job failures. But the DPM event log may have the following entry:

Event Type: Error
Event Source: DPM-EM
Event Category:None
Event ID: 2
Date: 2/19/2009
Time: 9:20:20 AM
User: N/A
Computer: %DPMServername%
Description:
Creation of recovery points for Backup Using Child Partition Snapshot\%ProtectedServerName-VM% on %HOSTName% have failed. The last recovery point creation failed for the following reason: (ID: 3159) DPM encountered a retryable VSS error. (ID: 30112)
DPM ID: 2^|^%DPMServername%^|^Recovery point creation failures^|^DPM^|^Backup^|^%HOSTName% ^|^a48c6c91-f4ae-4ed3-b5da-a3c22d980a48

RESOLUTION

The Hyper-v issue seems to be the result of the underlying state of VSS. VSS is hung in the "stopping" state because the registry writer is hung attempting to unregister a COM+ event subscription. This is a subscription for listening for COM messages from other VSS components. When analyzing the logs captured during the problem it was found the unsubscribe function had been waiting eight minutes when the trace ended (and still had not completed).

It could be that the machine is having COM issues. The VSS service is not going to be successful with processing subsequent jobs until this unsubscribe completes. If you experience any of the symptoms mentioned above, you should perform all of the action items noted below.

Action Item #1:

Verify all the Prerequisites are met for protecting Hyper-V with DPM:

Prerequisites and Known Issues with Hyper-V Protection
http://technet.microsoft.com/en-us/library/dd347840.aspx

Action Item #2:

Online backups are not possible if any of the following conditions are not met. Verify that all the W2K3 VM’s meet these requirements:

1. Hyper-V Integration components is installed and is running the latest version

NOTE: (On the Host/Parent partition you can check VMMS.exe = 6.0.6001.22352 (or newer) and in the guest, check vmbus.sys version 6.0.6001.22334 (or newer)

2. No Dynamic disks inside the guest.

3. All volumes are NTFS

4. All NTFS volumes must be >1GB and have >300MB free space.

5. Shadow copies within the VM are on the same volume or are Disabled

6. VM is in running state.

NOTE: Offline Backups of Windows 2000 Guest VMs fail. Cause: A synthetic SCSI Controller was configured for the VM with no drives attached. Windows 2000 Guests do not support the SCSI Controller, so it is not needed.

Action Item #3:

The root cause of symptoms noted in the Problem section appear to be COM related. After verifying the action items above install the following COM updates:

KB934016 "Availability of Windows Server 2003 Post-Service Pack 2 COM+ 1.5 Hotfix Rollup Package 12"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;934016

KB965230 "FIX: The COM+ Event System does not deliver timely or reliable statistics to subscribers of the IComTrackingInfoEvents event interface in Windows Server 2003"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;965230

KB968447 "The COM+ Event System stops processing the query for matching subscriptions when it detects a corrupted subscription on a Windows Server 2003-based computer"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;968447

Action Item #4:

Install the following two W2K3 VSS updates on the W2K3 virtual machines:

KB940349 “Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;940349

KB969219 “RPC 0x800706ba and 0x800706bf errors occur when backup software tries to create VSS shadow copies on a computer that is running Windows Server 2003 SP2”
http://support.microsoft.com/default.aspx?scid=kb;EN-US;969219

Install the latest VSS/Volsnap update on the W2K3 VM’s. If the Host is also running W2K3 it will be a good idea to also install:
KB967551 “Rollup update for the volsnap.sys driver in Windows Server 2003”
http://support.microsoft.com/default.aspx?scid=kb;EN-US;967551

Action Item #5:

If possible, install W2K8 SP2 since it will include the most recent Hyper-V writer updates. But, there are situations where installing SP2 will not be an option. As an alternative you can install KB967560 and KB971394 on the Windows Server 2008 Host machine.

KB967560 update is more recent then KB959978 which does address a known issue when you run a Windows Server 2003-based virtual machine on a Windows Server 2008 Hyper-V-based computer:

KB967560 “A backup operation fails on a two-node failover cluster that is running Windows Server 2008 after one of the disk resources is moved”

http://support.microsoft.com/default.aspx?scid=kb;EN-US;967560

KB971394 "A backup of virtual machines fails when you use the Hyper-V VSS writer to back up virtual machines concurrently on a computer that is running Windows Server 2008"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;971394

How to obtain the latest service pack for Windows Server 2008
http://support.microsoft.com/kb/968849

ADDITIONAL INFORMATION:

Virtualization with Hyper-V: Supported Guest Operating Systems
http://www.microsoft.com/windowsserver2008/en/us/hyperv-supported-guest-os.aspx

Author:
Tom O’Malley
Microsoft Enterprise Support
Sr. Support Escalation Engineer

Technorati Tags: Data Protection Manager 2007,Hyper-V,Windows Server 2008

So You Want to Try a Backup Network?

jeffhugh — Thu, 26 Mar 2009 20:42:04 GMT

What is a Backup Network?

A backup network allows backup traffic segmentation from your primary network. Keeping backup traffic off your production network allows you more bandwidth for your day-to-day work without having to wait for off hours to get backups of your data. This does not mean that you will not see potential performance issues as the servers are backed up, but the impact will be much less.

DPM Backup Network Scenarios

Let us look at a scenario when a backup network would be advantageous

This is a simple backup network. As you can see, the DPM Server connects to the production network and the backup network.

This will reduce the amount of traffic on the production network by forcing all backup traffic onto the backup network.

When setting up your backup network:

1. You must have either a DNS server in the production network or hosts files on the servers. If you choose to use a DNS server you must not create a DNS sub-domain. The DNS server must be accessible from the backup network.

2. Insure that DNS name resolution for the PS on the DPM server can resolve the backup address of the protected server and vice versa. (Use NSLookup using FQDNs to check)

Important: When setting up the backup network it is necessary to include the address range of all protected servers on the backup network.

Configuring a Backup Network

To configure the backup network you need to insure that NSLookup <FQDN of the Protected Server> resolves to the backup network and vice versa.

Here is the following from the DPM Powershell Console

Add-BackupNetworkAddress [-DPMServerName] <String> -Address *.*.*.*/subnetmask [-SequenceNumber]

Parameter

Description

-Address

The IP address or subnet mask of the network

-DPMServerName <String>

Hostname of a DPM server

-SequenceNumber

Specifies the priority of the address for use as backup

The chart above lists the necessary parameters to setup a backup network.

An example of setting up a backup network follows:

Add-BackupNetworkAddress -DpmServername <DPMServerHostName> -Address 192.168.1.0/24 -SequenceNumber 1

This will add the address range 192.168.1.0/24 as the backup network for the DPM server.

Once the commands have completed successfully, it is necessary to restart the DPM Agent on the Protected Servers and to restart the DPM Service on the DPM Server.

For more information, see the TechNet Library Article:

Using Backup Network Address: http://technet.microsoft.com/en-us/library/cc964298.aspx

Walt Whitman
Senior Support Engineer
Microsoft Enterprise Platforms Support

Windows Server Backup 2008 Restore from Network Location

jeffhugh — Wed, 04 Feb 2009 21:02:29 GMT

Hello my name Scott McArthur and I am a Senior Support Escalation Engineer in the Windows Platforms group. Today’s blog submission is how to use Windows Server Backup to do a complete system restore over the network. This blog is specific to Windows Server 2008 but the process is basically the same for Windows Vista.

Assumptions

You have added the Windows Server Backup feature to your 2008 install and run a Complete PC Backup to a network location, \\server\share

The following steps show how to do this restore by using the Windows Recovery Environment UI in addition to a sample way to do this with command line.

If the network adapter or hard drive requires a third-party driver, you should put those drivers on a USB flash drive.

1. Start the computer by using the Windows Server 2008 DVD

2. On the first screen Click Next.

3. Select the “Repair your computer” option in the lower-left corner of screen.

4. It will show you any currently installed operating systems. Click Next.

If this screen is blank you may have to load a third-party driver for your mass storage driver. You can click Load Drivers to load the mass storage driver from your USB flash drive.

5. Click “Windows Complete PC Restore”.

6. It will report “A valid backup location could not be found”. Click cancel.

7. Select “Restore a different backup” then next.

8. Click Advanced.

9. If the network adapter driver is included with Windows Server 2008 you can click “Search for a backup on the network. If the network adapter driver is not included you have to click “install a driver” and browse to your driver to load it. In my test I was using a Hyper-V virtual machine with the legacy network adapter. The legacy network adapter driver is in Windows Server 2008 so that it just works. The synthetic driver is not included.

You would receive an error similar to the following when you try to connect to the server.

10. Click Yes to the “Are you sure you want to connect to the network” and then specify the path of your backup. You can use IP address instead to eliminate any netbios/dns issues.

11. Select the backup listed and then click Next.

12. Select the backup then Next.

13. You are presented with the restore options.

The exclude disks option enables you to exclude disks from the restore process.

The advanced button has the following options.

14. Click Finish to confirm the settings.

If you selected “Format and repartition disks” you are prompted with “Windows Complete PC restore will format the disks you chose to restore, which will erase all existing data”. Click “I confirm that I want to format the disks and restore the backup”.

You can monitor the progress through the final dialog box.

The restart will automatically occur or you can delay it.

For some scenarios, you may want to do this manually. The following lists the basic general steps to do this at the command prompt.

1. At the following screen select the “Command prompt” option.

2. Run the following commands at the command prompt.

If your network adapter is not included with Windows you need to load the driver using drvload.

Drvload driver.inf

Where driver.inf is the inf for the network adapter. You can load this from an usb flash drive or the local drive if it is available.

Start /w wpeinit

Run ipconfig to verify the driver loaded and you have a valid ip address. If you have to set a static IP address, you could use netsh. For example

“netsh interface ipv4 set address name = "<idx>" source=static address=<staticIP> mask=<SubnetMask> gateway=<DefaultGateway>”

Netsh Commands for Interface (IPv4 and IPv6)

http://technet.microsoft.com/en-us/library/cc770948.aspx

Net use z: \\server\share /user:username

Where \\server\share is the location that you saved your backup to. There should be a WindowsImageBackup folder in the root of the share/directory that you saved your backup to.

wbadmin get versions -backuptarget:\\server\share

You should output similar to the following

The times of the backups displayed are based on the timezone of the current
operating system you have booted into.
The timezone used currently is (GMT -08:00) Pacific Standard Time

Backup time: 1/8/2009 11:20 AM
Backup target: Network Share labeled \\server\share
Version identifier: 01/08/2009-19:20
Can Recover: Volume(s), File(s), Application(s), Bare Metal Recovery

From this you must have the Version Identifier for the next step.

Note this basically wipes the drive so be careful when you use the next command

wbadmin start sysrecovery –backuptarget:\\server\share -version:01/08/2009-19:20 -recreatedisks –restoreallvolumes -quiet

Hopefully this helps show how to do a basic restore of a Windows Server 2008 computer.

Scott McArthur
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support