System Center: Data Protection Manager Engineering Team Blog

DPM and System State Backup Explained

J.C. Hornbeck — Thu, 16 May 2013 17:38:19 GMT

Chris Butcher | Senior Support Escalation Engineer

Hello again folks, this is Chris Butcher and I realized it has been too long since my last blog post. I recently worked with a customer and through our conversations realized that there is a lack of information out there on how System Center 2012 Data Protection Manager (DPM) protection for System State and Bare Metal Recovery (BMR) works. While it is a short read, it seemed like a good opportunity to be sure the information is out there.

So, as we were talking about the specifics of a system state backup, I found that the documentation wasn’t quite as thorough as it should be. This will focus primarily on that piece (system state) mainly because the BMR piece is pretty basic and doesn’t need too much attention.

The process for both will be focused on Windows 2008 and forward. This is when DPM began leveraging the new Windows Server Backup (WSB) feature as NTBackup had been deprecated.

What happens when a system state backup is taken against a protected server? Well, the first thing DPM does is communicate with WSB to request a backup of the server’s system state. While this seems basic, there are some other items to be considered during this process. Behind the scenes, DPM and WSB made a determination when the agent was installed as to which drive it would use. This by default is the drive with the most available free space. That information gets saved into the PSDataSourceConfig.XML file (which we will talk about later). This is the drive WSB will use to do backups to.

WARNING: While this seems like a pretty simple calculation that will likely not affect you, there is one caveat here to be aware of. If the server you are protection is a member of a cluster, it is possible that a cluster drive will be selected in this process. If that drive ownership has been switched to another node, then next time system state for the node is done, the drive is no longer available and the backup will fail. In this situation, you will need to modify the PSDataSourceConfig.XML to point it to a local drive. This process is detailed later.

WSB will then create a folder on the root of that drive called WindowsImageBackup. As it creates the backup, all of the data will be put there. Upon completion of the backup, the file will then be transferred over to the DPM server.

As shown here, you can see that my DPM server shows a backup done at 10:23PM.

On the computer I am protecting (in this case cbutch782), you will find the WindowsImageBackup folder on my C: drive with a creation date/time that matches.

This is where there may be confusion. The following two points aren’t common knowledge:

1. This folder and its contents do not get cleaned up after the backup or transfer is done. The best way to think of this is that the space is being reserved for the next time a backup is done.

2. The folder gets created every time a backup is done. The time/date stamp will reflect the time of your last system state backup.

How does this differ from a BMR backup? Well, with BMR (which inherently captures a system state as part of the process), the backup job will be done directly to a share on the DPM server. What happens is that the DPM server shares out the replica volume for that BMR backup and when it calls WSB, it just tells it to not use the drive with the most free space, but instead use the share that it has created for this job.

So, you are probably asking yourself how you can change this behavior. You may not want DPM to use a given drive for the system state backup. Well, you are in luck. There is a way to tell DPM which drive to use.

1. On the server you are backing up, navigate to C:\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

2. There you will find a file named PSDataSourceConfig.XML. Right click that file and choose to edit.

3. Find the section for <FilesToProtect> which is followed by a drive letter and WindowsImageBackup. This will be the current location WSB will put the files.

4. Modify the file to reflect the drive letter you want WSB to use. In my case, I changed it to use my E: drive.

5. Once this is done, close and save the file.

6. Go back to the DPM server and run a consistency check against the System State data source.

You can see that after running that in my case, I now have a new folder on my E: drive with the time/date that matches the consistency check (which of course also creates a recovery point).

And on the DPM server, we see the same time and date as the WindowsImageBackup folder.

Well, that’s about it for system state backups. As I said, it is primarily informational to allow our users to better understand the process and hopefully not be too surprised when the find a random folder somewhere on their protected computer. Fear not, it is expected!

Here are some other blog posts and troubleshooting steps you can walk through that deal with system state and BMR backups.

BLOG POSTS:

· http://blogs.technet.com/b/filecab/archive/2009/05/04/deciding-between-system-state-backup-and-allcritical-backup-in-windows-server-2008.aspx

· http://blogs.technet.com/b/dpm/archive/2011/10/31/troubleshooting-data-protection-manager-system-state-and-bare-metal-backup.aspx

· http://blogs.technet.com/b/dpm/archive/2010/05/12/performing-a-bare-metal-restore-with-dpm-2010.aspx

TechNet: Troubleshoot VSS issues which occur with Windows Server Backup (WBADMIN) on Windows Server 2008 and Server 2008 R2:

http://technet.microsoft.com/en-us/library/ee692290(WS.10).aspx

Chris Butcher | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

System Center All Up: http://blogs.technet.com/b/systemcenter/
System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/
System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/
System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/
System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

MVP Speaker Series Presentation on “Backup That Really Works”

Travis Wright MSFT — Thu, 16 May 2013 06:48:51 GMT

Robert Hedblom, one of our Microsoft MVPs, gave a great presentation a few days ago on “Backup That Really Works” using multiple components of the System Center 2012 SP1 suite. If you would like to watch the presentation, check it out on the Partner and Customer Solutions blog!

If you would like to review the deck, you may find it located here.

Check out this post for more details on the Windows Server/System Center MVP Speaker Series.

SC 2012 SP1 – DPM: Leveraging DPM ScaleOut feature to protect VMs deployed on a big cluster

Neela Syam Kolli, Program Manager — Wed, 01 May 2013 12:16:57 GMT

Windows 2012 improved the CSV clustering by increased scale and performance. Windows 2012 clusters can be as big as 64 nodes and 8000 VMs. Windows 2012 clusters can now have their storage on a Cluster Storage Volume or can be a remote SMB share which in turn can be a scale out clusters. As part of DPM 2012 SP1s mission of enabling efficient backup for Windows 2012 private cloud deployments, DPM leveraged Windows 2012 CSV capabilities and improved backup performance by 900% (assuming VM had 10% churning per day). DPM 2012 SP1 can now protect 64 node clusters as opposed to 16 node clusters. Even though DPM 2012 SP1’s scale numbers have increased to 800 VMs with 100GB average VM size with 3% churning per day and once a day backup, one DPM server cannot protect 64 node clusters. Until DPM 2012 SP1, VMs on a cluster can only be protected by a single DPM server. This meant that DPM 2012 could not protect a 64 node cluster. With DPM 2012 SP1, VMs on a single cluster can be protected by multiple DPM servers. This is achieved by DPM’s architectural breakthrough implementation of DPM ScaleOut feature. By using this feature, agents running on a Hyper-V cluster can be “attached” to multiple DPM servers. Each DPM server can be configured to protect “some” VMs on a cluster. The strong affinity of a VM and its “backing up” DPM server is still maintained. So, even though the VM is migrated within a cluster or across clusters, same DPM server will continue to protect the VM even if the VM is migrated to other Hyper-V deployments. Refer to the blog article “VM Mobility: Uninterrupted data protection” blog article located here.

Enabling DPM Scale out feature is basically about agent installation on node(s) and attaching them to multiple DPM servers. This can be achieved by performing below mentioned steps.

Calculate the number of DPM servers required for protecting a cluster using DPM 2012 SP1 Hyper-V Calculator

For ex., assume that this particular deployment has 64 node cluster with 4000 VMs
100GB average VM size
Once a day backup
3% a day data churning
Per DPM 2012 SP1 Hyper-V Calculator, 5 DPM servers are required to protect this 64 node cluster. Let us name them as DPM1,… DPM5

Deploy DPM agent on all nodes via one of the DPM servers say DPM1 as mentioned here or manually as described here
Now all 64 nodes’ agent can communicate with DPM1 and DPM1 can protect any VM on 64 node cluster
Next step is to connect all 64 nodes to DPM2, … DPM5 with the following steps

On each node of 64 node cluster run the following commands to connect to all required DPM servers

SetDpmServer.exe –Add –dpmServerName DPM2
SetDpmServer.exe –Add –dpmServerName DPM3
SetDpmServer.exe –Add –dpmServerName DPM4
SetDpmServer.exe –Add –dpmServerName DPM5

On each DPM server (DPM2 to DPM5) for each cluster node

Attach-ProductionServer.ps1 DPMServerName nodeN <user name> <password> <domain> as described in step5 here
Or use DPM’s administrator console, management, Agents, click “Install” and select the “attach agents” as shown below

Perform agent attach operation for all DPM servers and all 64 nodes of the cluster
By following these steps, all DPM servers are connected to all 64 nodes on the cluster

For configuring VM backup, create PG(s) with/without colocation depending on various parameters, protect VMs on each DPM sever. As mentioned above, once a VM is protected by a DPM server, that VM is managed by this particular DPM server for the rest of the VM life cycle. By following this procedure, 5 DPM servers are needed to protect 4000 VMs on a 64 node cluster (actual DPM requirements vary depending on the VM size, number of VMs, VM churning etc). To protect them one can take multiple methods to protect VMs. If this is a organically growing environment, one can start protecting with one DPM server and when it reaches its limits, deploy another DPM server to protect rest of the VMs and so on. If the deployment is already big and customers are opting-in for backup on a regular basis, one can deploy multiple DPM servers at the same time and load balance the VM protection across multiple DPM servers. Backup admin can protect VMs in a DPM server by performing following steps.

1. Opt-in for new protection and expand cluster

2. Select VM to be protected

3. Configure VM protection parameters

Due to the increased cluster sizes, the first step can take a long time for the scanning the cluster and configure the backups. Considering this, DPM 2012 SP1 dramatically improved on the time it takes to achieve first step above for clusters. DPM improved this by two ways. One expanding caching for cluster parameters and other by improving the VM enquiry performance by performing enquiry of all VMs per node in a single step. Here are the steps in achieving optimal enquiry performance when enquiring VMs in a CSV cluster. First time when DPM is started to protect VMs of a cluster, following steps are to be done to improve the backup configuration performance.

a) Click and expand each node of the cluster as shown below. Overall node level resource population should take about 3 to 5 minutes depending on the Hyper-V load running etc. So, scanning whole cluster would roughly take about 5 to 7 minutes.

b) Once the node level enquiry is complete, then expand the Cluster that will show all the VMs that can be selected for protection as shown below.

c) Follow usual protection steps after this.

Once PG is created on the cluster, DPM will cache the cluster, nodes and VM info and will update the caching as part of nightly jobs. At times, refreshed data is necessary for protecting newly deployed VMs. In that case, either backup admin can wait for next day or force fresh enquiry on the cluster manually by initiating the fresh enquiry process as described in steps #a and #b above.

Other points to consider:

Secondary DPM protection is not supported for a scale-out DPM server deployments.

In each DPM server connected to this cluster show all VMs that are present in the cluster even though a particular VM might already been protected by another DPM server. Backup admin is advised not to protect the VM on multiple DPM servers. If a VM is protected initially by DPM1 and then protected by DPM2, DPM2 will take ownership of the VM and protecting from this DPM server. Backups on DPM1 will fail. If this is done by mistake, DPM admin can follow the steps mentioned below.

· Stop protect (with retain data to keep the current recovery points that are in this server) on DPM2

· Go to DPM1 and force CC on the VM and backups would resume on DPM1 and DPM1 will now be the owner of this VM

As the scale out option enables the protection of any VM on cluster by any DPM server, AutoProtectInstance.ps1 that enables auto protection of all VMs on a cluster to a DPM server cannot work in this environment.

As scale out feature can potentially do unlimited number of backups from one node causing production server workload, we limited the number of concurrent backups to 8 per node.

If the number of concurrent backups is exceeded, the recovery point job will fail with following details and error code.

Type: Recovery point

Status: Failed

Description: DPM could not run the backup job for the data source because the number of currently running backup and recovery jobs on the Production server has reached its limits.

Data source: \Backup Using Child Partition Snapshot\Server name

Production Server: XXXXXX (ID 3185 Details: Internal error code: 0x809909E5)

Backup admin will see following Alert messages as a reason for backup failure. As this effects production deployments and is dependent on node to node, this is a node specific parameter.

DPM could not run the backup job for the data source because the number of currently running backup and recovery jobs on the Production server has reached its limits. Data source: %DatasourceName; Production Server: %ServerName; Reduce the number of backup/recovery jobs running on this production server, or wait for some backup and recovery jobs to complete and retry the operation.
DPM could not run the recovery job for the data source because the number of currently running backup and recovery jobs on the Production server has reached its limits. Data source: %DatasourceName; Production Server: %ServerName; Reduce the number of backup/recovery jobs running on this production server, or wait for some backup and recovery jobs to complete and retry the operation.

The number of concurrent backups is a configurable parameter described below error messages located in DataSoureResourceLimit.xml located in DPM agent installation folder on production server. DPMRA need to be restarted by going to services and click restart of DPMRA service to take this change effected.

<DatasourceLimits>
    <Writer writerId="66841cd4-6ded-4f4b-8f17-fd23f8ddc3de" version="0" isParallelRecoveryAllowed="true">
        <MaxLimit value="8" type="1"/> –> This represents the number of backups allowed for Hyper-V workloads. Fine tune this number based on the number of VMs, parallel backup load that can be allowed on production server etc.
        <MaxLimit value="8" type="2"/>
        <MaxLimit value="8" type="3"/>
        <MaxLimit value="8" type="4"/> –> This represents the number of recoveries allowed for Hyper-V workloads. Fine tune this number based on the number of VMs, parallel recovery load that can be allowed on the production server etc.
    </Writer>
</DatasourceLimits>

DPM Scale out feature is supported only for Windows 2012 Hyper-V CSV workloads.

Support Tip: Hyper-V hosts fail and log Event ID 5120 when being backed up

J.C. Hornbeck — Mon, 29 Apr 2013 18:09:00 GMT

Michael Vargo | Senior Support Escalation Engineer

Some customers are experiencing failures on Windows Server 2012 Hyper-V hosts that utilize Cluster Shared Volumes (CSVs) when backing up virtual machines (VM’s) using System Center 2012 Data Protection Manager (DPM 2012) with Service Pack 1 (SP1).

When this occurs the following error is logged on the Hyper-V hosts.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Event ID: 5120
Logged: <date/time>
Details: Cluster Shared Volume 'Volume2' ('ClusterStorage Volume 2') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

You will also see the VM’s pause temporarily, go in to a paused state, shut down or see memory spikes on the Hyper-V hosts. Configuring serialized backups doesn't help the problem.

It has been determined that any backup product using shadow copies results in the same errors, thus this isn't necessarily a DPM 2012 SP1 issue. The Windows Server 2012 cluster CSV team is investigating this problem.

UPDATE (5/20/2013):

The Windows and DPM teams have released a new fixes to address some general Cluster and CSV backup issues and are available for download. There are no other configuration changes necessary after applying these updates.

DPM UPDATE
KB2802159 - Description of Update Rollup 2 for System Center 2012 Service Pack 1 (http://support.microsoft.com/kb/2802159)

WINDOWS UPDATE
KB2838669 - Update that improves cluster resiliency in Windows Server 2012 is available (http://support.microsoft.com/kb/2838669)

Additionally, the update also resolves the issues that are described in the following Microsoft Knowledge Base (KB) articles:

2799728 (http://support.microsoft.com/kb/2799728/ ) Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster

2801054 (http://support.microsoft.com/kb/2801054/ ) VSS_E_SNAPSHOT_SET_IN_PROGRESS error when you try to back up a virtual machine in Windows Server 2012

2796995 (http://support.microsoft.com/kb/2796995/ ) Offloaded Data Transfers fail on a computer that is running Windows 8 or Windows Server 2012

2813630 (http://support.microsoft.com/kb/2813630/ ) Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster

2824600 (http://support.microsoft.com/kb/2824600/ ) Virtual machine enters a paused state or goes offline when you try to create a backup of the virtual machine on a CSV volume in Windows Server 2012

If you continue to see problems protecting Windows 2012 Hyper-V guests after installing the above hotfix, please open a support case for further investigation.

Michael Vargo | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

SC 2012 SP1 – DPM: Efficient Protection of Windows 2012 Dedup Volume

Neela Syam Kolli, Program Manager — Mon, 29 Apr 2013 14:17:17 GMT

Windows 2012 introduced deduplication feature that provides great savings for the file systems with lot of redundant data. File systems in general, most of the data is cold meaning most of the files are not changed. Windows 2012 deduplicates the cold file contents where all the common “chunks” in various files are stored in common area and the actual file will have links to the common chunk area. This will lead to a huge savings when there is lot of duplicate content in the files.

DPM 2012 SP1 can now protect a Windows 2012 deduplicated volume efficiently. When user chooses to protect a full volume that is deduplicated, DPM recognizes that this is a deduplicated volume and copy the content efficiently providing huge network and DPM storage savings. When a file system is deduplicated, Windows 2012 keeps all common chunks under chunk store located in sysvol. All the files will have links called reparse points that will point to chunk store. DPM initially copies the whole chunk store and all files in deduped format. In subsequent delta replications (DRs), DPM tracks changes in both chunk store as well as on the files and transfer only changed content. This means that DPM transfers the deduplicated volume in a dedup format.

Deduplication can be enabled on a volume as described here. Once this is enabled, deduplication logic will work on the “cold” files as configured by user and deduplicates data causing the storage reduction on the volume. Once the PG is created with the whole volume backup, DPM has intelligence built in that detects deduped volume and backs up data efficiently. For ex., if a volume has 100GB of files before deduplication and its storage consumption gone down to 70GB after deduplication, DPM transfer the content as part of Initial Replication (IR) as 70GB over the wire and store it as 70GB. This provides great DPM network and storage savings. Here are the steps to be followed to leverage this capability.

1) Assume that the PS1 is the production server where the file system volume (Vol1) is residing and DPM1 is DPM server.

2) Install Deduplication role on PS1

3) Enable Deduplication on Vol1

4) Install DPM server on DPM1

5) Install Deduplication role on DPM1 machine

6) Install DPM agent on PS1

7) Create Protection Group (PG) and select Vol1 on PS1 with appropriate protection settings

8) DPM will not only recognize that this is a deduplicated volume but also transfers the content efficiently

Even though DPM efficiently backups the file system, backup admin can still leverage DPM’s Item Level Recovery (ILR) capability to recover small set of files or directories instead of the whole volume. DPM is able to achieve this by leveraging Windows 2012 Dedup technology to understand the file system and recover the required items. This is the reason, DPM server should be running on Windows 2012 and Dedup role need to be installed. Note that the Dedup capability should not be enabled on the DPM storage (replica or shadow copy volumes). This efficient backup capability can be availed only when full volume is backed up and restored. Here is the table that shows various scenarios and DPM’s protection and recovery efficiency capabilities.

Internal workings of DPM Dedup Backup and Recovery:

Windows 2012 deduplicates data at a volume level and stores all “dedup” chunks in sysvol folder called chunkstore. All the files that has the “duplicate” content will point to this chunk store. By having links for duplicate content, Windows is able to reduce the storage consumption. DPM agent on the file server recognizes that the volume has dedup enabled, reads the files in “shallow” form and stores on DPM in “shallow” format. DPM also copies the whole chunk store located in sys volume folder as is. DPM expanded its “expressfull” technology to Dedup file system protection as well. This means, the DPM will continue to track the file changes and at the time of backup DPM will just copy the changed content.

There are various kinds of recovery options available with DPM. Each kind of recovery has a specific requirement to leverage the dedup efficiencies. All of these details are captured below. Note that all of the below scenarios assumes that the source volume was deduped at protection time DPM protected full volume efficiently.

Once the Dedup file system is protected to primary DPM server, the file system cannot be protected to secondary DPM server. Deduped file system protection to DPM server can be further extended to Online protection by opting-in for Windows Azure Backup. As the Windows Azure Backup supports “subset” of the protected resources, the protection to Azure will be done only for files that were selected for DPM Azure and will be done in unoptimized format. End User Recovery feature is not supported for Dedup Volume protection.

One frequently asked question is, why shouldn’t we enable dedup on DPM storage volumes (replica and shadow copy volumes). This needs understanding of how DPM stores its backup data. DPM has replica volume which reflects latest and greatest snapshot of the production server. As part of IR, Replica Volume will reflect the production server and a snapshot is created. At the time of next backup, DPM copies the new content onto replica volume which will cause a Copy On Write (COW) onto shadow copy volume as VolSnap is keeping the old snapshot intact. After backup completes, DPM creates a snapshot on Replica volume. VolSnap will do COW for any subsequent changes to this “snapshotted” volume. So, when dedup engine try to deduplicate the content, all writes on Replica Volume will lead to COW and so bloats up diff area. So, actual DPM storage consumption will go up due to this diff area increase. Another issue is that DPM’s CC logic will not work as the files on DPM side and on production side are mismatching. This makes CC think that backups are not proper and transfer all content again. So, dedup should not be enabled on DPM storage volumes.

Another interesting scenario where dedup is enabled on the volume that is already being backed up. When dedup is enabled on the volume, dedup will change almost all of files as part of dedup logic even though actual content is not changed. In the next backup, DPM sees this as file changes and will transfer all deduped files. This leads to a one time spike in DPM backup storage consumption.

SC 2012 SP1 – DPM: Windows 2012 VM Mobility – Uninterrupted Data Protection

Neela Syam Kolli, Program Manager — Wed, 24 Apr 2013 07:10:18 GMT

Technorati Tags: DPM 2012 SP1 VM Mobility private cloud protection

One of the important features enabled in Windows 2012 is the ability to live migrate VMs anywhere. Whether the VM migrating within a cluster, across clusters, standalone to standalone or standalone to cluster or vice versa, VM will not be paused or interrupted at the time of VM live migration. This efficiency enables fabric administrators manage their fabric and load balance the environment and ensure that their private cloud deployments are running optimally. While this provides freedom and flexibility for fabric administrators, this poses management problem for backup administrators. As the fabric administrator leveraging these capabilities to optimize the fabric performance and taking these decisions independently backup administrators will not be aware of this until backup software fails to find the VM and so failing the backups. This kind of dynamic fabric provides problems for the backup administrators.

DPM 2012 SP1’s motto is to provide flexible and yet powerful backup solutions for your private cloud deployments. As part of this target, DPM introduced intelligence where DPM can detect the VM movement and continue VM protection, all without any active involvement from backup administrator. Also, not only DPM can detect the VM migration but also can continue protection efficiently when VM migration does not involve storage migration. As the VM migrates from one host(s) to another host(s), same DPM server will continue to retain the backup responsibility for this VM.

At the time of backup, DPM detects the VM’s new host by communicating with VMM. At the time of backup, DPM interacts with VMM to find the VM’s new host. Based on the information, DPM communicates with new host(s) for taking backups. In order to achieve this, backup administrators need to connect DPM(s) to VMM server and all host(s) where VM “will” be migrated to, so that DPM can communicate with the new host(s) to backup VM when it migrates.

Enable DPM communication with VMM:

In order for DPM to interact with VMM, make DPM machine account as “ReadOnly” administrator on VMM and as discussed here. This will allow DPM to enquire about any VM which might be residing anywhere in VMM’s fabric/cloud. DPM communicates with VMM using VMM power shell. So, VMM client should be installed on DPM Server. Also, configure DPM with the VMM server name by setting DPM global property called “KnownVMMServers”. Here are the steps that need to be performed for setting up DPM server ready for VM Live Migration: Uninterrupted data protection.

1) Install VMM Console on the DPM server. Ensure that the VMM console and VMM server are of same version.

2) Install DPM protection agent on all “targeted” hosts and attach to this DPM server

3) Using elevated DPM Windows PowerShell, run Set-DPMGlobalProperty as shown below on the DPM server

Set-DPMGlobalProperty –DPMServerName DPM01 –KnownVMMServers VMMTST01

Above command preps DPM server to communicate with VMM server named VMMTST01. Subsequently DPM consults VMM server for any VM backup to find its new host(s). Please note that DPM server can interact with only one VMM server and so the variable should be set with only one VMM server. DPM PowerShell can be used to verify that the global property is set properly by using Get-DPMGlobalProperty as shown below.

4) Start the DPMVMMHelper service

Also, all “potential target” hosts should be configured to connect with this particular DPM server. As mentioned previously, DPM now supports DPM scale out feature. So, same Hyper-V host(s) can be configured to communicate with multiple DPM servers. For an interrupted data protection, DPM should already be attached to all host(s) where the VMs are going to be migrated.

Windows 2012 VM live migrate “anywhere”, makes the VM migration possible in various kinds of Hyper-V deployments. There are three kinds of VM live migration at a high level.

A VM live migration where the migration involves only compute but not storage
A VM live migration where the migration involves both compute and storage
A VM live migration where migration involves only storage but not compute

Whenever there is a storage migration involved in a VM live migration, DPM will do onetime full CC of the VM (ie., read VM on both “new” Hyper-V host(s) and DPM server, compare checksum and then transfer changed content to DPM server). Normal expressfull Delta Replication (DR) backups will resume after CC. When live migration doesn’t involve storage migration, DPM do not need to go through full CC and so expressfull DR backups will continue. Here is the list of combinations that live migration can be done and DPM’s protection behavior.

As DPM is dependent on the VMM to know where the VM has been migrated to, for an effective backup, it is advised to do the VM migration using VMM Live Migration workflow as described here. In some of the scenarios, Windows 2012 VM live migration with storage migration causes VM to be “rearranged” on the destination. This leads to almost whole size of VM being backed up by DPM via CC in the next backup. DPM has a new service named DPM VMM Helper Service which will be interacting with the VMM server. This service will be running on system context.

While connecting all hosts to single DPM enable “uninterrupted VM protection” and SC 2012 SP1 – DPM enhanced its scale, a single DPM server cannot protect all VMs in the private cloud deployment. Considering this, DPM 2012 SP1 introduced a new capability called scaleout feature where multiple DPM servers can protect VMs on a single host or a single cluster. This is further discussed in the blog “SC 2012 SP1 – DPM: Leveraging DPM ScaleOut feature to protect VMs deployed on a big cluster”.

DPM’s communication with VMM can be reverted by resetting the “KnownVMMServers” global property as shown below.

Set-DPMGlobalProperty –DPMServerName DPM01 –KnownVMMServers “”

This command will ensure that DPM will not look for/communicate with the VMM server. Also, if same command is issued with a different VMM server name, DPM will start communicating with the new VMM server.

VM Recovery

With the mobility in picture, DPM recovery scenarios are changed a little bit. Any recovery points on or after the latest recovery point can do Original Location Recovery (OLR) meaning the VM can be recovered to location where the VM is currently running. For all these recovery points, DPM can do Alternate Location Recovery (ALR) and Item Level Recovery (ILR) as well. For all recovery points before last VM live migration, only ALR and ILR are possible. OLR scenario is not supported for these recovery points.

Due to various restrictions, VM mobility cannot be combined with Secondary DPM protection (or DPM DR).

Reduce backup storage consumption of private cloud deployments using System Center 2012 SP1 – DPM

Neela Syam Kolli, Program Manager — Thu, 18 Apr 2013 09:36:46 GMT

System Center 2012 SP1 - DPM has enabled data protection for Windows 2012 Private Cloud deployments. Customers have been looking for a way to exclude some of the VHD/VHDX that have temp data or downloads or page files which have no value @ recovery time. Based on this feedback, we have introduced a new feature called “exclude VHDX”, where customer can configure DPM to exclude some of the VHDs on their backup. All backup admin has to do is run the following command on DPM server.

PowerShell

PS C:\> Set-DPMGlobalProperty -DPMServerName TestingServer -HyperVPagefileExclusions "*Temp.vhd"

By issuing this command, DPM excludes any file name that ends with “Temp.VHD” in the Delta Replication (DR). Since these files are included in the VM configuration and Hyper-V expects these files to be there for successful VM start, DPM copies these files for the first time and excludes in subsequent DRs. At the time of recovery, DPM will recover the “excluded” files as well. This will ensure that the Hyper-V is able to run VM successfully.

This command can take wild cards and multiple names. For ex., "*_pagefile.vhd,*mynotuseful*.vhd*" excludes backup of any vhd/vhdx that has word “mynotuseful” in their names and VHD file name that ends with “_pagefile.vhd”. Note that the file name strings are case insensitive. This command applies to all VMs protected by this DPM server. If there are multiple DPM servers present in the environment, this command needs to be run on each DPM server. One important aspect to consider at the VM deployment is pagefile size increases and its impact on VHD size. While Hyper-V allows changes to VHD file size after initial creation, it is advised to plan for VHD size at initial time itself. DPM backs up VHD only in the beginning and any subsequent changes in the VHD will not be backed up and could lead to issues at the time of VM recovery.

If this feature is enabled on the DPM server that has VMs being protected already, DPM will stop backing up the “exclude VHDs” in subsequent backups.

One of the biggest disk churning that has no value for customer @ recovery time is page file. Using this “exclude VHDs” feature, customer can remove the page file changes in the DPM backups. Here are the steps involved in excluding pagefile in backup.

1) Add a new VHD say “TempVHD” to the VM

2) Start the VM

3) Format the TempVHD disk and configure it to a drive letter say “E:\”

4) Move the pagefile to E:\

5) Run Set-DPMGlobalProperty -DPMServerName TestingServer -HyperVPagefileExclusions "Temp.vhd" on DPM Server

Once these steps are done, DPM will copy whole TempVHD as part of Initial Replication (IR) and will be excluded in subsequent DRs. If DPM goes through a CC or VM file spec changes, TempVHD will be copied in full and subsequent DRs will be skipped.

Our research found that, even though VM is churning at 10% of VM size a day, only 30% to 40% of this churning is due to the actual application data that is important to customer at recovery time but rest of the churning due to pagefile changes and temp file changes that has no use for customer at recovery time. In these kinds of environments, DPM 2012 SP1 brings in huge backup storage and network bandwidth savings. While actual storage savings vary depending on the environment, consider the following Windows 2012 64 node cluster with 4,000 VMs deployment.

Here is the storage consumption comparison between DPM 2012 and DPM 2012 SP1.

While the above example refers to Hyper-V CSV cluster, this feature works with all kinds of Hyper-V deployments that are protected by DPM 2012 SP1. Here are the list of deployments that are supported by DPM 2012 SP1.

1) Windows 2012 Hyper-V standalone

2) Windows 2012 Hyper-V CSV cluster

3) Windows 2012 Hyper-V standalone with remote Windows SMB file server (standalone or cluster)

4) Windows 2012 Hyper-V cluster with remote Windows SMB file server (standalone or cluster)

5) Windows 2008 R2 standalone

6) Windows 2008 R2 CSV cluster

As described in the TechNet article, Exclude VHD names can be added, removed or extended. Following command can be used to find out the current exclusions.

PowerShell

PS C:\> Get-DPMGlobalProperty –PropertyName HyperVPagefileExclusions

Note that, this is a DPM Global Property and so the values can be “set” only. So if additions or deletions to be done, first get the global property, modify string pattern, and then set the property again.

Update Rollup 2 for System Center 2012 Service Pack 1–DPM updates

J.C. Hornbeck — Thu, 11 Apr 2013 18:15:02 GMT

Update Rollup 2 for Microsoft System Center 2012 Service Pack 1 (SP1) is now available and includes the following updates for Data Protection Manager:

Issue 1: An express full backup job in System Center 2012 Service Pack 1 may stop responding on a Hyper-V cluster that has 600 or more virtual machines.

Issue 2: When a System Center 2012 Service Pack 1 item level restore operation is performed on a SharePoint host header-based site collection, the restore operation is unsuccessful.

Issue 3: When you open Data Protection Manager on a computer that is running System Center 2012 Service Pack 1, the Welcome screen does not indicate the correct version of Service Pack 1.

Issue 4: When you perform a disconnected installation of the Data Protection Manager 2012 Service Pack 1 agent, you receive the following error message: Protection agent cannot be installed on a machine where DPM is installed.

Issue 5: When you use Data Protection Manager 2012 Service Pack 1 for tape backup, you receive the following error message: The operation failed because of a protection agent failure. (ID 998 Details: The parameter is incorrect (0x80070057))

Issue 6: Backups of CSV volumes may be unsuccessful with metadata file corruption in Data Protection Manager 2012 Service Pack 1.

Issue 7: The Data Protection Manager console may require more time to open than expected when many client systems are being protected.

You can get all the details and a download link here:

KB2802159 - Description of Update Rollup 2 for System Center 2012 Service Pack 1 (http://support.microsoft.com/kb/2802159)

J.C. Hornbeck | Knowledge Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

How to provide redundancy for End User Recovery (EUR) of Distributed File System (DFS) shared data using DPM

J.C. Hornbeck — Mon, 25 Feb 2013 19:26:12 GMT

Hi, Michael Vargo here, and I wanted to take a minute and talk about how you can provide redundancy for End User Recovery of DFS shared data using Data Protection Manager.

The environment diagram below provides a baseline for an explanation of the options that can be utilized to provide redundancy for End User Recovery (EUR) of Distributed File System (DFS) shared data. Note that Microsoft does not “officially” support EUR redundancy with DPM but it can be achieved.

The decisions that must be made regarding which redundancy plan you will implement is primarily based on what resources in the environment are most likely to fail. Are you planning for a failure of a DFS server, a DPM server or an entire site?

In the event that a fileserver#1 should fail you will not need to have a separate copy of the DFS data backed up to another DPM server for users to be able continue accessing previous versions of DFS data. The clients accessing the DFS data will be redirected to fileserver#2 and can still access the recoverable data (previous versions) from DPM#1. However, DPM #1 will no longer be backing up the DFS data if fileserver#1 is down. This problem can only be addressed by having redundant DPM servers. If fileserver#1 will only be down for a short period, you may not want to make any changes to DPM and EUR.

If you are planning for a DPM server failure, you can establish redundancy through the use of secondary DPM protection or another DPM server protecting the replica of the DFS data on fileserver#2. Secondary protection is achieved by connecting the protection agent to DPM#1 from DPM#2. You will then be presented with a “Protected Server” option when enumerating data sources under DPM#1 from DPM#2 which allows you to create a redundant copy of the DFS data on DPM#2. You can then choose to “switch disaster protection” should DPM#1 go down. However, the use of a secondary DPM server provides no benefit to DFS protection and EUR. Switching protection does not recreate the required shares on the new primary DPM server or update any Active Directory objects. We will discuss these items in more detail shortly.

The information above is provided to explain why we recommend the use of a second DPM server that has no association with DPM#1 to provide redundancy for DFS backups and EUR. DPM#1 backs up DFS data on fileserver#1 and DPM#2 backs up the replicated DFS data on fileserver#2. This will provide the ability to continue DPM backups of the DFS data and EUR access to the DFS data should fileserver#1, DPM#1 or site#1 become unavailable. DPM#1 and DPM#2 should both be configured to meet your data retention requirements. Optimally we would enable EUR for DPM#1 and DPM#2, but DPM only supports enabling EUR on one DPM server at a time in Active Directory when protecting the same DFS shares from separate DPM Servers. You will need to disable EUR for DPM#1 by deleting the AD objects that get created when enabling EUR in the event we need to implement the disaster recovery plan.

There are two categories of items that get added when enabling EUR. The first is a set of AD objects that get created with the Extension of the AD schema as a result of enabling EUR. The second is a set of shares that get created on the DPM server.

The first AD object that gets created is cn=ms-sharemapconfiguration,cn=system,dc=domain,dc=local object. This gets created as a result of running DPMADSchemaExtension.exe . It is available on the DPM server in the c:\program files\Microsoft DPM\DPM\End User Recovery directory. It is run when enabling end-user recovery from the DPM options on the End-user Recovery tab. We frequently see issues where enabling end-user recovery fails when run from DPM where it fails with a message similar to “The Active Directory could not be configured.” You can also copy DPMADSchemaExtension.exe to a domain controller and run it manually as a user who is a member of both the "Schema Admins" and "Enterprise Admins" security groups.

The additional items will not be created until you successfully synchronize the DPM server with the protected DFS data after the schema extension. After the synchronization job you will see an object created under the cn=ms-sharemapconfiguration container for each DFS name space protected by DPM. It has a name in the format CN=GUID and a class of ms-srvShareMapping. The import information in this object includes the ms-backupSrvShare attribute which points to the DPM server that is protecting the DFS data and the ms-productionSrvShare which indicates the DFS node that is being protected by DPM.

The second set of items created upon the completion of a synchronization job after the schema is update are shares on the DPM server. These are the shares that users’ access when viewing the “previous versions” tab in the properties of an object on a EUR enabled DFS share. There will be one share for each protected DFS namespace. The screen shot below shows the shares for DFS namespaces Namespace1 and public. They are associated the name space on Sharepoint01 with the location of the replica of the files in the DPM storage pool.

The AD objects under cn=ms-sharemapconfiguration,cn=system,dc=domain,dc=local will automatically be removed if you uncheck “enable end-user recovery” from the DPM options on the End-user Recovery tab. However, if the DPM server has crashed or is otherwise unavailable you must manually remove these entries. The recommended tool to access and remove these objects is ADSIedit.msc. It will allow you to drill down to the cn=ms-sharemapconfiguration,cn=system,dc=domain,dc=local container and see all of the child objects that represent each of the DFS name spaces. All of the child objects representing the failed DPM#1 server must be removed before you enable EUR on DPM#2.

You can use repadmin.exe to create a query that will list all of the AD objects associated with the DFS name space being protected by a failed DPM server.

repadmin /showattr dc01 ncobj:domain: /filter:"(&(objectclass=ms-srvsharemapping)(ms-productionsrvshare=\\sharepoint01\namespace1))" /subtree

The above command would connect to a DC named dc01 and dump all attributes for all objects with an objectclass of ms-srvsharemapping where the ms-productionsrvshare attribute contains a value of \\sharepoint01\namespace1.

You could limit the output with the /atts: option to just dump specific values from the object. For example:

repadmin /showattr dc01 ncobj:domain: /filter:"(&(objectclass=ms-srvsharemapping)(ms-productionsrvshare=\\sharepoint01\namespace1))" /subtree /atts:name > ms-productionshare.txt

Michael Vargo | Senior Support Escalation Engineer | Microsoft CTS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv

Hotfix: Backups are unsuccessful when you use Windows Azure Online Backup

J.C. Hornbeck — Tue, 19 Feb 2013 16:42:30 GMT

When you try to back up data by using Windows Azure Online Backup, is the backup process is unsuccessful? If so this hotfix may be for you.

The Windows Azure Access Control Service (ACS) certificate uses a new certification authority (CA) chain and this can prevent communication between it and the Windows Azure Online Backup Agent. This communication fails because the backup agent cannot verify the server certificate that is presented by the Azure ACS service.

For all the details and a download link for the fix please see the following:

KB2816671 - FIX: Backups are unsuccessful when you use Windows Azure Online Backup (http://support.microsoft.com/kb/2816671)

J.C. Hornbeck | Knowledge Engineer | Management and Security Division

Get the latest System Center news on Facebook and Twitter:

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv