Michael Vargo | Senior Support Escalation Engineer
UPDATED 1/6/2014 – New hotfixes available
Some customers are experiencing failures on Windows Server 2012 Hyper-V hosts that utilize Cluster Shared Volumes (CSVs) when backing up virtual machines (VM’s) using System Center 2012 Data Protection Manager (DPM 2012) with Service Pack 1 (SP1) or later.
When this occurs the following error is logged on the Hyper-V hosts.
Log Name: System Source: Microsoft-Windows-FailoverClustering Event ID: 5120 Logged: <date/time> Details: Cluster Shared Volume 'Volume2' ('ClusterStorage Volume 2') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.
You may also see the VM’s pause temporarily, go in to a paused state, shut down, see memory spikes or CSV go into a failed state on the Hyper-V hosts. Configuring serialized backups doesn't help the problem.
It has been determined that any backup product using shadow copies results in the same errors, thus this isn't necessarily a DPM 2012 issue.
The Windows and DPM teams have released a series of fixes to address some general Cluster, CSV backup and DPM issues and are now available for download.
KB2886362 - Hotfix for System Center 2012 SP1 Data Protection Manager (http://www.microsoft.com/en-us/download/details.aspx?id=40318)
NOTE The above fix requires a prerequisite fix: 2836751 - Description of Update Rollup 3 for System Center 2012 Service Pack 1 (http://support.microsoft.com/kb/2836751/EN-US)
KB2878635 - Update is available that improves the resiliency of the cloud service provider in Windows Server 2012: December 2013 (http://support.microsoft.com/kb/2878635/EN-US)
Also see KB2784261 - Recommended hotfixes and updates for Windows Server 2012-based Failover Clusters (http://support.microsoft.com/kb/2784261/EN-US)
Michael Vargo | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division
Get the latest System Center news on Facebook and Twitter:
System Center All Up: http://blogs.technet.com/b/systemcenter/ System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/ System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/ System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/ System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/ System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm
Windows Intune: http://blogs.technet.com/b/windowsintune/ WSUS Support Team blog: http://blogs.technet.com/sus/ The AD RMS blog: http://blogs.technet.com/b/rmssupp/
The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/ The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/ The Forefront TMG blog: http://blogs.technet.com/b/isablog/ The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/
Same here. KB2878365 fixed it for us on our 2012 Cluster. Now we're 2012 R2 with DPM 2012 R2 the issue looks like it's returned. Feeling the R2 team were unaware that this was an issue in 2012.Also just like Boudewijn commented we have 3 CSVs and the errors only appears on one. Glad I saw this post as I was about to move 30+ machines off of that CSV, as I thought it must be bad. Now I'm thinking that wouldn't fix anything.
hi,any fix for this issue on windows 2012 r2 cluster?i'm experiences this issue with Veeam backup.
I have this issue with 3 node WS2012R2 HV cluster with one CSV and DPM2012R2.
The workaround for me is to place all VMs on one node (the current CSV owner), turn off dynamic optimization in VMM and then backups works fine. You can try this. Please give us a feedback if it works for You too.
I have ticket opened in Microsoft, so if i Have any sollution will publish it.
MS says that it's known issue with EQL Hardware Provider and are cooperating with DELL to solve this. I think I fou nd another workaround. You can distribute VMS on all nodes and have Dynami Optimization on if you limit DPM server and agents to backup
only one VMs i time. You have to change/create registry key inHKLM\Software\Microsoft\Microsoft Data Protection Manager\2.0\Configuration\MaxAllowedParallelBackups. You should have there DWORD Microsoft Hyper-V and have to set it's value to 1.
Second thing to do is to change file DsResourceLimits.xml located on DPM server in C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\bin and on the Hyper-V hosts in C:\Program Files\Microsoft Data Protection Manager\DPM\bin. Change MaxLimit value to
1 in line with job type="8": . So it shoul look like:
My first backup off 19 VMs on 3 node cluster succeeded.