The Storage Team Blog about file services and storage features in Windows and Windows Server.
Hi folks, Ned here again. Today I talk about a new feature in Windows Server 2012 R2 DFSR, restoring preserved files. DFSR has had conflict, deletion, and preexisting file handling since its release, but until now, we required out of band tools to recover those files. Those days are done: we now have Get-DfsrPreservedFiles and Restore-DfsrPreservedFiles.
Readers of my posts on AskDS and FileCab know I like to take the edge off. I figure everyone should learn we aren’t a corporate hive mind, just a collection of humans striving to make great software. For now though, I’ll keep the cheap laughs to a minimum, as when you’re restoring data it’s usually causing some hair-pulling and tears.
Let’s get to it!
DFSR uses a set of conflict-handling algorithms during initial sync and ongoing replication to ensure that the appropriate files replicate between servers, as well as to preserve remote deletions.
The ConflictAndDeleted folder has a 4GB “first in/first out” quota in Windows Server 2012 R2 (it’s 660MB in older operating systems). The PreExisting folder has no quota.
When content moves to these folders, DFSR tracks it in the ConflictAndDeletedManifest.xml and PreExistingManifest.xml. DFSR mangles all files and folders in the ConflictAndDeleted folder with version vector information to preserve uniqueness. DFSR also mangles the top-level files and folders in the PreExisting folder, but leaves all the subfolder contents unaltered.
The result is that it can be difficult to recover data, because much of it has heavily obfuscated names and paths. While you can use the XML files to recover the data on an individual basis, this doesn’t scale. Moreover, if you just set up replication and accidentally chose the empty replicated folder as primary, you want all those files back out of preexisting immediately with a minimum amount of fuss.
In this walkthrough, I create a replicated folder that intentionally contains conflicted, deleted, and preexisting content for restoration testing. To follow along, you need a couple of Windows Server 2012 R2 computers with DFSR installed: let’s call them SRV01 and SRV02.
1. On server SRV02 only, create C:\RF01 and add some test files (such as all the contents of the C:\Windows\SYSWOW64 folder). Make sure it has some subfolders with files in it.
2. Create a new replication group named RG01 with a single replicated folder named RF01. Add SRV01 and SRV02 as members to the RG and replicate the C:\RF01 directory replicated folder. You must specify SRV01 the primary (authoritative) server in this case. I recommend the new DFSR Windows PowerShell for this, so all my examples below go that route.
Important: Choosing SRV01 to be primary is an “intentional mistake” in this scenario, as I want to create preexisting files on the downstream server. Ordinarily, you would make SRV02 primary, as it contains all the data to replicate and SRV01 contains none.
3. Verify that replication completes – in this case, SRV01 logs DFSR event 4112 and SRV02 logs DFSR event 4104. All files in c:\rf01 will appear to vanish from SRV02.
4. Create a test BMP and RTF file on SRV01 in C:\RF01. Create a subfolder, and then create another BMP and RTF test file in that subfolder. Make sure those files replicate. Note: BMP and RTF are convenient choices because they are default file creation options in the Windows Explorer shell. In addition, unlike Notepad with TXT files, their editors follow standard conventions for opening and closing files.
5. Pause replication between SRV01 and SRV02 using the Suspend-DfsReplicationGroup cmdlet. For instance, to pause for 5 minutes:
6. Modify the top-level BMP file on both servers (the same file), making sure to modify the SRV02 copy first (i.e. earlier, so that it will lose the conflict).
7. Let replication resume from the suspension in step 5.
8. Create another BMP file on SRV01, and then delete that file and the subfolder you created in step 4, along with its contents.
9. Validate that DFSR deletes the files from both servers that the file you modified on both servers now holds the version that came from SRV01.
10. On SRV01, manually recreate the same-named file that you previously deleted in step 8, ensure it replicates to SRV02, then delete it from SRV01 and verify that the deletion replicates. This creates a scenario with multiple deleted file versions.
11. Copy the following folders and files elsewhere as a backup on SRV02:
Note: Restoring preserved files does not require running the DFSR service or an active RG/RF. You can operate on the preserved data independently on Windows Server 2012 R2, including locally on a Windows 8.1 Preview computer running RSAT with a local copy of the DfsrPrivate folder. Backing up the preserved files prior to restoration is a best practice, since restore operations are destructive by default (they move files instead of copying).
Now I have some conflicts, some deletions, and some preexisting files and folders, all on SRV02 in the c:\rf01\dfsrprivate folder. Let’s go to work.
The Get-DfsrPreservedFiles cmdlet tells you everything you want to know about files and folders in the ConflictAndDeleted and PreExisting stores, according to the XML manifests. Let’s inventory the preserved files, in order to see which files DFSR saved and some details about them. All it needs is a single parameter called –path that points to the manifest.
1. On SRV02, see the conflicted and deleted files:
2. On SRV02, see the preexisting files:
3. On SRV02, retrieve only RTF files with their original path and new names:
4. On SRV02, see only conflicted files, when the conflict occurred, and what server originated the conflict:
No longer are you left wondering when a user deleted a file or from which server, nor if a particular file is still in the ConflictAndDeleted cache on the other nodes. It’s all there at your fingertips.
The Restore-DfsrPreservedFiles cmdlet can restore one or more files and folders in the ConflictAndDeleted and PreExisting stores, according to the XML manifests. All it needs is the manifest and a decision about how to recover the files and where to put them, but there are some additional options:
Important: By default, this cmdlet moves all files preserved files from PreExisting. It also moves the latest version of files from ConflictAndDeleted and removes the remaining ones (this is intentional, as otherwise every time you ran this cmdlet, you would restore an increasingly older version of conflicted files). We strongly recommend backing up the ConflictsAndDeleted and Preexisting folder before you use this cmdlet!
Let’s look at some examples of recovery.
Note: some testing can “break” the test environment we created above, because there will no longer be any files to restore once you move them. I recommend you make a few backup copies of your saved DfsrPrivate folder so you can go through this a few times.
1. On SRV02, move all preexisting files to their original location:
This is very quick, as I’m moving the data locally on the same volume. Look here when I restore ~1GB and 4,500 files and folders of SYSWOW64:
I am back in business in 2.3 seconds later, here.
2. On SRV02, copy all conflicted and deleted files to the original location, preserving all versions of the files so that users can decide which ones to keep:
3. On SRV02, move all versions of the preserved files verbosely to an alternate location for later analysis and manual restore:
Note: an unintended artifact of this type of “alternate location” operation is the creation of an empty set of folders based on the RF itself, but at one level deeper. You can ignore or delete that top folder named after the RF itself; it will contain only empty folders. This is something we might fix later, but it is totally benign and cosmetic.
4. On SRV02, move the newest version of all conflicted and deleted files to the original location, removing all versions of moved files from the ConflictAndDeleted folder, skipping any files that already exist and leaving them in the ConflictAndDeleted folder:
As you can tell, these cmdlets are very simple to use and can meet most data restoration needs. Now if Tim from Sales accidentally deletes his shared document that he started this morning – and is therefore not in the nightly backup – you have one more way to get it back.
Update May 2014: See it all in video! TechEd North America 2014 with live demos and walkthroughs:
I hope this is another good example of why you should move to Windows Server 2012 R2.
- Ned “self-preservation” Pyle
Very practical guide Ned! Thank you and keep them coming!
Thank you for the clear illustration.
Hello Ned, thanks for your very informative posts.
I am looking at how to design a distributed file server infrastructure between several locations separated by a WAN. The two technologies I am looking at are DFS-R and Double-Take High Availability. The later is a one-way only replication but it has nice alerting if there are replication problems (e.g. e-mail, SNMP trap).
Is there a way with DFS-R to get better error alerting (by better, I mean closer to real-time, more informative as to what is in sync and what is not, and more alerting mechanisms)? I have been using a script based on blogs.technet.com/.../automating-dfs-replication-health-reports.aspx , which works OK, but is a once-a-day thing and really just tells me a binary of if the replication is 100% or not. (It uses dfsradmin to generate a health report and then looks for XML strings like "<backlogInbound>-[1-9][0-9]*</backlogInbound>" to check if the replication is backlogged.)
Also, is there a way to configure DFS-R for disaster-recovery scenarios where we would have users only using one master file share, with the DR side never to be used or changed by users, but we could switch people over still quickly if the primary has failed? Is a DFS-R read-only replication partner the way to go here?
Thanks for your advice!
Tim Miller Dyck
As always really interesting (and fun) reading, however I have to ask; will there (ever) be a solution for DFSR and file locking ?
Reading some earlier posts by you, such as this one way back in 2009: blogs.technet.com/.../understanding-the-lack-of-distributed-file-locking-in-dfsr.aspx
and in (WS) 2012 blogs.technet.com/.../dfs-replication-improvements-in-windows-server-2012.aspx
lacking a file locking feature in DFSR in Windows Server 2012 R2 kinda [insert profanity here] big time...will there ever be a file locking feature or are will forever doomed to use (very) expensive third party products like PeerSync ?
@ Tim. PowerShell would make that a bit easier now, but the fundamental alerting issue remains. SCOM 2012 does have backlog monitors and you can configure alerts on those.
Read-only fits the bill there, as long as you keep in mind that DFSR is not a synchronous replication system (there is always data latency, even if split second) nor a backup system (if I delete the real copy, the DR copy will be deleted as well!).
@ Brook. Never say never, but currently we are not tackling the file locking issue. It is - of course - the #1 ask I get, so we are aware of everyone's feelings on it. It's just a big thing to tackle and we haven't had the resources.