Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

NedPyle [MSFT] — Wed, 27 Apr 2011 13:00:38 GMT

Yes, the method that got you to a virtualized DC doesn't matter. The issue isn't specific to virtualization either - if you have a SAN that performs "volume snapshots" of physical servers you could run into these scearios as well. Anything that makes an entire server time travel backwards as a single image is at risk.

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

Janson Ragon — Wed, 27 Apr 2011 09:07:37 GMT

Hi Ned,

Does this also apply to scearios where you have done a P2V or V2V of a server either using VMM 2008 R2, or using a more manual method such as Disk2VHD?

Cheers

Janson

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

NedPyle [MSFT] — Mon, 25 Apr 2011 13:46:09 GMT

It's only documented internally at this point, AFAIK. I'm sure if a backup vendor asked us how to do this we'd explain it to them though. Ultimately, you're just setting:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Database restored from backup = 1

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

Pronichkin — Sat, 23 Apr 2011 04:58:50 GMT

> Windows Server 2008 and 2008 R2 only implement protection this as part of Hyper-V integration components so third party full disk image restores or other virtualization products have to implement it themselves.

I've asked before and I'll ask it again :) Do we document how we did this and how they are supposed to do this? I mean is there any special sauce or we just need to call the NTDS VSS writer in the end of restore and make it aware that the restore took place?

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

NedPyle [MSFT] — Thu, 21 Apr 2011 21:35:44 GMT

Ah I see what you mean - incrementals are totally valid as well as long as you also have the full of course. There's no expectation that anyone needs to always run Full (capital F) backups everytime, just that the backups fully cover all the files. I modified the article to be clearer about this.

There is an easier solution after restoring a full VHD but I am loathe to talk about it publically as it is so often absued and used incorrectly. I am going to anyway though:

A D2 burflag in FRS resets the database. If you were to delete the DFSR databases on all the volumes after the restore, you will fix the problem. This causes a complete non-authoritative sync and fixes everything up just like how DCs do when their invocation ID is reset. Now, the reason I don't like to bring this up is because you are now potentially syncing many terabytes of data, and you are definitely LOSING all conflicts (meaning that data that was newer on this server is arbitrarily deleted forever). It's not a great solution. You can't mark the restore authoriattive either when the databse is deleted; you'd insetad have to do (as you mentioned) a database deletion as well as recreate your RG and set his server primary all over again.

I would also make mention that DFSR performance is going to be a lot better with a passthrudisk than a VHD file; it's very IO dependent. And if you are using passthru disks you have nothing to worry about. Especially since DFSR keeps a database on every volume that is replicating files.

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

Jeff.Miles — Thu, 21 Apr 2011 21:03:32 GMT

That does make sense. Ultimately I'm just trying to determine if we need to modify our backup strategy for our DFSR server.

So if we lose our DFSR member guest in Hyper-V, and we restore the C_guest.vhd (OS) and D_files.vhd (1.5 TB of files + DFSR database), and then boot up that guest disconnected from the network, and then do a further restore of a backup taken inside the guest before it failed, then when we connect it back to the network, it should begin communicating with the other DFSR members properly?

Will it come back online as non-primary, so that it receives changes made on other members while it was down, even if the downed member was the hub?

I think where my current confusion lies is in that second "inside guest" restore; You specify that the backup should be "full (and specifically including System State) backup using VSS". Does that mean incremental backups of the DFSR folder target aren't valid in the disaster recovery scenario? If we're backing up using UNC path to the folder target on the guest with a backup agent, is that going to be sufficient to get the DFSR member back online?

If not, it almost sounds like it'd be easier to restore the vhd's as a new guest, and rejoin the replication group as a new member, with the 1.5TB vhd acting as pre-seeded data for an initial replication.

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

NedPyle [MSFT] — Thu, 21 Apr 2011 20:20:28 GMT

Hi Jadus01,

1. Most backup software allows you to exclude files by path, extension (Windows Server Backup does, for example) so you could block them that way if you were inclined.

2. If you were NOT so inclined in #1 above and wanted to back up the VHDs as files, #2 still saves you. If you backup *inside* a guest OS (i.e. the guest OS is running backup software and is online), any restore of the VHD file is not terminal because you could then run the guest's own restore internal to the guest. If you had 100 VHDs on a host and only 2 of them were DCs, this might make sense as 98 of your backups can be restored with teh full disk and the two DCs can be fixed with their internal backups after the fact.

We want you to treat the multi-master guest OSes as 'real' computers, not as VHD files. Full VHD backups simply don't work on them like they do with a simple file server or web server or app server; multi-master is too complex and interrelated. Plus even in the "good" case where we have USN rollback protection in place, your "restore" is losing all changed data. It's more like an amputation that saves the patient's life but costs him a hand.

Now: if you had all the multi-master related guests on the same host, you could restore a full host and not see any problems. But if you;re doing that, you're breaking a principle of availability - all your eggs are in one basket. So for different reasons, that's just as dangerous.

Thanks, and continue let me know if I'm not making sense. I want to make sure everybody gets this and understands what happens if you restore full VHDs.

re: Disk Image Backups and Multi-Master Databases (or: how to avoid early retirement)

Jeff.Miles — Thu, 21 Apr 2011 20:04:54 GMT

Thanks for this Ned, this is highly useful information. I've got a Server 2008 R2 Hyper-V guest that serves as our primary DFSR hub and has VHD's for each folder target/replication group. Couple questions though:

1. If I understand correctly, we shouldn't be using any tool to back up these VHD's from the host (using Symantec Backup Exec, DPM, etc) correct? Is there a recommendation to back up the guest OS without backing up the VHD, so in the event of a OS failure, we can restore without pooching DFSR?

2. You stated "It is ok to take these kinds of outside backups as long as you are also getting a backup that runs within the running multi-master guest computers." How would this scenario be safer in a restore process? Would it be safe to restore the VHD backup, and then the file-level backup overtop of that? I just don't understand how this 'becomes' safe.

If that does work, we'd still do the redundant process since a full backup & restore of our 1.5TB VHD takes much less time than a file level backup & restore from inside the guest. We could restore our weekly full VHD + any file-level differentials in short order.