[Today’s post comes to us courtesy of Shawn Sullivan]

There is a common misunderstanding as to what a proper backup and restore procedure should be for an EBS 2008 Management or Messaging server running as a virtual machine. We see many customers who are taking “snapshots” in their chosen hypervisors and are under the assumption that they can successfully restore them in disaster recovery or migration scenarios. Another common method we see is the use of 3^rd party backup solutions that create volume images of the server and store them in a centralized network location. This post will shed light as to why methods like this are unsupported, what the supported procedures are, and how to recover from an improper restoration of the SBS server.

As you are aware, the EBS 2008 Management and Messaging servers are domain controllers in the root of the Active Directory forest. The contents of an Active Directory database and changes to them are tracked and replicated using metadata attributes called Update Sequence Numbers (USNs) and High Watermark Vectors (HWMVs). When a domain controller is restored from backup, another metadata attribute called an invocation ID is used to notify other domain controllers in the network that this server is indeed being reintroduced into the current domain from a previous state. Depending on the type of recovery you are performing, the restored domain controller’s copy of Active Directory will either absorb the up-to-date changes from its replication partners (non-authoritative) or its replication partners will absorb its copy (authoritative).

Every restoration of a domain controller must follow the above framework. The only backup type that will properly preserve critical metadata attributes mentioned above is the System State Backup. The only restore type that will utilize these attributes to properly reintroduce the server into the existing domain is the System State Restore. Any other backup/restore method, including virtual machine snapshots, volume image snapshots, and offline copy backups of the NTDS database system files will put your server in a state called USN rollback if performed. We will discuss USN rollback in more detail in a little bit.

There are definitive methods for backing up the system state data of an SBS server that are supported by Microsoft. The following strategies available to you are:

• Windows Server Backup: You can use the built-in GUI backup utility to backup your server at the per volume level. If you choose to Enable system recovery, you will backup all drives that include OS data required for system state restore. When it comes to recovering your data, you can target just the system drives, just the system state data, single applications (Sharepoint or Exchange) or the entire server including the ability to perform a bare metal restore.
• Wbamin: This application is used beneath the surface by the SBS 2008 Backup Wizard and Windows Server Backup. Available as a command line utility, it allows you to backup just the system state data in your single targeted backup job. This is always useful in scenarios where you are about to make a change to critical system data and you want to protect yourself first without having backing up the entire drive. Please see our previous post that includes the “how to” steps.
• Data Protection Manager
• Any third party backup application that takes an Active Directory-aware system state backup using Microsoft supported backup APIs or by using Microsoft’s provided Volume Shadow Copy VSS writers (NTDS writer). Check with the particular vendor to verify whether the application meets these criteria.
• Windows Server Backup on the Hyper-V host machine can be enabled to backup the entire Hyper-V infrastructure, including all guest virtual machines. The following KB article gives all of the details, however here are a couple important points and caveats:
  • You must first register the Hyper-V VSS writer with Windows Server Backup through a registry edit.
  • You cannot target individual Hyper-V components or virtual machines for backup or restore. It is an all or nothing operation.
  • Virtual machines must have Integration Services installed and the Backup (volume snapshot) service must be enabled in order to be backed up.
  • Virtual machines that contain dynamic disks will not be backed up through this method.
    
    We recommend that you use this method to supplement your regularly scheduled backups that you should be taking from within the virtual machine guest OS itself. Due to its inability to target individual virtual machines for backup and restore, we do not recommend that you use this as your primary method.

USN Rollback

When a proper system state restore is performed, the invocation ID attribute of the restored database is reset, which notifies each replication partner that this server has indeed been restored from a previous state. Replication attempts from the restored DC to its partners would include this new invocation ID along with data that it has previously replicated. As long as the partner receives the previously acknowledged USN with a new invocation ID, there are no issues.

USN rollback happens to a domain controller when it replicates previously acknowledged USN with the same invocation ID. In this circumstance, there is no way to ensure that the changes pushed from this particular DC are the latest. When this inconsistency is recognized by its partner, they will notify it. The source server will then pause its netlogon service and disable outbound replication. This is to prevent any changes originating from this server’s copy of AD from being pushed out to the domain. At this point, you must remove this server from the existing domain and promote it back in as a clean domain controller. The following KB article goes into greater detail, including the events you will receive and includes all of the recovery steps that are required.

Note: Besides restoring a DC improperly, disconnecting a domain controller from the domain beyond 180 days (tombstone lifetime) and then bringing it back into the network will result in the same scenario.