[Today’s post comes to us courtesy of Shawn Sullivan]
There is a common misunderstanding as to what a proper backup and restore procedure should be for an EBS 2008 Management or Messaging server running as a virtual machine. We see many customers who are taking “snapshots” in their chosen hypervisors and are under the assumption that they can successfully restore them in disaster recovery or migration scenarios. Another common method we see is the use of 3rd party backup solutions that create volume images of the server and store them in a centralized network location. This post will shed light as to why methods like this are unsupported, what the supported procedures are, and how to recover from an improper restoration of the SBS server.
As you are aware, the EBS 2008 Management and Messaging servers are domain controllers in the root of the Active Directory forest. The contents of an Active Directory database and changes to them are tracked and replicated using metadata attributes called Update Sequence Numbers (USNs) and High Watermark Vectors (HWMVs). When a domain controller is restored from backup, another metadata attribute called an invocation ID is used to notify other domain controllers in the network that this server is indeed being reintroduced into the current domain from a previous state. Depending on the type of recovery you are performing, the restored domain controller’s copy of Active Directory will either absorb the up-to-date changes from its replication partners (non-authoritative) or its replication partners will absorb its copy (authoritative).
Every restoration of a domain controller must follow the above framework. The only backup type that will properly preserve critical metadata attributes mentioned above is the System State Backup. The only restore type that will utilize these attributes to properly reintroduce the server into the existing domain is the System State Restore. Any other backup/restore method, including virtual machine snapshots, volume image snapshots, and offline copy backups of the NTDS database system files will put your server in a state called USN rollback if performed. We will discuss USN rollback in more detail in a little bit.
There are definitive methods for backing up the system state data of an SBS server that are supported by Microsoft. The following strategies available to you are:
When a proper system state restore is performed, the invocation ID attribute of the restored database is reset, which notifies each replication partner that this server has indeed been restored from a previous state. Replication attempts from the restored DC to its partners would include this new invocation ID along with data that it has previously replicated. As long as the partner receives the previously acknowledged USN with a new invocation ID, there are no issues.
USN rollback happens to a domain controller when it replicates previously acknowledged USN with the same invocation ID. In this circumstance, there is no way to ensure that the changes pushed from this particular DC are the latest. When this inconsistency is recognized by its partner, they will notify it. The source server will then pause its netlogon service and disable outbound replication. This is to prevent any changes originating from this server’s copy of AD from being pushed out to the domain. At this point, you must remove this server from the existing domain and promote it back in as a clean domain controller. The following KB article goes into greater detail, including the events you will receive and includes all of the recovery steps that are required.
Note: Besides restoring a DC improperly, disconnecting a domain controller from the domain beyond 180 days (tombstone lifetime) and then bringing it back into the network will result in the same scenario.