Well, I broke down and bought a new workstation last weekend. It is a dual core AMD64 with 2 gigs of RAM and a 300 GB SATA drive and a 19 inch flat panel. Nice box! I lived with a Dell Precision 610 for about 4 years so it was definitely time for an upgrade. I started transferring files from the old workstation to the new one. I decided to migrate my Win2k3 Replica DC running in Virtual Server from the old workstation to the new as well. Since I was actually logging onto the local "old workstation", I shut down the "virtual" domain controller and stopped the Virtual Server service. I then made a remote desktop connection to the old from the new and started copying files... I had shares open from the old to the new and from the new to the old...copying about 10GB of pictures and about 7 GB of songs...life is good. Well, I had some serious disk space problems on the old box and I decided to delete the "offline backups" of my Virtual Server (VHD/VMC files). Life is still good. After all the files were copied off the old box, I decided to fire up my replica DC in Virtual Server...but it did not start. I checked the Virtual Server management interface and there was an error "file not found". Uh oh...I deleted the actual working files! Well, a quick search for *.vhd on that box produced nothing. I then looked on my SBS box and found a pair of VHD/VMC files dated 12-23-2005. I made that backup when blogging my Migration Scenario. Good, well within the tombstone period. I was lucky! I then install Virtual Server 2005 on my new box, copy the files over and boot her up. She boots up fine, life is good again. Whew... Then just to be sure, I go check event viewer on my recently restored "virtual" domain controller and am presented with the lovely set of event below:
Event Type: ErrorEvent Source: NTDS ReplicationEvent Category: (5)Event ID: 2095Date: 2/3/2006Time: 6:50:44 PM iUser: NT AUTHORITY\ANONYMOUS LOGONComputer: WIN2KDCDescription:During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is themproper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC. Remote DC:7e3836bf-dbd3-4c43-80d7-679ec27932c8 Partition:DC=company,DC=local USN reported by Remote DC:472083 USN reported by Local DC:369571
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: ErrorEvent Source: NTDS GeneralEvent Category: (12)Event ID: 2103Date: 2/3/2006Time: 6:50:44 PMUser: NT AUTHORITY\ANONYMOUS LOGONComputer: WIN2KDCDescription:The Active Directory database has been restored using an unsupported restoration procedure. Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused. User Action See previous event logs for details.
Event Type: WarningEvent Source: NTDS GeneralEvent Category: (5)Event ID: 1113Date: 2/3/2006Time: 6:50:44 PMUser: NT AUTHORITY\ANONYMOUS LOGONComputer: WIN2KDCDescription:Inbound replication has been disabled by the user.
Event Type: WarningEvent Source: NTDS GeneralEvent Category: (5)Event ID: 1115Date: 2/3/2006Time: 6:50:44 PMUser: NT AUTHORITY\ANONYMOUS LOGONComputer: WIN2KDCDescription:Outbound replication has been disabled by the user.
Event Type: WarningEvent Source: NTDS GeneralEvent Category: (9)Event ID: 1173Date: 2/3/2006Time: 6:50:44 PMUser: NT AUTHORITY\ANONYMOUS LOGONComputer: WIN2KDCDescription:Internal event: Active Directory has encountered the following exception and associated parameters. Exception:e0010002 Parameter:0 Additional Data Error value:8451 Internal ID:108132e
So now what?? Clicking on the "links" in the events took me to these articles: http://support.microsoft.com/kb/875495/en-us and http://support.microsoft.com/kb/885875/en-us. Anytime you seen an article with "USN Rollback" in the title, smoke 'em if you got 'em. I alway have to go consult with Mark Stanfill (from Inside SBS fame, http://blogs.technet.com/sbs) when someone even mentions USN Rollback. Ok, those are scary events. I immediately shut down the "virtual" replica domain controller and boot up into Directory Services Restore Mode and go check my backups. YES, I even do backups of my "virtual" domain controller. I have one from 1-30-2006. I kick off NTBACKUP, browse over to my old workstation (where all my backups go), load up the BKF file and do a system state restore. 15 minutes later, I reboot and am presented with this event in the Directory Service event log:
Event Type: InformationEvent Source: NTDS ReplicationEvent Category: (5)Event ID: 1109Date: 2/3/2006Time: 7:16:30 PMUser: N/AComputer: WIN2KDCDescription:Active Directory has been restored from backup media, or has been configured to host an application partition. The invocationID attribute for this domain controller has been changed. The highest update sequence number at the time the backup was created is as follows. InvocationID attribute (old value):367df4eb-81d5-4903-b533-6cdf4510919f InvocationID attribute (new value):61ebb45e-641d-4872-907c-caf57b286c5b Update sequence number:462230 The invocationID is changed when a domain controller is restored from backup media or is configured to host a writeable application directory partition.
Life is good again! This was NOT luck, I had planned for this. If you are using Virtual Server in a production environment, please...please...please...please read this article and this article AND start doing system state backups on your virtual servers as well. It is NOT good enough to simply do an offline backup of the VHD/VMC files (meaning shut down the virtual server and copy the files). My practice is to do a system state twice a week and an offline backup once per month. Of course on my SBS box, backups are daily. With those three, I sleep well at night knowing my 2 user environment is safely backed up and restorable. Also, the above errors could also happen with "other" types of backups (insert GHOST). I love creative backup solutions! Make sure they work! Test your restore procedure. Test it again! In summary, I kept my replica intact with the built-in software (NTBackup) and a 470 MB file on an IDE drive. Anyone checked the prices of drives lately?
Life is good again...now I can go watch all those movies I recorded with my new Media Center PC <while I was restoring a domain controller that was running on the same box in Virtual Server>!