Authoritative restore issue with LVR enabled attributes

  • Article

I recently came across a DR scenario a colleague worked on that I thought should be documented so administrators can adequately plan for this or prevent it from occurring in the first place. This scenario can only happen in a forest that started life as a W2K forest and was upgraded to W2K3 FFL.

This is fairly long and may be better suited for 2-3 blogs, but I decided to cover this in one blog.

   

Key acronyms in this blog

   

LVR - Link Value Replication

Legacy - attribute value without individual replication metadata

Present- a value with additional replication metadata attached

Absent - a deleted value with additional metadata attached

TSL - tombstone lifetime

   

The restore problem

The deep technical problem description: Authoritative restores of objects, to recover the forward link portion of linked attributes, are not sufficient to return the forward link attribute contents to its former state. This problem can only occur in forests that have existed in W2K FFL and have been upgraded to W2K3 FFL. When I say existed in W2K FFL, I really mean there are objects with populated forward links prior to the upgrade to W2K3FFL.

The easier to grasp practical example problem description: Authoritative restores of group objects to recover removed group memberships are not sufficient for specific environments that have upgraded their forests to W2K3 FFL.

Note: Technically this restore problem applies to any LVR enabled attribute. See the next section for a description of LVR.

Note: The removed membership does not involve deleting the member objects, only removing them from a group's membership.

   

   

What is Link Value Replication?

Rather than reproduce what is already documented here, I will briefly summarize LVR replication.

LVR replication changes the smallest unit of replication for multi-valued linked attributes (the ones with distinguished name values) to a single value. This change has a few benefits:

  • multi-valued linked attributes can now safely grow beyond 5000 entries
  • replication of additions and deletions are more efficient
  • prevention of update losses due to the same attribute being updated at multiple DCs at roughly the same time.

   

   

The accidental modification and the recovery goal

Reminder: This scenario exists for any LVR enabled attribute. The example will focus on group membership.

An AD administrator has removed users from a set of groups (users not deleted, only removed from groups). The administrator realizes some of these group membership changes should not have occurred. The administrator's goal is to return these group memberships to their prior state.    

   

How should an administrator recover from this situation?

Well, the following is one way to recover from this situation. The following assumes the backed up system and restore to system are running W2K3 SP1. Perform a system state restore to alternant hardware isolated from production. The members were not deleted, so isolation prevents loss of data to the user accounts like passwords if changed after the backup was taken.

Authoritatively restore the affected users rather than the modified groups. The idea here is to use ldf output created by NTDSUTIL to restore the group membership. Transport the LDF output created by NTDSUTIL to the production environment and import it to recover.

Reminder: I kept this example simple in that only users are members of groups. Of course, other groups as well as computers can be members as well.

   

   

Prevent this scenario from ever happening

This recovery certainly seems like a lot of work when it can be prevented in the first place.

Preventing this type of DR situation would require all LVR enabled attributes to have all of their values converted from legacy to present.

This can be accomplished by removing all the values and re-adding them.

   

   

   

My favorite part…the diagnosis of why authoritative restore of group object is not sufficient

Environment:

W2K3 2 DC forest with a group having 9 members (9 legacy and 1 LVR enabled absent entry)

(present and absent are LVR enabled entries, legacy are not LVR enabled members)

W2k3entr2-vm11

W2k3entr2-vm12

   

Replication metadata from each DC on the relevant group object prior to accidental modification

C:\>repadmin /showobjmeta w2k3entr2-vm11 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

12 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

   

14062 Default-First-Site-Name\W2K3ENTR2-VM11 14062 2008-01-17 14:19:58 10 member

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 instanceType

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

-----snip-----

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 14:45:23 Default-First-Site-Name\W2K3ENTR2-VM11 14151 14151 1

CN=user1,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

-----snip-----

   

C:\>repadmin /showobjmeta w2k3entr2-vm12 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

12 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

12428 Default-First-Site-Name\W2K3ENTR2-VM11 14062 2008-01-17 14:19:58 10 member

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 instanceType

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

   

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 14:45:23 Default-First-Site-Name\W2K3ENTR2-VM11 12487 14151 1

CN=user1,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip----

Notice above there are 10 entries associated with the member attribute of DOM1\GLOBALGROUP. The last 7 were snipped for the sake of brevity.

Important: In DSA.msc or LDP.exe or your favorite LDAP reader, you should see only 9 values…the 9 members of this group. The ABSENT entry is similar to a tombstoned object where it references the knowledge of a removed value in a LVR enabled attribute and will be garbage collected after TSL.

   

System state backup taken of w2k3entr2-vm12

   

The accidental modification of data in AD

One user is removed from the group (user object not deleted)

C:\>repadmin /showobjmeta w2k3entr2-vm11 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

12 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 objectClass

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 cn

14062 Default-First-Site-Name\W2K3ENTR2-VM11 14062 2008-01-17 14:19:58 10 member

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 instanceType

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

   

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 14:45:23 Default-First-Site-Name\W2K3ENTR2-VM11 14151 14151 1

CN=user1,OU=userstore,DC=dom1,DC=root

ABSENT member 2008-01-17 15:13:00 Default-First-Site-Name\W2K3ENTR2-VM11 14178 14178 1

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

-----snip-----

   

C:\>repadmin /showobjmeta w2k3entr2-vm12 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

12 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 objectClass

12352 Default-First-Site-Name\W2K3ENTR2-VM12 12352 2008-01-17 14:15:11 1 cn

12428 Default-First-Site-Name\W2K3ENTR2-VM11 14062 2008-01-17 14:19:58 10 member

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 instanceType

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 14:45:23 Default-First-Site-Name\W2K3ENTR2-VM11 12487 14151 1

CN=user1,OU=userstore,DC=dom1,DC=root

ABSENT member 2008-01-17 15:13:00 Default-First-Site-Name\W2K3ENTR2-VM11 12511 14178 1

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip----

Notice that CN=user2,OU=userstore,DC=dom1,DC=root used to be LEGACY, and is now ABSENT. Also notice that the member attribute still has a version of 10. This is visible evidence of the new LVR code in action. The replication metadata on the member attribute is not touched…and therefore we no longer suffer the inefficient replication of the full multi-valued attribute. Furthermore, the LDAP transaction is now limited to the modified values instead of having to re-write the entire multi-valued attribute.

   

   

Auth restore of group on w2k3entr2-vm12

This object replicates w2k3entr2-vm11 as shown below.

C:\>repadmin /showobjmeta w2k3entr2-vm11 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

13 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 objectClass

14208 Default-First-Site-Name\W2K3ENTR2-VM11 14208 2008-01-17 15:47:09 2 cn

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100010 member <--

Notice the member attribute version has increased by 100,000 and replicated to w2k3entr2-vm11. Completely expected behavior for an auth restore.

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 instanceType

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 15:26:08 Default-First-Site-Name\W2K3ENTR2-VM12 14212 16386 100001

CN=user1,OU=userstore,DC=dom1,DC=root

ABSENT member 2008-01-17 15:13:00 Default-First-Site-Name\W2K3ENTR2-VM11 14178 14178 1

CN=user2,OU=userstore,DC=dom1,DC=root <-- Notice this entry is still at version 1 so the auth restore did not touch this attribute entry. This is because this entry did not exist in the restored database as an LVR entry. It was a legacy entry at the time of backup. So auth restore did not return the groups effective (legacy + LVR) membership to prior state.

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip----

   

DOM1\GLOBALGROUP on w2k3entr2-vm12 after auth restore but before first inbound replication cycle of the domain partition

C:\>repadmin /showobjmeta w2k3entr2-vm12 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

13 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 objectClass

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 cn

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100010 member

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 instanceType

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 15:26:08 Default-First-Site-Name\W2K3ENTR2-VM12 16386 16386 100001

CN=user1,OU=userstore,DC=dom1,DC=root

LEGACY member <-- Things look the same here as before the backup time except the member version above is 100,000 greater. So changes were rolled back to prior state….until, look below.

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip----

   

w2k3entr2-vm11 receives inbound replication from W2K3entr2-vm12

C:\>repadmin /showobjmeta w2k3entr2-vm12 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

13 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 objectClass

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 cn

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100010 member

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 instanceType

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 15:26:08 Default-First-Site-Name\W2K3ENTR2-VM12 16386 16386 100001

CN=user1,OU=userstore,DC=dom1,DC=root

ABSENT member 2008-01-17 15:13:00 Default-First-Site-Name\W2K3ENTR2-VM11 16449 14178 1

CN=user2,OU=userstore,DC=dom1,DC=root <-- this entry replicated to w2k3entr2-vm12 just as it had prior to the restore.

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip-----

   

So, auth restore of the modified group did not recover the groups effective membership to the prior state.

   

Corrected the membership by importing NTDSUTIL generated ldif in the production domain. See "How should an administrator recover from this situation" above.

C:\>repadmin /showobjmeta w2k3entr2-vm11 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

13 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 objectClass

14208 Default-First-Site-Name\W2K3ENTR2-VM11 14208 2008-01-17 15:47:09 2 cn

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100010 member

14209 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 instanceType

13977 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 15:26:08 Default-First-Site-Name\W2K3ENTR2-VM12 14212 16386 100001

CN=user1,OU=userstore,DC=dom1,DC=root

PRESENT member 2008-01-17 16:10:37 Default-First-Site-Name\W2K3ENTR2-VM12 14251 16466 2

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

   

C:\>repadmin /showobjmeta w2k3entr2-vm12 cn=globalgroup,ou=groups,dc=dom1,dc=root

   

13 entries.

Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 objectClass

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 cn

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100010 member

16385 Default-First-Site-Name\W2K3ENTR2-VM12 16385 2008-01-17 15:26:08100001 instanceType

12352 Default-First-Site-Name\W2K3ENTR2-VM11 13977 2008-01-17 14:14:55 1 whenCreated

10 entries.

Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver

======= ============ ============= ================= ======= ======= ===

Distinguished Name

=============================

ABSENT member 2008-01-17 15:26:08 Default-First-Site-Name\W2K3ENTR2-VM12 16386 16386 100001

CN=user1,OU=userstore,DC=dom1,DC=root

PRESENT member 2008-01-17 16:10:37 Default-First-Site-Name\W2K3ENTR2-VM12 16466 16466 2

CN=user2,OU=userstore,DC=dom1,DC=root

LEGACY member

CN=user3,OU=userstore,DC=dom1,DC=root

----snip----

Note: What was LEGACY prior to the accidental delete is now PRESENT. This strategy achieves the administrator's goal of putting the group membership back where it was.

   

   

These postings are provided "AS IS" with no warranties, and confers no rights. The content of this site are personal opinions and do not represent the Microsoft corporation view in anyway. In addition, thoughts and opinions often change. Because a weblog is intended to provide a semi-permanent point-in-time snapshot, you should not consider out of date posts to reflect current thoughts and opinions.