It struck me the other day what a statistical improbability it was that I haven’t really talked much about lingering object problems in AD yet in this blog. They are one hot topic to support people, even if they are not our most common problem. The hot topic part comes from the epic pain in the nether regions it can be to resolve them, and keep them from coming back.
In this post we’re going to talk about one type of lingering objects: lingering backlinks. The idea for this post came about as the result of my needing to rewrite a KB article on this issue. Hopefully you’ll see the article out there soon, but in the meantime you can have it, and my extra commentary with it.
Let’s have a recap first on what a lingering object is. A lingering object is an object which exists in one domain controller’s (DC) database but has been deleted and since garbage collected on all other domain controllers in the forest. The lingering part is that the one DC just didn’t get told to delete the object…and didn’t even try to replicate AD with the other domain controllers for the entire length of time a “deleted” object is kept around in the Active Directory database before it is really “garbage collected” or really deleted from the database. That length of time is known as the tombstonelifetime, or TSL (60 or 180 days depending).
More info on that stuff’s been around for awhile. Here's some information about lingering objects in the KB for a Windows 2000-based forest or in a Windows Server 2003-based forest .
But what’s a lingering backlink? Well, as to what a back link is, we keep track of group memberships for users (among other things) by making the distribution or security groups member multi-valued attribute have a link to the member attribute on user objects, and vice versa (vice versa in group membership anyway). You remove a user from that group in Active Directory Users and Computers (DSA.MSC) and the links are removed. Likewise, if either the user or the group are deleted the link is removed. In DSA.MSC what you see following the deletion is that the member attribute of the remaining object will no longer contain a value entry for that now-removed object.
Here are two resources that give more information and can hopefully improve your understanding of back and forward links Directory Data Store .
Now, a lingering object is basically an object which still exists on one replica domain controller but has been deleted and garbage collected by all other replicas. For most scenarios we have tried-and-true methods for fixing these. Good documentation is out there on how to identify the issue and how to resolve it for most cases, starting with the KB article listed above, earlier in this post.
So a lingering backlink is a pair of linked attributes where the link is remaining as the result of one of the link pair being a lingering object. For a user’s group membership that can be a real pain if the global catalog which Exchange or Outlook is querying has a lingering backlink. In fact they would likely get email when they shouldn’t.
Seems simple enough, doesn’t it? But how do we get in the situation of having these lingering backlinks and what kind of issues can they cause?
This is where I’m going to paste some parts I wrote or rewrote for that Knowledge Base article:
After recovering from an Active Directory disaster recovery problem, or problems with lingering objects in your Active Directory, users in the forest may see problems where users may receive email messages from some distribution groups to which they no longer hold membership to.
This problem may occur because you have lingering linked values in your Global Catalogs in Active Directory. The following attributes may have lingering linked values which can cause this behavior:
It should be noted that this behavior can affect other linked values and updates to non-writable naming contexts (Global Catalog partitions) as well, including updates to security group membership for users.
There are two possible causes of this behavior: Lingering Object Replication problems resulting from domain controller replication partners that are out of synch past the TombstoneLifetime value, and lingering object problems resulting from Active Directory authoritative restores done on Global Catalog domain controllers with a pre-Service Pack 1 version of NTDSUTIL.EXE if the authoritative restore was done against all objects.
More information on Lingering Object Replication problems and solutions can be found at these locations: http://support.microsoft.com/kb/317097 and at Technet http://technet2.microsoft.com/windowsserver/en/library/4a1f420d-25d6-417c-9d8b-6e22f472ef3c1033.mspx?mfr=true .
For Active Directory authoritative restore caused linked value replication problems, the Pre-Service Pack 1 version of NTDSUTIL.EXE would increment non-writable objects (Global Catalog partial attribute set objects representing objects in other domains) versions as well as writable objects (objects which reside in the domain that the GC is a domain controller for) when the Administrator chose the “restore database” global choice for authoritative restore rather than “restore object” or “restore subtree” on a global catalog domain controller. The version number mentioned is used to track what the latest version of an object is and whether an update for that should be applied to a domain controller or discarded. Authoritative restore increments this number by 100,000 every time it “marks” an object authoritative.
Global Catalog updates for these objects may not be accepted since the local copy has a higher version number in the above scenario. As a result, updates to distribution group memberships may not be retained as expected. This behavior can affect other updates to non-writable naming contexts (Global Catalog partitions) as well, including security group membership lists.
There are several approaches which can be used to resolve this problem.
For a small forest, consider removing the global catalog role from all global catalogs in the forest. Once that is done promote global catalogs anew since they will then source from writable domain controllers rather than problematic global catalog domain controllers and they will not then receive the improperly version incremented objects once promotion is complete.
Disable outbound AD replication on all global catalogs in the forest as you use Repadmin.exe to rehost one global catalogs partitions, using writable domain controllers as the source to pull from. Repeat the rehost process for all partitions on all global catalogs until all are complete.
Run the Replfix.exe utility to process lingering linked values. This process performs the following functions:
1. Transfers the forward link values for the attributes in the partial attribute set to a global catalog that hosts a writable copy of a domainNC object
2. Transfers the same values to a global catalog that hosts a read-only copy of a domainNC object
3. Compares the previous global catalogs and then creates a fixup script if one is required
4. Run the fixup script on the global catalog that hosts a writable copy of a domainNC object
You must run this process for each domain that could possibly have lingering linked values.
Note The Replfix.exe utility is not available for public download. If you experience the problem that the "Symptoms" section describes, contact Microsoft Product Support Services for more information. For a complete list of Microsoft Product Support Services telephone numbers, and for information about support costs, visit the following Microsoft Web site:
..and though I didn’t put this in the KB article I have a little screenshot of the command line option which can result in this issue occurring in a bit of a “perfect storm” manner if done on a global catalog with the pre Service Pack 1 version of ntdsutil.exe followed by that DC being left out of touch, replication-wise, with the rest of the forest for greater than the TSL past these objects being deleted:
A few final thoughts regarding lingering objects and backlinks. We have some really good solutions for the more typical lingering object scenarios. Things like the repadmin.exe tool’s /rehost switch, it’s /removelingeringobjects switch, replfix and such. I simply thought this was tough enough of a technical nut to crack, and interesting enough, that I’d pass along to you all.
I also wanted to make certain that our customers world-wide know that they can call us at Microsoft for assistance with these issues if they occur. Each “lingering object” scenario is different but our Customer Services and Support team for Directory Services are the folks who have the largest number of, broadest and most in depth experiences handling these issues of anyone. So please don’t hesitate to engage us if we can help, contact info is always available at http://support.microsoft.com/ .
Happy Friday everyone!