For the first technical post I thought it would be best to lay the groundwork and mention a tool which can be used in a wide variety of AD replication scenarios. We'll talk about one of the useful features in this post, and follow up with one other for the next post before we move onto another topic.
In my personal point of view, there are only two kinds of AD admins: those who love REPLMON, and those who love REPADMIN. As you may be able to infer from the title of this post, I am the latter. Not that there is anything necessarily evil about REPLMON advocates, but let's face it: it's not the cat's meow (or the bee's knees) of AD replication troubleshooting. Not like Repadmin.exe anyhow.
For those who haven't had an opportunity to use this tool before it is the general purpose, command line tool for troubleshooting directory replication. Repadmin.exe is included in the Windows Support Tools, which can be downloaded for free (free is one of my favorite words) here:
We had the tool available in Windows 2000, but in the Windows 2003 time frame it has more features. Syntax between the versions is slightly different, with a few deprecated commands. One useability thing which was a great improvement, generally speaking, was the addition of code to help with lookup of domain controller names. Windows 2000 would often require the MSDCS GUID to be used in the syntax. General idea is that our development folks (you know who you are) have been improving the tool as needed over the years, and they deserve HUGE kudos for it.
Back to the technical part...
One of the more frequent calls we at Microsoft Directory Services support get is problematic network connections which prevent AD replication. I used the big fancy generic term "problematic" since the actual originating causes of the AD replication failures are many. Could be misconfigured DNS on the DC NIC, wrong binding order of same, closed networking ports anywhere between the DCs in question, missing SRV records-or any of a myriad others. Bottom line is that, once you've fixed that problem, you need to test to make sure you are, in fact, replicating succesfully.
Repadmin's your tool for that.
Let's say that I add a bit of text to a user account in order to see if that data arrives. The Mark One Eye-ball is one method (just open DSA.MSC on the opposite end) but there's a better way, and it can track it as the change travails the forest.
REPADMIN /SHOWOBJMETA * "cn=user,ou=weisenheimers,dc=wacky,dc=org" >usermeta.txt
The asterisk can be replaced by a single domain controller name (like DC1). The asterisk runs against all in forest, so have your EA creds ready for that, and make sure to take into account network topology concerns since the tool queries all DCs for the object metadata.
The distinguishedname in quotes can actually be replaced by the objectGUID. Very cool for when you receive an event citing an objectGUID but no other identifying piece of info. The syntax of that field is “<GUID=objectguid>”.
The command above will output the replication metadata to a text file. This data is the information which the directory replicator uses to track updates for an object's attributes. Since AD replicates changes on a per attribute basis (or in the case of linked value replication, single value update for a multivalued attribute) you will see attribute name, local USN, time and date stamp (good for forensics), and originating DC and USN (also good for forensics).
Here's a sample:
repadmin running command /showobjmeta against server DC1.wacky.org
Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute
========= =============================================== =========
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 objectClass
80472995 Crazyland\DC1 80472995 2005-10-13 07:49:08 3 cn
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 description
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 instanceType
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 whenCreated
20341575 Crazyland\DC1 20341575 2004-09-10 13:12:48 2 displayName
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 nTSecurityDescriptor
80472995 Crazyland\DC4 33405674 2005-10-13 07:49:06 5 name
80336274 Crazyland\DC5 28321657 2005-10-10 14:35:17 4 userAccountControl
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 codePage
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 countryCode
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 homeDirectory
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 homeDrive
80391546 Crazyland\DC1 80391546 2005-10-11 18:59:09 15 dBCSPwd
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 localPolicyFlags
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 scriptPath
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 logonHours
80566031 Crazyland\DC1 80566031 2006-6-11 18:59:09 15 unicodePwd
80391546 Crazyland\DC1 80391546 2005-10-11 18:59:09 15 ntPwdHistory
80391546 Crazyland\DC1 80391546 2005-10-11 18:59:09 15 pwdLastSet
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 primaryGroupID
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 userParameters
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 profilePath
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 objectSid
33433553 Crazyland\DC2 33433553 2006-06-12 15:16:35 1 comment
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 accountExpires
80391546 Crazyland\DC1 80391546 2005-10-11 18:59:09 15 lmPwdHistory
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 sAMAccountName
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 sAMAccountType
20341645 Crazyland\DC1 20341645 2004-09-10 13:13:00 1 operatingSystem
20341645 Crazyland\DC1 20341645 2004-09-10 13:13:00 1 operatingSystemVersion
20341577 Crazyland\DC1 20341577 2004-09-10 13:12:48 1 dNSHostName
80416919 Crazyland\DC5 28388830 2005-10-12 08:19:49 11 servicePrincipalName
20341573 Crazyland\DC1 20341573 2004-09-10 13:12:48 1 objectCategory
20341574 Crazyland\DC1 20341574 2004-09-10 13:12:48 1 isCriticalSystemObject
80450630 Crazyland\DC1 80450630 2005-10-12 20:51:07 22 lastLogonTimestamp
So, in our replication test to see if things were working, we made a change on DC2-we added a comment to the comment field of a user object in DSA.MSC. Since there is a replication connection where DC1 pulls changes from DC2. Find the “comment” entry above and you’ll see it arrived. Run the SHOWOBJMETA command for this object again against all DCs and you can track it as it successfully replicates throughout the forest.
Final concepts on tracking AD replication using SHOWOBJMETA. First, keep in mind that we have writable and non-writable partitions. In other words, a DC in the WACKY.ORG domain hosts a writable partition for dc=wacky,dc=org, but since it is a GC it will host the non-writable partitions for all other domains in the forest, such as dc=very,dc=wacky,dc=org and dc=notso,dc=wacky,dc=org.
The reason this is mentioned is that if the object and attributes are published as part of the partial attribute set in it’s schema definition, then you may show Global Catalog servers responding in the SHOWOBJMETA that they have (or haven’t) gotten that update.
The other end of that idea is that the non-writable DCs (meaning DCs not in the domain which the user belongs to) which are also not GCs will show the message below:
repadmin running command /showobjmeta against server DC8.very.wacky.org
DsReplicaGetInfo() failed with status 8333 (0x208d):
Directory object not found.
So, we haven’t really gone over any stunningly difficult issues today, but we’ve made an intro to one of our best AD replication troubleshooting tools.
Next post, we’ll going to go over the REPADMIN /SHOWREPL command and how to use it. In the meantime, if anyone has any questions or specific requests then fire away. This blog is for you all, so requests are taken...
Looking at your blog, can I use this to see where a user account was deleted. I've got doubts about my office in Papeete.
To figure out where a user was deleted (and how to reanimate it assuming u are in AD w2k3 *only*),you need 2 (free) tools:
-> Adrestore, where u can donwload here http://www.sysinternals.com/Utilities/AdRestore.html
-> repadmin /showobjmeta dcname deletedobjectDN | find /i "isdeleted"
-> and obviously, auditing already activated on your Default Domain Controler Policy :).
Here we go !
Suppose that yann was deleted:
1) in a command shell, enter adrestore yann. Adrestore will query the tombstone to list all users deleted objects that contains "yann" in their CN.
Adrestore will also output the DN of the deleted user yann. Focus on the right yann.
2)Enter the second command with repadmin and just replace deletedobjectDN by the one u found
with adrestore, and also replace dcname with the name of one of your domain controllers. Make sure u put the strings corresponding to deletedobjectDN in quotations if there are any spaces in it. Reapdmin will show u where(on which DC) and when(time/date) yann was deleted thanks to the "isdeleted" attribute that was piping.
Next, go to your DC, and search for eventID 630 to find occurences corresponding to "deleted object user". Done !
If u are in ad2k3 u can reanimate yann by entering adrestore -r yannn and choose the right user to restore.
NB1: in order to use adrestore, u *must* be domain admins.
NB2: are u from Papeete tahiti ? because it's my homeland man ! I leave tahiti for now 10 years to France where i work in an AD & Exchange team. So .... iaorana ! ;o)
NB3: for those whose wondering what i am talking about.... iaorana means both "hello" or "goodbye" in tahitian language, depending if u arrive or leave :).
NB4: Tim, thanks a lot for your blog. It will leverage our skills on the underground technology of AD . Keep up the good work !
Thanks for the post! The above will work very well, but you may hit one snag. When an object is marked for deletion the distinguishename changes since it is then in the "container" for Deleted Items. So the repadmin command may struggle with that...
But, your comment reminded me about being able to insert the objectGUID of te object in and track it that way.
So, get the objectGUID of the object that was deleted and try using that. Just substitute the DN string entry for the syntax with this:
You can get the objectGUID by using LDP.EXE to view it, or by dumping the database using LDP.EXE.
Let me know if I misunderstood something from your comment. Thanks again for reading!
Thanks for clarification.
Endeed, when an object is deleted, in our case a user yann ;), the DN's changed to this one for example;
Adrestore permits me to find this DN. And when we put all together with repadmin, the command looks like:
repadmin /showobjmeta DC1 "CN=dac\0ADEL:2a299250-27ea-4a05-bdf7-5ca9558ff733,CN=Deleted Objects,DC=mydomain,DC=fr" | find /i "isdeleted". The results of this is:
17730966 SITE1\DC2 17730966 2005-10-27 10:37:11 1 isDeleted.
So u can find where and when a user was deleted. That works for me.
Why using the deleted object DN instead of guid ?
Because I remembered I was not able to work with the guid of the object , so i tried with the DN of the deleted object and that works fine.
Just a little mistake in my repadmin command above. So just correct with the right user that is "yann" and not "dac"; the latter one was the user i had investigated last year in my AD environnement.
repadmin /showobjmeta DC1 "CN=yann\0ADEL:2a299250-27ea-4a05-bdf7-5ca9558ff733,CN=Deleted Objects,DC=mydomain,DC=fr" | find /i "isdeleted".