Problem Environment
Contoso has a very large branch-office deployment of Active Directory. Each branch-office is an Active Directory site, and each DC is a GC because of potentially unreliable WAN links.
Any time a Scheme upgrade is performed in which the Partial Attribute Set (PAS) is updated, replication fails for numerous partitions until all hub DCs have been updated. The last time they upgraded the Schema, it was several months before all DCs received the updated PAS.
Details
Active Directory replication failed after upgrading the schema for Exchange 2007 with an event id 8464.
“Synchronization attempt failed because the destination DC is currently waiting to synchronize new partial attributes from source. This condition is normal if a recent schema change modified the partial attribute set. The destination partial attribute set is not a subset of source partial attribute set.”
They were getting this message because the DCs in their hub site had not received the updated Partial Attribute Set (PAS) for the affected partitions.
It had been several days since the Schema was upgraded, and given enough time, this problem would eventually correct itself. Our goal was to understand why it took so long for all DCs to reach replication convergence, and to decrease the time it takes for all domain controllers to receive the updated PAS.
In order to visualize their replication environment, I had them run our AD Topology Diagrammer tool. (ADTD or sometimes referred to by its former name: AD Map) The Visio diagram revealed one hub site and over a thousand branch-office sites. Each branch office has replication connections to the hub site only.
Cause
Replication of the updated PAS did not occur because of the following:
Each of the bridgehead domain controllers in the hub site have over 200 replication connections for each partition. The customer has a DC in each branch office, each office is its own site, and each DC is a GC. The sheer amount of replication connections interfered with the timely update of the PAS. This environment would be a great candidate for RODCs (since there would not be any outbound connections from the branch office sites), but now was not the time to talk topology redesign.
Solution
Our immediate goal, to get this resolved quickly, was to get the PAS updated on each DC in the Hub site for both domain partitions reported in the events.
First we had to identify which GC's have the updated PAS. Here is the repadmin.exe command that we used:
The following command will dump the PAS from every DC in the forest for the partition specified in the DN path *. The resulting output lets us know which GCs haven’t gotten the updates yet:
“repadmin /showattr gc: dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Corp.txt”
* Since “dc=corp,dc=contoso,dc=com” is specified in the above command, it will dump the PAS for the “Corp” domain partition.
The following command would be used to dump the PAS for the “Branches” domain partition:
“repadmin /showattr gc: dc=branches,dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Branches.txt”
The value of interest in the output is listed after the "v1.cAttrs =" text.
If you want to check a single GC, you could run:
repadmin /showattr DCName dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset
During this issue we came up with several methods to get these servers updated. I will list out each of the methods (A, B, and C) with the top choice listed first.
A. Set up a replication connection, disable inbound and outbound replication, and force replication:
1. Create a manual replication connection to a DC that already has the updated partial attribute set (unless you already have a connection to this dc).
“repadmin /add <Naming Context> <Destination DC> <Source DC> /readonly (if needed)”
2. Disable inbound and outbound replication (one very fast way to clear out the queue).
“repadmin /options dc_name +DISABLE_INBOUND_REPL”
“repadmin /options dc_name +DISABLE_OUTBOUND_REPL”
3. Force replication with the DC over the newly created connection.
“Repadmin /replicate <dc-with-low-cAttr> <DC-with-high-cAttr> <DNpath> /force [/readonly if needed]”
4. Run the repadmin command to check the inbound replication queue.
“repadmin /queue”
You should just see the one item queued, but I have seen a few more replication requests sneak in on a very busy dc.
5. You should be able to re-enable inbound and outbound replication immediately.
“repadmin /options dc_name -DISABLE_INBOUND_REPL”
“repadmin /options dc_name -DISABLE_OUTBOUND_REPL”
B. Rehost the partition:
1. “repadmin /rehost <rehosted GC hostname> <DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition>”
If the command returns a “replication operation was preempted” error, then perform the following steps:
2. Run “repadmin /rehost specifying the name of both the target source DC (the good DC hosting a writable copy of the rehosted partition).”
“repadmin /rehost <rehosted GC hostname> < DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition >.”
3. Press Ctrl+C to stop the rehost command before the preemption error occurs.
4. Rerun the repadmin /rehost command, which should complete normally.
“repadmin /rehost < rehosted GC hostname > < DN path of rehosted naming context > < fully qualified DNS name of good DC hosting a writable copy of the domain partition>”
5. Run repadmin /showreps /v against the destination DC and confirm replication of the read-only partition completed.
C. Move extraneous replication connections off the DC that needs to be updated:
1. This can be accomplished by moving the DC to a test site (so that it has fewer replication connections). OR:
2. Temporarily configure this server so that it is not a preferred bridgehead server so that the KCC removes the connections
As always, let me know if you have any questions!
Justin
Hi Everybody!
Ever have an application's window that you need to move but can't---either because you have connected via RDP and the window is on a different monitor, or because you can't get to the title bar to grab it with the mouse?
The following keyboard shortcuts should do the trick
Alt + Spacebar
M
---Then just use the arrow keys to move the window
Here is a simple command that you can run right now in order to save you from some down-time the next time your file system permissions get set back to the Windows defaults. Proactively running this from time to time (think: task scheduler) can save you a lot of time and money the next time disaster strikes. There are multiple backup solutions and utilities that you can use for this purpose, however this one is easy to use and the price is right. (free)
Subinacl.exe
http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B&displaylang=en
Here is example syntax that you can use to proactively back up your NTFS permissions:
Subinacl /noverbose /output=c:\ntfs_perms.txt /subdirectories "Path to the Folder whose NTFS permissions we have to Backup" To backup the permissions of the folder, subfolders and files on folder called Data on the G: drive:
subinacl /noverbose /output=c:\ntfs_perms.txt /subdirectories G:\data\
If you wanted to just backup the NTFS permissions for the entire drive, the command would look like this:
subinacl /noverbose /output=c:\ntfs_G_drive_perms.txt /subdirectories G:\*.*
Most of you will probably not be concerned with backing up down to the file level, and are satisfied with just backing up the permissions at the directory level. Backing up the permissions for just the directories can be achieved with the following syntax:
subinacl /noverbose /output=c:\G_driveNTFSperms.txt /subdirectories=directoriesonly G:\*.*
The contents of the file created by subinacl are viewable in your favorite text editor:
To restore the permissions on the drive using the file that you backed them up to:
Subinacl /playfile c:\G_driveNTFSperms.txt
Test it out thoroughly in your lab environment before rolling it out to production.
Thanks for reading,
Justin Turner
Hi Everybody!
Have you heard about the new certification from Microsoft? In the grand scheme of things (as far as MSFT certification goes) it is the next level above the MCITP. (MCITP is what many consider to be the replacement for the old MCSE)
Check out the pic for a better description:
Public registration for the new Master certification is finally open. Full details about the program are available here.
Quick overview:
- Cert requirements:
- -Mandatory three week training session
- -in-class written and lab-based exams
- -Pass qualification lab exam
It seems that they haven't updated the registration link yet. Here is the direct link.
They've updated it now. Here is the registration page.
Steps to apply:
- Fill out application here ($125) IMPORTANT: If the pre-requisite exams don't show up on your MCP transcript then you forfeit the application fee. (70-219 or 70-297, MCSE 2000 or MCSE 2003 or IT Pro: Enterprise Admin (MCITP: EA))
- Upload resume (if you are eligible)
- Register: Pay (and choose class dates)
Training will be in Redmond only in FY09. Dates for the first training session: (at half the price)
- Exchange Server 2007: October 6-25, 2008
- SQL Server 2008: October 13 – November 1, 2008
- Windows Server 2008: November 3-22, 2008
Pre-recorded Live meeting session linked to from Trika's blog here.
Hi Everybody!
This should be a pretty good presentation on Windows Vista internals. Mark Russinovich has co-authored several books on Windows Internals, and is one of the developers for all of the cool troubleshooting utilities over at our Sysinternals site.
Springboard Series Virtual Roundtable
Under the Hood: Windows Vista Performance...Need Answers?
Join Mark Russinovich and a panel of industry experts for a LIVE virtual roundtable to explore your top of mind performance issues, common misconfigurations, and tips on how to fix them. From boot times and applets to disk performance and battery life, find out how to optimize Windows Vista and what you can do to improve overall system performance.
Submit your performance questions live during the event or send them in advance to vrtable@microsoft.com.
Save the date!
Wednesday, September 24, 2008
9:00am Pacific Standard Time
Find answers to your Windows Vista adoption questions with resources, tools, monthly straight-talk articles, and upfront guidance based on early adopter and community feedback. To learn more, visit www.microsoft.com/springboard,
Springboard Series: The resource for Windows desktop IT professionals
This tip can help clear some confusion when you encounter the following error while forcing replication:
result 8461 The replication operation was preempted
Here is the output of running repadmin /showreps on a brand new domain controller:
DC=contoso,DC=com
5th_Ward\ContosoDC2 via RPC
objectGuid: 5ed02b33-a6ab-4576-b109-bb688221e6e3
Last attempt @ 2008-08-21 17:51.44 failed, result 8461: The replication operation was preempted.
Last success @ (never).
If you use the verbose switch you will see the following:
DC invocationID:
5ed02b33-a6ab-4576-b109-bb688221e6e3 DO_SCHEDULED_SYNCS WRITEABLE
COMPRESS_CHANGES NO_CHANGE_NOTIFICATIONS NEVER_SYNCED PREEMPTED USNs:
577738/OU, 0/PU Last attempt @ 2008-08-21 17:51:44 was delayed for a
normal reason, result 8461 (0x210d): The replication operation was preempted.
Last success @ (never).
The reason we log this message is because there is an ongoing higher priority replication operation that has not completed yet. In order to see what inbound replication items are queued (and have preempted us) run the following command on the dc:
repadmin /queue
If you have a lot of items queued run:
repadmin /queue >c:\repl_queue.txt
This will output a list of inbound replication tasks that are queued. They are listed by task id in order of priority. The replication operation that we are waiting on is at the top of the list.
If you want to find out what objects/attributes are currently being replicated in to this dc we can enable diagnostic logging for "5 Replication Events" (coming in a future post) However, Steve Patrick(Spat) shows us a cool repadmin cmd in his post here.
repadmin /showchanges ContosoDC2 5ed02b33-a6ab-4576-b109-bb688221e6e3 DC=corp,DC=com >changes.txt
Part 2 of this post will come at a later date. Stay tuned...
Hi Everybody!
I just wanted to write a quick note to let you all know that the RODC compatibility pack for Windows Server 2003 and Windows XP clients is available for download.
Description of the Windows Server 2008 read-only domain controller compatibility pack for Windows Server 2003 clients and for Windows XP clients
http://support.microsoft.com/kb/944043
Not familiar with RODC?
check out these links:
Read-only Active Directory Features
Step-by-Step Guide for Read-only Domain Controllers
Thanks!
Justin
Hi Everybody!
This one comes at the request of several customers. Many of you out there are trying to determine which version of Server 2008 you will deploy. For most it comes down to deciding between two of the five major versions: Server 2008 Standard edition and Server 2008 Enterprise edition. Given the amount of features included in the OS and all of the different versions we shipped, trying to determine what version includes what feature can be confusing.
Fortunately we have released a very detailed treatise on this very subject in the form of a 247 page document appropriately titled, Windows Server 2008 Reviewers Guide.
Inside you will find a pretty thorough support matrix and technical nuggets like:
- In Standard Edition you are limited to one standalone DFS Namespace. (DFS Root) This limit does not apply to domain-based DFS implementations.
- Cross-File Replication for DFS-R is not available in the Standard or Web editions.
- Server Core is available in all editions except for Itanium.
- Hyper-V is included in Enterprise, Datacenter and Standard editions as long as you don't buy the version that say "without Hyper-V"
- TS Licensing in Windows Server 2008 now allows you to track per-user CALs
- Still no support for Cluster (failover) in Standard Edition, but you can now have 16 nodes with the Enterprise and Datacenter editions (8 with Itanium)
Here are some screen snags taken right from the guide:
Enjoy!
It can be a little tedious to verify replication status in a large Active Directory environment via the Sites and Services snap-in. Here is a command I use quite frequently to check the replication status of all domain controllers:
REPADMIN /SHOWREPL * /CSV >showrepl.csv
View the file in Microsoft Excel and perform the following filtering options to get a good quick overview of replication health:
1. Hide columns A and B
2. Select the row just under Column headers and choose Window / Freeze Pane (In Excel 2007: View tab, Window, Freeze Panes, Freeze Top Row)
3. Highlight the entire spreadsheet and choose Data / Filter / Auto-Filter
4. Click on the down-arrow for the "Last Failure Status" column, and choose "does not
equal" then type in "0" (In Excel 2007: Uncheck the box next to "0")
You are left with a list of domain controllers having replication problems. From a cmd prompt, use:
"net helpmsg ErrorCodeNumber" to identify the replication error
(eg. net helpmsg 1396)
Hi Everybody!
This is just an update to let you all know about two software releases:
We just released Virtual Server R2 SP1 on Monday.
You can download it for free from here.
Some of the updates:
- Support for hardware-assisted virtualization
- Supports up to 256 GB of physical memory, and up to 512 virtual machines
- Quick Migration
Today we released the Release Candidate of Windows Home Server.
From the Home Server site:
"With Windows Home Server, you can store your music, photos, and other files on a central hub-like hard drive, accessible from every PC in your house. Protect your files and your PCs with automatic backup and a simple restore process—even gain access to files on your PCs from anywhere with an Internet connection through secure Web access."
You can sign up to get it here. (also free, but not RTM code)
Thanks!
Justin
For this tip you will need a somewhat newer version of ntfrsutl.exe
You can grab a version out of the Service Pack 2 Support Tools download here.
Beginning with the version of ntfrsutl.exe in KB 823230 we have the ability to force FRS replication to occur across site boundaries immediately instead of waiting for the schedule to open up.
Here is the command's syntax:
ntfrsutl forcerepl [computer] /r SetName /p PartnerDnsName
= Force FRS to start a replication cycle ignoring the schedule
The PartnerDNSName is the FQDN of the server that you want to source from.
Here is an example using a DC Name of ContosoDC1 and a PartnerDNSName of ContosoDC2:
ntfrsutl forcerepl contosodc1 /r "domain system volume (sysvol share)" /p ContosoDC2.Contoso.com
Running the command initiates replication, and returns the following information:
LocalComputerName = contosodc1
ReplicaSetGuid = (null)
CxtionGuid = (null)
ReplicaSetName = domain system volume (sysvol share)
PartnerDnsName = ContosoDC2.Contoso.com
As you can see there are two additional parameters that you can specify, ReplicaSetGuid and CxtionGuid, but neither are required.
I've added a new tag called "Quick Tips." These are going to be smaller posts where I offer some time-saving tip, or some other similar type of goodie.
Look out for the first one soon!
Some time ago I did a webcast presentation on Active Directory User and Group Restore.
I've included the link for those of you that may have missed it.
Check out the on-demand presentation here:
http://www.msusapartnerreadiness.com/WS_abstract.asp?eid=15004864
(Unfortunately registration is required, but that takes only a few seconds)
Let me know if you would like to see more like this one.
Thanks!
I see a lot of customers unnecessarily using the boot.ini /3GB switch. Explaining when and when not to use it takes quite some time. Thanks to the Platforms Performance team, I now have a very nice quality post to point them to.
The helpdesk phone had been ringing incessantly all day. Many people throughout the AD forest were unable to login to their respective domains. It seems that accounts throughout the forest had somehow been deleted. John, tired from having been up all night watching "White and Nerdy", was called in to help identify what was going on. Fortunately he had recently enabled auditing for account deletions due to a recent problem that he had. After some serious filtering he was able to find the following event in the Security event log:
Event Type: Success Audit
Event Source: Security
Event Category: Account Management
Event ID: 630
Date: 1/17/2007
Time: 12:30:44 AM
User: Contoso\JuniorAdmin
Computer: DisgruntledXP
Description:
User Account Deleted:
Target Account Name: JustinTurner
Target Domain: Contoso
Target AccountID: Justin Turner []DEL:3f4567f2-f90b-493e-81a3-dcfc75596cd7
Caller User Name: JuniorAdmin
Caller Domain: Contoso
This was a little offsetting to say the least. "JuniorAdmin" was the name of the account for one of his Junior Network Administrators that they just fired for getting them into that last mess. He quickly disabled the account, and then attempted to identify what kind of mess they were in now. His heart sank into his stomach when he discovered that JuniorAdmin was a member of the Schema and Enterprise Admins security groups...
I had planned on providing an in-depth discussion about forest recovery, and then realized that there is already more than enough information on this topic. Since I have already advertised this, I will go ahead and provide what I hope will serve as a good general overview, and then point you to a few good resources for the process. There is now a Server 2003 specific forest recovery whitepaper, but the process is unchanged from Windows 2000. There are some additional server 2003 specific goodies added however. (like repadmin /removelingeringobjects)
Before we dive right into the process I want to point out a couple of reasons for why you might have to perform an Active Directory forest recovery.
There are a few reasons that I won't mention, but the two most common I see are:
1. The security of your directory has been compromised either through virus, hacker, or disgruntled employee.
2. A change was made to the schema which needs to be undone.
This really is a big deal, and is not something you want to jump straight to without first consulting Microsoft PSS/CSS/EPS/Platforms Support. (we've had so many different names, I don't remember the current one :-) The team you would be dealing with for this particular issue would be Platforms Directory Services. We want to try to determine what caused the forest failure, and also to ensure that a forest recovery is the best recovery option. An entire forest recovery is obviously one of the last steps you would want to try, so it really is best to explore all other recovery options first.
The five hundred thousand foot overview of the process is:
1. Recover one dc from the forest root domain first from backup.
2. Recover one dc from each of the remaining domains from backup.
3. Restore additional DC's by promoting them via dcpromo.
What follows is a general overview of the process that is outlined in both the Windows 2000 and Server 2003 forest recovery whitepapers referenced earlier. Please reference the particular whitepaper for the specific steps.
There are three major stages of a forest recovery:
Pre-recovery, Recovery, and Post Recovery
Pre-Recovery:
1. Determine the current forest structure/topology
2. Find one trusted backup to use per domain
3. Shutdown, and disconnect if possible, all DC's in the forest
Recovery:
1. Isolate the server, (unplug network cable) and perform a system state restore (ensure you choose the Advanced option to perform a Primary restore of Sysvol) Only choose this option for the first DC in a domain.
2. Verify DC was successfully restored after rebooting
3. Configure DNS
4. Disable Global Catalog (if enabled)
5. Raise RID pool by 100,000
6. Seize FSMO roles
7. Perform metadata cleanup of all other DC's in the forest root domain (also delete DC computer objects for dc's that will not be restored from backup in this domain)
8. Reset machine account twice
9. Reset the krbtgt account password twice
10. Reset all trust passwords twice
11. Restore the first DC in each of the remaining domains from backup (perform Recovery steps 1-10 to recover one dc in each of the remaining domains)
As you restore each DC, you will want to point them to the recovered forest root DC for DNS.
12. Connect the restored DC's back to the network (prior to performing this step ensure that no old dc's are still online)
13. Perform a full replica set sync of AD
14. Enable forest root dc as a GC
15. Seize schema master on forest root dc (if the schema master wasn't the dc that was restored)
16. Recover additional DC's in each of the domains using dcpromo
Post-Recovery:
1. Revert forest back to original DNS configuration
2. Redistribute FSMO roles
3. Enable additional Global catalog servers
4. Get a good system state backup from at least two dc's in each domain
As you can see, this is a very lengthy process. The whitepaper walks you through each step in detail. There is a good index in the paper that has step by step instructions for every single process as well.
Finally I just want to expand on a couple of the items listed above.
Some considerations to take when identifying which DC's to restore:
You will only be restoring one DC per domain. The recovery process will go much quicker if the restored DC was a DNS server, and was not a GC at the time the backup was taken. For some of you this may be an easy choice as you may only be able to find one good backup. I find that when it comes to these situations, many have trouble locating a decent system state backup. (but maybe my view is skewed because the customers that have tested their disaster recovery plan don't call us?) Additionally the process will go by quicker if the DC that you restore in the forest root domain was the Domain Naming and or Schema master. Selecting one that was a RID master will also help. If you are unable to locate a backup from one of these FSMO masters then you will just need to seize the role after the server is restored. To help you out with this there is a cool repadmin command that shows you the last time a dc's system state was backed up: repadmin /showbackup DCName
Don't try to shortcut this process by leaving out steps:
For example: When it says to shutdown and/or disconnect each dc. Do exactly that. We want to ensure that a restored dc does not replicate in bad data from a dc that we forgot to (or couldn't) shutdown. So at the very least ensure that you have your servers that you are restoring disconnected from the network. Also ensure that you reset each of the passwords listed twice. Ensure that you are very thorough with your metadata cleanup stage. Otherwise you will have a not so fun time troubleshooting why your DC's aren't replicating.
There is a typo several times in both whitepapers that greatly changes the meaning of the step:
"Delete server objects and computer objects for all domain controllers in the forest root domain that you are restoring from backup..."
This should read "...that you aren't restoring from backup" I will attempt to get this changed in the whitepapers.
Repadmin is your friend:
There are a few steps where you will use various repadmin commands. Learning repadmin syntax ahead of time will aid in the process. It is also very useful for performing day-to-day AD operations as well.
Some options that you will need to use:
/showbackup
/syncall
/showreps
/options
You may also end up having to use /add, /sync, and /removelingeringobjects as well. However, if you follow the step where it says not to restore a DC that was a GC (or just uncheck that after the restore) then you shouldn't have to worry about lingering objects.
Well that's all I have to say about that. :-) I'll add more later if I think of something else that I left out.
Post any comments or questions you have about this or any other topic that I have blogged about.
Up next: Cluster service failure troubleshooting
Thanks for reading!
Justin
