Active Directory, Cluster and other fun stuff...

Replication error 8464 after schema upgrade

Justin Turner [MSFT] — Wed, 28 Oct 2009 16:00:25 GMT

Problem Environment

Contoso has a very large branch-office deployment of Active Directory. Each branch-office is an Active Directory site, and each DC is a GC because of potentially unreliable WAN links.

Any time a Scheme upgrade is performed in which the Partial Attribute Set (PAS) is updated, replication fails for numerous partitions until all hub DCs have been updated. The last time they upgraded the Schema, it was several months before all DCs received the updated PAS.

Details

Active Directory replication failed after upgrading the schema for Exchange 2007 with an event id 8464.

“Synchronization attempt failed because the destination DC is currently waiting to synchronize new partial attributes from source. This condition is normal if a recent schema change modified the partial attribute set. The destination partial attribute set is not a subset of source partial attribute set.”

They were getting this message because the DCs in their hub site had not received the updated Partial Attribute Set (PAS) for the affected partitions.

It had been several days since the Schema was upgraded, and given enough time, this problem would eventually correct itself. Our goal was to understand why it took so long for all DCs to reach replication convergence, and to decrease the time it takes for all domain controllers to receive the updated PAS.

In order to visualize their replication environment, I had them run our AD Topology Diagrammer tool. (ADTD or sometimes referred to by its former name: AD Map) The Visio diagram revealed one hub site and over a thousand branch-office sites. Each branch office has replication connections to the hub site only.

Cause

Replication of the updated PAS did not occur because of the following:

Each of the bridgehead domain controllers in the hub site have over 200 replication connections for each partition. The customer has a DC in each branch office, each office is its own site, and each DC is a GC. The sheer amount of replication connections interfered with the timely update of the PAS. This environment would be a great candidate for RODCs (since there would not be any outbound connections from the branch office sites), but now was not the time to talk topology redesign.

Solution

Our immediate goal, to get this resolved quickly, was to get the PAS updated on each DC in the Hub site for both domain partitions reported in the events.

First we had to identify which GC's have the updated PAS. Here is the repadmin.exe command that we used:

The following command will dump the PAS from every DC in the forest for the partition specified in the DN path *. The resulting output lets us know which GCs haven’t gotten the updates yet:

“repadmin /showattr gc: dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Corp.txt”

* Since “dc=corp,dc=contoso,dc=com” is specified in the above command, it will dump the PAS for the “Corp” domain partition.

The following command would be used to dump the PAS for the “Branches” domain partition:

“repadmin /showattr gc: dc=branches,dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Branches.txt”

The value of interest in the output is listed after the "v1.cAttrs =" text.

If you want to check a single GC, you could run:

repadmin /showattr DCName dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset

During this issue we came up with several methods to get these servers updated. I will list out each of the methods (A, B, and C) with the top choice listed first.

A. Set up a replication connection, disable inbound and outbound replication, and force replication:

1. Create a manual replication connection to a DC that already has the updated partial attribute set (unless you already have a connection to this dc).

“repadmin /add <Naming Context> <Destination DC> <Source DC> /readonly (if needed)”

2. Disable inbound and outbound replication (one very fast way to clear out the queue).

“repadmin /options dc_name +DISABLE_INBOUND_REPL”

“repadmin /options dc_name +DISABLE_OUTBOUND_REPL”

3. Force replication with the DC over the newly created connection.

“Repadmin /replicate <dc-with-low-cAttr> <DC-with-high-cAttr> <DNpath> /force [/readonly if needed]”

4. Run the repadmin command to check the inbound replication queue.

“repadmin /queue”

You should just see the one item queued, but I have seen a few more replication requests sneak in on a very busy dc.

5. You should be able to re-enable inbound and outbound replication immediately.

“repadmin /options dc_name -DISABLE_INBOUND_REPL”

“repadmin /options dc_name -DISABLE_OUTBOUND_REPL”

B. Rehost the partition:

1. “repadmin /rehost <rehosted GC hostname> <DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition>”

If the command returns a “replication operation was preempted” error, then perform the following steps:

2. Run “repadmin /rehost specifying the name of both the target source DC (the good DC hosting a writable copy of the rehosted partition).”

“repadmin /rehost <rehosted GC hostname> < DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition >.”

3. Press Ctrl+C to stop the rehost command before the preemption error occurs.

4. Rerun the repadmin /rehost command, which should complete normally.

“repadmin /rehost < rehosted GC hostname > < DN path of rehosted naming context > < fully qualified DNS name of good DC hosting a writable copy of the domain partition>”

5. Run repadmin /showreps /v against the destination DC and confirm replication of the read-only partition completed.

C. Move extraneous replication connections off the DC that needs to be updated:

1. This can be accomplished by moving the DC to a test site (so that it has fewer replication connections). OR:

2. Temporarily configure this server so that it is not a preferred bridgehead server so that the KCC removes the connections

As always, let me know if you have any questions!

Justin

Quick Tip: Move Window position via keyboard

Justin Turner [MSFT] — Tue, 23 Jun 2009 16:10:10 GMT

Hi Everybody!

Ever have an application's window that you need to move but can't---either because you have connected via RDP and the window is on a different monitor, or because you can't get to the title bar to grab it with the mouse?

The following keyboard shortcuts should do the trick

Alt + Spacebar

---Then just use the arrow keys to move the window

Quick Tip: Back up your NTFS security permissions

Justin Turner [MSFT] — Thu, 26 Feb 2009 15:09:00 GMT

Here is a simple command that you can run right now in order to save you from some down-time the next time your file system permissions get set back to the Windows defaults. Proactively running this from time to time (think: task scheduler) can save you a lot of time and money the next time disaster strikes. There are multiple backup solutions and utilities that you can use for this purpose, however this one is easy to use and the price is right. (free)

Subinacl.exe

http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B&displaylang=en

Here is example syntax that you can use to proactively back up your NTFS permissions:

Subinacl /noverbose /output=c:\ntfs_perms.txt /subdirectories "Path to the Folder whose NTFS permissions we have to Backup"

To backup the permissions of the folder, subfolders and files on folder called Data on the G: drive:

subinacl /noverbose /output=c:\ntfs_perms.txt /subdirectories G:\data\

If you wanted to just backup the NTFS permissions for the entire drive, the command would look like this:

subinacl /noverbose /output=c:\ntfs_G_drive_perms.txt /subdirectories G:\*.*

Most of you will probably not be concerned with backing up down to the file level, and are satisfied with just backing up the permissions at the directory level. Backing up the permissions for just the directories can be achieved with the following syntax:

subinacl /noverbose /output=c:\G_driveNTFSperms.txt /subdirectories=directoriesonly G:\*.*

The contents of the file created by subinacl are viewable in your favorite text editor:

To restore the permissions on the drive using the file that you backed them up to:

Subinacl /playfile c:\G_driveNTFSperms.txt

Test it out thoroughly in your lab environment before rolling it out to production.

Thanks for reading,

Justin Turner

Technorati Tags: Permissions,Server 2008,Server 2003,Security

Microsoft Certified Master registration...

Justin Turner [MSFT] — Tue, 02 Sep 2008 20:35:00 GMT

Hi Everybody!

Have you heard about the new certification from Microsoft? In the grand scheme of things (as far as MSFT certification goes) it is the next level above the MCITP. (MCITP is what many consider to be the replacement for the old MCSE)

Check out the pic for a better description:

Public registration for the new Master certification is finally open. Full details about the program are available here.

Quick overview:

Cert requirements:
-Mandatory three week training session
-in-class written and lab-based exams
-Pass qualification lab exam

It seems that they haven't updated the registration link yet. Here is the direct ~~link~~.

They've updated it now. Here is the registration page.

Steps to apply:

Fill out application here ($125) IMPORTANT: If the pre-requisite exams don't show up on your MCP transcript then you forfeit the application fee. (70-219 or 70-297, MCSE 2000 or MCSE 2003 or IT Pro: Enterprise Admin (MCITP: EA))
Upload resume (if you are eligible)
Register: Pay (and choose class dates)

Training will be in Redmond only in FY09. Dates for the first training session: (at half the price)

Exchange Server 2007: October 6-25, 2008
SQL Server 2008: October 13 – November 1, 2008
Windows Server 2008: November 3-22, 2008

Pre-recorded Live meeting session linked to from Trika's blog here.

Sysinternals developer answers your toughest Vista questions...

Justin Turner [MSFT] — Fri, 29 Aug 2008 23:37:00 GMT

Hi Everybody!

This should be a pretty good presentation on Windows Vista internals. Mark Russinovich has co-authored several books on Windows Internals, and is one of the developers for all of the cool troubleshooting utilities over at our Sysinternals site.

Springboard Series Virtual Roundtable
Under the Hood: Windows Vista Performance...Need Answers?
Join Mark Russinovich and a panel of industry experts for a LIVE virtual roundtable to explore your top of mind performance issues, common misconfigurations, and tips on how to fix them. From boot times and applets to disk performance and battery life, find out how to optimize Windows Vista and what you can do to improve overall system performance.

Submit your performance questions live during the event or send them in advance to vrtable@microsoft.com.

Save the date!
Wednesday, September 24, 2008
9:00am Pacific Standard Time

Find answers to your Windows Vista adoption questions with resources, tools, monthly straight-talk articles, and upfront guidance based on early adopter and community feedback. To learn more, visit www.microsoft.com/springboard,

Springboard Series: The resource for Windows desktop IT professionals

Quick TIP: The replication operation was preempted

Justin Turner [MSFT] — Fri, 22 Aug 2008 01:20:00 GMT

This tip can help clear some confusion when you encounter the following error while forcing replication:

result 8461 The replication operation was preempted

Here is the output of running repadmin /showreps on a brand new domain controller:

DC=contoso,DC=com
5th_Ward\ContosoDC2 via RPC
objectGuid: 5ed02b33-a6ab-4576-b109-bb688221e6e3

Last attempt @ 2008-08-21 17:51.44 failed, result 8461: The replication operation was preempted.
Last success @ (never).

If you use the verbose switch you will see the following:

DC invocationID:
5ed02b33-a6ab-4576-b109-bb688221e6e3 DO_SCHEDULED_SYNCS WRITEABLE
COMPRESS_CHANGES NO_CHANGE_NOTIFICATIONS NEVER_SYNCED PREEMPTED USNs:
577738/OU, 0/PU Last attempt @ 2008-08-21 17:51:44 was delayed for a
normal reason, result 8461 (0x210d): The replication operation was preempted.
Last success @ (never).

The reason we log this message is because there is an ongoing higher priority replication operation that has not completed yet. In order to see what inbound replication items are queued (and have preempted us) run the following command on the dc:

repadmin /queue

If you have a lot of items queued run:

repadmin /queue >c:\repl_queue.txt

This will output a list of inbound replication tasks that are queued. They are listed by task id in order of priority. The replication operation that we are waiting on is at the top of the list.

If you want to find out what objects/attributes are currently being replicated in to this dc we can enable diagnostic logging for "5 Replication Events" (coming in a future post) However, Steve Patrick(Spat) shows us a cool repadmin cmd in his post here.

repadmin /showchanges ContosoDC2 5ed02b33-a6ab-4576-b109-bb688221e6e3 DC=corp,DC=com >changes.txt

Part 2 of this post will come at a later date. Stay tuned...

RODC compatibility pack available now...

Justin Turner [MSFT] — Thu, 29 May 2008 15:00:04 GMT

Hi Everybody!

I just wanted to write a quick note to let you all know that the RODC compatibility pack for Windows Server 2003 and Windows XP clients is available for download.

Description of the Windows Server 2008 read-only domain controller compatibility pack for Windows Server 2003 clients and for Windows XP clients

http://support.microsoft.com/kb/944043

Not familiar with RODC?

check out these links:

Read-only Active Directory Features

Step-by-Step Guide for Read-only Domain Controllers

Thanks!

Justin

Windows Server 2K8 Reviewers Guide

Justin Turner [MSFT] — Tue, 25 Mar 2008 06:32:00 GMT

Hi Everybody!

This one comes at the request of several customers. Many of you out there are trying to determine which version of Server 2008 you will deploy. For most it comes down to deciding between two of the five major versions: Server 2008 Standard edition and Server 2008 Enterprise edition. Given the amount of features included in the OS and all of the different versions we shipped, trying to determine what version includes what feature can be confusing.

Fortunately we have released a very detailed treatise on this very subject in the form of a 247 page document appropriately titled, Windows Server 2008 Reviewers Guide.

Inside you will find a pretty thorough support matrix and technical nuggets like:

In Standard Edition you are limited to one standalone DFS Namespace. (DFS Root) This limit does not apply to domain-based DFS implementations.
Cross-File Replication for DFS-R is not available in the Standard or Web editions.
Server Core is available in all editions except for Itanium.
Hyper-V is included in Enterprise, Datacenter and Standard editions as long as you don't buy the version that say "without Hyper-V"
TS Licensing in Windows Server 2008 now allows you to track per-user CALs
Still no support for Cluster (failover) in Standard Edition, but you can now have 16 nodes with the Enterprise and Datacenter editions (8 with Itanium)

Here are some screen snags taken right from the guide:

Enjoy!

Quick TIP: Quickly verify AD replication status

Justin Turner [MSFT] — Thu, 29 Nov 2007 10:36:00 GMT

It can be a little tedious to verify replication status in a large Active Directory environment via the Sites and Services snap-in. Here is a command I use quite frequently to check the replication status of all domain controllers:

REPADMIN /SHOWREPL * /CSV >showrepl.csv

View the file in Microsoft Excel and perform the following filtering options to get a good quick overview of replication health:

1. Hide columns A and B

2. Select the row just under Column headers and choose Window / Freeze Pane (In Excel 2007: View tab, Window, Freeze Panes, Freeze Top Row)

3. Highlight the entire spreadsheet and choose Data / Filter / Auto-Filter

4. Click on the down-arrow for the "Last Failure Status" column, and choose "does not
equal" then type in "0" (In Excel 2007: Uncheck the box next to "0")

You are left with a list of domain controllers having replication problems. From a cmd prompt, use:

"net helpmsg ErrorCodeNumber" to identify the replication error

(eg. net helpmsg 1396)

Virtual Server 2005 R2 SP1 and Home Server RC

Justin Turner [MSFT] — Wed, 13 Jun 2007 14:10:20 GMT

Hi Everybody!

This is just an update to let you all know about two software releases:

We just released Virtual Server R2 SP1 on Monday.

You can download it for free from here.

Some of the updates:

Support for hardware-assisted virtualization
Supports up to 256 GB of physical memory, and up to 512 virtual machines
Quick Migration

Today we released the Release Candidate of Windows Home Server.

From the Home Server site:

"With Windows Home Server, you can store your music, photos, and other files on a central hub-like hard drive, accessible from every PC in your house. Protect your files and your PCs with automatic backup and a simple restore process—even gain access to files on your PCs from anywhere with an Internet connection through secure Web access."

You can sign up to get it here. (also free, but not RTM code)

Thanks!

Justin