Active Directory, Cluster and other fun stuff... : Active Directory

Replication error 8464 after schema upgrade

justintu — Wed, 28 Oct 2009 16:00:25 GMT

Problem Environment

Contoso has a very large branch-office deployment of Active Directory. Each branch-office is an Active Directory site, and each DC is a GC because of potentially unreliable WAN links.

Any time a Scheme upgrade is performed in which the Partial Attribute Set (PAS) is updated, replication fails for numerous partitions until all hub DCs have been updated. The last time they upgraded the Schema, it was several months before all DCs received the updated PAS.

Details

Active Directory replication failed after upgrading the schema for Exchange 2007 with an event id 8464.

“Synchronization attempt failed because the destination DC is currently waiting to synchronize new partial attributes from source. This condition is normal if a recent schema change modified the partial attribute set. The destination partial attribute set is not a subset of source partial attribute set.”

They were getting this message because the DCs in their hub site had not received the updated Partial Attribute Set (PAS) for the affected partitions.

It had been several days since the Schema was upgraded, and given enough time, this problem would eventually correct itself. Our goal was to understand why it took so long for all DCs to reach replication convergence, and to decrease the time it takes for all domain controllers to receive the updated PAS.

In order to visualize their replication environment, I had them run our AD Topology Diagrammer tool. (ADTD or sometimes referred to by its former name: AD Map) The Visio diagram revealed one hub site and over a thousand branch-office sites. Each branch office has replication connections to the hub site only.

Cause

Replication of the updated PAS did not occur because of the following:

Each of the bridgehead domain controllers in the hub site have over 200 replication connections for each partition. The customer has a DC in each branch office, each office is its own site, and each DC is a GC. The sheer amount of replication connections interfered with the timely update of the PAS. This environment would be a great candidate for RODCs (since there would not be any outbound connections from the branch office sites), but now was not the time to talk topology redesign.

Solution

Our immediate goal, to get this resolved quickly, was to get the PAS updated on each DC in the Hub site for both domain partitions reported in the events.

First we had to identify which GC's have the updated PAS. Here is the repadmin.exe command that we used:

The following command will dump the PAS from every DC in the forest for the partition specified in the DN path *. The resulting output lets us know which GCs haven’t gotten the updates yet:

“repadmin /showattr gc: dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Corp.txt”

* Since “dc=corp,dc=contoso,dc=com” is specified in the above command, it will dump the PAS for the “Corp” domain partition.

The following command would be used to dump the PAS for the “Branches” domain partition:

“repadmin /showattr gc: dc=branches,dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset >partialattributeset-Branches.txt”

The value of interest in the output is listed after the "v1.cAttrs =" text.

If you want to check a single GC, you could run:

repadmin /showattr DCName dc=corp,dc=contoso,dc=com /gc /atts:partialattributeset

During this issue we came up with several methods to get these servers updated. I will list out each of the methods (A, B, and C) with the top choice listed first.

A. Set up a replication connection, disable inbound and outbound replication, and force replication:

1. Create a manual replication connection to a DC that already has the updated partial attribute set (unless you already have a connection to this dc).

“repadmin /add <Naming Context> <Destination DC> <Source DC> /readonly (if needed)”

2. Disable inbound and outbound replication (one very fast way to clear out the queue).

“repadmin /options dc_name +DISABLE_INBOUND_REPL”

“repadmin /options dc_name +DISABLE_OUTBOUND_REPL”

3. Force replication with the DC over the newly created connection.

“Repadmin /replicate <dc-with-low-cAttr> <DC-with-high-cAttr> <DNpath> /force [/readonly if needed]”

4. Run the repadmin command to check the inbound replication queue.

“repadmin /queue”

You should just see the one item queued, but I have seen a few more replication requests sneak in on a very busy dc.

5. You should be able to re-enable inbound and outbound replication immediately.

“repadmin /options dc_name -DISABLE_INBOUND_REPL”

“repadmin /options dc_name -DISABLE_OUTBOUND_REPL”

B. Rehost the partition:

1. “repadmin /rehost <rehosted GC hostname> <DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition>”

If the command returns a “replication operation was preempted” error, then perform the following steps:

2. Run “repadmin /rehost specifying the name of both the target source DC (the good DC hosting a writable copy of the rehosted partition).”

“repadmin /rehost <rehosted GC hostname> < DN path of rehosted naming context > <fully qualified DNS name of good DC hosting a writable copy of the domain partition >.”

3. Press Ctrl+C to stop the rehost command before the preemption error occurs.

4. Rerun the repadmin /rehost command, which should complete normally.

“repadmin /rehost < rehosted GC hostname > < DN path of rehosted naming context > < fully qualified DNS name of good DC hosting a writable copy of the domain partition>”

5. Run repadmin /showreps /v against the destination DC and confirm replication of the read-only partition completed.

C. Move extraneous replication connections off the DC that needs to be updated:

1. This can be accomplished by moving the DC to a test site (so that it has fewer replication connections). OR:

2. Temporarily configure this server so that it is not a preferred bridgehead server so that the KCC removes the connections

As always, let me know if you have any questions!

Justin

Quick TIP: The replication operation was preempted

justintu — Fri, 22 Aug 2008 01:20:00 GMT

This tip can help clear some confusion when you encounter the following error while forcing replication:

result 8461 The replication operation was preempted

Here is the output of running repadmin /showreps on a brand new domain controller:

DC=contoso,DC=com
5th_Ward\ContosoDC2 via RPC
objectGuid: 5ed02b33-a6ab-4576-b109-bb688221e6e3

Last attempt @ 2008-08-21 17:51.44 failed, result 8461: The replication operation was preempted.
Last success @ (never).

If you use the verbose switch you will see the following:

DC invocationID:
5ed02b33-a6ab-4576-b109-bb688221e6e3 DO_SCHEDULED_SYNCS WRITEABLE
COMPRESS_CHANGES NO_CHANGE_NOTIFICATIONS NEVER_SYNCED PREEMPTED USNs:
577738/OU, 0/PU Last attempt @ 2008-08-21 17:51:44 was delayed for a
normal reason, result 8461 (0x210d): The replication operation was preempted.
Last success @ (never).

The reason we log this message is because there is an ongoing higher priority replication operation that has not completed yet. In order to see what inbound replication items are queued (and have preempted us) run the following command on the dc:

repadmin /queue

If you have a lot of items queued run:

repadmin /queue >c:\repl_queue.txt

This will output a list of inbound replication tasks that are queued. They are listed by task id in order of priority. The replication operation that we are waiting on is at the top of the list.

If you want to find out what objects/attributes are currently being replicated in to this dc we can enable diagnostic logging for "5 Replication Events" (coming in a future post) However, Steve Patrick(Spat) shows us a cool repadmin cmd in his post here.

repadmin /showchanges ContosoDC2 5ed02b33-a6ab-4576-b109-bb688221e6e3 DC=corp,DC=com >changes.txt

Part 2 of this post will come at a later date. Stay tuned...

RODC compatibility pack available now...

justintu — Thu, 29 May 2008 15:00:04 GMT

Hi Everybody!

I just wanted to write a quick note to let you all know that the RODC compatibility pack for Windows Server 2003 and Windows XP clients is available for download.

Description of the Windows Server 2008 read-only domain controller compatibility pack for Windows Server 2003 clients and for Windows XP clients

http://support.microsoft.com/kb/944043

Not familiar with RODC?

check out these links:

Read-only Active Directory Features

Step-by-Step Guide for Read-only Domain Controllers

Thanks!

Justin

Windows Server 2K8 Reviewers Guide

justintu — Tue, 25 Mar 2008 06:32:00 GMT

Hi Everybody!

This one comes at the request of several customers. Many of you out there are trying to determine which version of Server 2008 you will deploy. For most it comes down to deciding between two of the five major versions: Server 2008 Standard edition and Server 2008 Enterprise edition. Given the amount of features included in the OS and all of the different versions we shipped, trying to determine what version includes what feature can be confusing.

Fortunately we have released a very detailed treatise on this very subject in the form of a 247 page document appropriately titled, Windows Server 2008 Reviewers Guide.

Inside you will find a pretty thorough support matrix and technical nuggets like:

In Standard Edition you are limited to one standalone DFS Namespace. (DFS Root) This limit does not apply to domain-based DFS implementations.
Cross-File Replication for DFS-R is not available in the Standard or Web editions.
Server Core is available in all editions except for Itanium.
Hyper-V is included in Enterprise, Datacenter and Standard editions as long as you don't buy the version that say "without Hyper-V"
TS Licensing in Windows Server 2008 now allows you to track per-user CALs
Still no support for Cluster (failover) in Standard Edition, but you can now have 16 nodes with the Enterprise and Datacenter editions (8 with Itanium)

Here are some screen snags taken right from the guide:

Enjoy!

Quick TIP: Quickly verify AD replication status

justintu — Thu, 29 Nov 2007 10:36:00 GMT

It can be a little tedious to verify replication status in a large Active Directory environment via the Sites and Services snap-in. Here is a command I use quite frequently to check the replication status of all domain controllers:

REPADMIN /SHOWREPL * /CSV >showrepl.csv

View the file in Microsoft Excel and perform the following filtering options to get a good quick overview of replication health:

1. Hide columns A and B

2. Select the row just under Column headers and choose Window / Freeze Pane (In Excel 2007: View tab, Window, Freeze Panes, Freeze Top Row)

3. Highlight the entire spreadsheet and choose Data / Filter / Auto-Filter

4. Click on the down-arrow for the "Last Failure Status" column, and choose "does not
equal" then type in "0" (In Excel 2007: Uncheck the box next to "0")

You are left with a list of domain controllers having replication problems. From a cmd prompt, use:

"net helpmsg ErrorCodeNumber" to identify the replication error

(eg. net helpmsg 1396)

Quick TIP: Force FRS replication

justintu — Fri, 27 Apr 2007 15:16:00 GMT

For this tip you will need a somewhat newer version of ntfrsutl.exe

You can grab a version out of the Service Pack 2 Support Tools download here.

Beginning with the version of ntfrsutl.exe in KB 823230 we have the ability to force FRS replication to occur across site boundaries immediately instead of waiting for the schedule to open up.

Here is the command's syntax:

ntfrsutl forcerepl [computer] /r SetName /p PartnerDnsName
= Force FRS to start a replication cycle ignoring the schedule

The PartnerDNSName is the FQDN of the server that you want to source from.

Here is an example using a DC Name of ContosoDC1 and a PartnerDNSName of ContosoDC2:

ntfrsutl forcerepl contosodc1 /r "domain system volume (sysvol share)" /p ContosoDC2.Contoso.com

Running the command initiates replication, and returns the following information:

LocalComputerName = contosodc1
ReplicaSetGuid = (null)
CxtionGuid = (null)
ReplicaSetName = domain system volume (sysvol share)
PartnerDnsName = ContosoDC2.Contoso.com

As you can see there are two additional parameters that you can specify, ReplicaSetGuid and CxtionGuid, but neither are required.

AD User and Group Restore Webcast

justintu — Thu, 05 Apr 2007 08:32:00 GMT

Some time ago I did a webcast presentation on Active Directory User and Group Restore.

I've included the link for those of you that may have missed it.

Check out the on-demand presentation here:

http://www.msusapartnerreadiness.com/WS_abstract.asp?eid=15004864

(Unfortunately registration is required, but that takes only a few seconds)

Let me know if you would like to see more like this one.

Thanks!

Active Directory Forest Recovery...

justintu — Thu, 18 Jan 2007 15:56:00 GMT

The helpdesk phone had been ringing incessantly all day. Many people throughout the AD forest were unable to login to their respective domains. It seems that accounts throughout the forest had somehow been deleted. John, tired from having been up all night watching "White and Nerdy", was called in to help identify what was going on. Fortunately he had recently enabled auditing for account deletions due to a recent problem that he had. After some serious filtering he was able to find the following event in the Security event log:

Event Type: Success Audit
Event Source: Security
Event Category: Account Management
Event ID: 630
Date: 1/17/2007
Time: 12:30:44 AM
User: Contoso\JuniorAdmin
Computer: DisgruntledXP
Description:
User Account Deleted:
Target Account Name: JustinTurner
Target Domain: Contoso
Target AccountID: Justin Turner []DEL:3f4567f2-f90b-493e-81a3-dcfc75596cd7
Caller User Name: JuniorAdmin
Caller Domain: Contoso

This was a little offsetting to say the least. "JuniorAdmin" was the name of the account for one of his Junior Network Administrators that they just fired for getting them into that last mess. He quickly disabled the account, and then attempted to identify what kind of mess they were in now. His heart sank into his stomach when he discovered that JuniorAdmin was a member of the Schema and Enterprise Admins security groups...

I had planned on providing an in-depth discussion about forest recovery, and then realized that there is already more than enough information on this topic. Since I have already advertised this, I will go ahead and provide what I hope will serve as a good general overview, and then point you to a few good resources for the process. There is now a Server 2003 specific forest recovery whitepaper, but the process is unchanged from Windows 2000. There are some additional server 2003 specific goodies added however. (like repadmin /removelingeringobjects)

Before we dive right into the process I want to point out a couple of reasons for why you might have to perform an Active Directory forest recovery.

There are a few reasons that I won't mention, but the two most common I see are:

1. The security of your directory has been compromised either through virus, hacker, or disgruntled employee.

2. A change was made to the schema which needs to be undone.

This really is a big deal, and is not something you want to jump straight to without first consulting Microsoft PSS/CSS/EPS/Platforms Support. (we've had so many different names, I don't remember the current one :-) The team you would be dealing with for this particular issue would be Platforms Directory Services. We want to try to determine what caused the forest failure, and also to ensure that a forest recovery is the best recovery option. An entire forest recovery is obviously one of the last steps you would want to try, so it really is best to explore all other recovery options first.

The five hundred thousand foot overview of the process is:

1. Recover one dc from the forest root domain first from backup.

2. Recover one dc from each of the remaining domains from backup.

3. Restore additional DC's by promoting them via dcpromo.

What follows is a general overview of the process that is outlined in both the Windows 2000 and Server 2003 forest recovery whitepapers referenced earlier. Please reference the particular whitepaper for the specific steps.

There are three major stages of a forest recovery:

Pre-recovery, Recovery, and Post Recovery

Pre-Recovery:

1. Determine the current forest structure/topology

2. Find one trusted backup to use per domain

3. Shutdown, and disconnect if possible, all DC's in the forest

Recovery:

1. Isolate the server, (unplug network cable) and perform a system state restore (ensure you choose the Advanced option to perform a Primary restore of Sysvol) Only choose this option for the first DC in a domain.

2. Verify DC was successfully restored after rebooting

3. Configure DNS

4. Disable Global Catalog (if enabled)

5. Raise RID pool by 100,000

6. Seize FSMO roles

7. Perform metadata cleanup of all other DC's in the forest root domain (also delete DC computer objects for dc's that will not be restored from backup in this domain)

8. Reset machine account twice

9. Reset the krbtgt account password twice

10. Reset all trust passwords twice

11. Restore the first DC in each of the remaining domains from backup (perform Recovery steps 1-10 to recover one dc in each of the remaining domains)

As you restore each DC, you will want to point them to the recovered forest root DC for DNS.

12. Connect the restored DC's back to the network (prior to performing this step ensure that no old dc's are still online)

13. Perform a full replica set sync of AD

14. Enable forest root dc as a GC

15. Seize schema master on forest root dc (if the schema master wasn't the dc that was restored)

16. Recover additional DC's in each of the domains using dcpromo

Post-Recovery:

1. Revert forest back to original DNS configuration

2. Redistribute FSMO roles

3. Enable additional Global catalog servers

4. Get a good system state backup from at least two dc's in each domain

As you can see, this is a very lengthy process. The whitepaper walks you through each step in detail. There is a good index in the paper that has step by step instructions for every single process as well.

Finally I just want to expand on a couple of the items listed above.

Some considerations to take when identifying which DC's to restore:

You will only be restoring one DC per domain. The recovery process will go much quicker if the restored DC was a DNS server, and was not a GC at the time the backup was taken. For some of you this may be an easy choice as you may only be able to find one good backup. I find that when it comes to these situations, many have trouble locating a decent system state backup. (but maybe my view is skewed because the customers that have tested their disaster recovery plan don't call us?) Additionally the process will go by quicker if the DC that you restore in the forest root domain was the Domain Naming and or Schema master. Selecting one that was a RID master will also help. If you are unable to locate a backup from one of these FSMO masters then you will just need to seize the role after the server is restored. To help you out with this there is a cool repadmin command that shows you the last time a dc's system state was backed up: repadmin /showbackup DCName

Don't try to shortcut this process by leaving out steps:

For example: When it says to shutdown and/or disconnect each dc. Do exactly that. We want to ensure that a restored dc does not replicate in bad data from a dc that we forgot to (or couldn't) shutdown. So at the very least ensure that you have your servers that you are restoring disconnected from the network. Also ensure that you reset each of the passwords listed twice. Ensure that you are very thorough with your metadata cleanup stage. Otherwise you will have a not so fun time troubleshooting why your DC's aren't replicating.

There is a typo several times in both whitepapers that greatly changes the meaning of the step:

"Delete server objects and computer objects for all domain controllers in the forest root domain that you are restoring from backup..."

This should read "...that you aren't restoring from backup" I will attempt to get this changed in the whitepapers.

Repadmin is your friend:

There are a few steps where you will use various repadmin commands. Learning repadmin syntax ahead of time will aid in the process. It is also very useful for performing day-to-day AD operations as well.

Some options that you will need to use:

/showbackup

/syncall

/showreps

/options

You may also end up having to use /add, /sync, and /removelingeringobjects as well. However, if you follow the step where it says not to restore a DC that was a GC (or just uncheck that after the restore) then you shouldn't have to worry about lingering objects.

Well that's all I have to say about that. :-) I'll add more later if I think of something else that I left out.

Post any comments or questions you have about this or any other topic that I have blogged about.

Up next: Cluster service failure troubleshooting

Thanks for reading!

Justin

Technorati tags: Active Directory, AD, Server 2003, Disaster Recovery

Active Directory Puzzle from Technet Mag...

justintu — Thu, 04 Jan 2007 14:29:43 GMT

Check out this cool poster size image now available for download from our website.

Link courtesy of Michael Kleef.

The "Active Directory Component Jigsaw" picture is a pretty cool overview of AD that was originally only available to TechNet Magazine subscribers. Full picture size is over 8 MB.

Enjoy!

Cluster service failure update...

justintu — Thu, 21 Dec 2006 09:52:00 GMT

Just a quick note to say that they did update KB 269229 with my comment about requiring the SERVICE account to be included in the "Impersonate client after authentication" user right. (reference this post for background info)

From the article:

"Note If you create a Group Policy setting to update the Impersonate a client after authentication rights policy setting, make sure that the Cluster service account is listed in the policy setting in addition to the Local Administrators group and the account that is called SERVICE."

It is still easy to overlook this in the article, so I don't anticipate and end to these issues. If any of you find this requirement missing from other MSFT documentation then please comment the article, or post a comment here and I will get it corrected.

Thanks,

Justin

Technorati tags: Cluster, Active Directory

Cluster service failure after AD lockdown...

justintu — Thu, 14 Dec 2006 14:22:00 GMT

Users were unable to connect to their shares. John discovered that the Cluster service wasn't started, and that any attempts to start it resulted in an error 1068. He attempted to ping the virtual server's IP address and it returned a "request timed out" message. He got the same error when trying to ping the cluster node's public adapter.

When he got to the node he found the Cluster service in a Starting state. He soon discovered that he had no network connectivity to or from either Cluster node, and that their network cards were missing from "Network Connections" The only changes made to the network were just a few minor group policy settings to lock down permissions a bit. Maybe that had something to do with this? It looked like it was going to be a long night...

This is another fairly common problem. This is not really just a Cluster problem, but that is usually how it is presented to me. Of course if networking is not functional, then Cluster isn't going to work either. :) I have worked at least three of these issues in the last two months, and thought it warranted discussion since there isn't a public KB article on this particular scenario yet. I hope to fully document every error encountered here, so that others may find this post when they run into this situation. (KB articles sometimes take a while to get published)

System event log:

SAM event ID: 12291 "SAM failed to start the TCP/IP or SPX/IPX listening thread"
IPSec event ID: 4292 "The IPSec driver has entered Block mode."
DfsSvc event ID: 14523 "DFS could not contact any DC for Domain DFS operations."

Application event log:

EventSystem event ID: 4609 "The COM+ Event System detected a bad return code during its internal processing. HRESULT was 80004015 from line 142 of d:\nt\com\complus\src\events\tier2\service.cpp."

Other problems discovered with this node:

The Com+ Event System, Network Connections and Shell Hardware Detection services were in a Starting state.

The following services failed to start:

Cluster Service: Error 1068: The dependency service or group failed to start.
File Replication: Error 1068: The dependency service or group failed to start.
---dependencies opens up a window titled "Service Dependencies" and the message is: Wind32: Access is denied.
IPSEC Services: Error 1899: The endpoint mapper database entry could not be created.
System Event Notification: Error 1068: The dependency service or group failed to start.
--trying to view the dependencies on the server returns the following message: Win32: Access is denied
Task Scheduler: "The endpoint mapper database could not be loaded"

We have three services failing with "the dependency service or group failed to start."
When we try to view the dependencies we get an access denied message.

Let's look in the registry to see what each of these services depend on:

Cluster service:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ClusSvc
DependOnService:

ClusNet
RpcSs
W32Time
NetMan

File Replication:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs
DependOnService:

EventLog
RpcSs
EventSystem

System Event Notification:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SENS
DependOnService:

EventSystem

So the common dependencies are RpcSs and EventSystem

RpcSs is the Remote Procedure Call (RPC) service, and EventSystem is the Com+ Event System service. We know from earlier that Com+ Event System is one of the services stuck in a Starting state, so that is why the File Replication and System Event Notification services haven't started. One of the other dependencies for the Cluster service is NetMan, which is the Network Connections service. Network Connections is also one of the services stuck in a Starting state.

So now the real question is: Why are the Com+ Event System and Network Connections services not starting?

If we view the dependencies for these two services, we just find RpcSs listed. So it all boils down to RPC. However, the Remote Procedure Call (RPC) service is actually started.

If you do a search in the knowledge base on these errors, you are likely to come across this article:

909444 Systems that have changed the default Access Control List permissions on the %windir%\registration directory may experience various problems after you install the Microsoft Security Bulletin MS05-051 for COM+ and MS DTC

This discusses changes made by a hotfix that would cause these problems. The fix is to correct NTFS permissions on the %SystemRoot%\Registration directory. However the permissions here are the same as in the article.

You may also come across this one:

916254 COM+-related events may be logged in Event Viewer when you install Windows XP Service Pack 2 and join the computer to a domain

Most would come across this second article and instantly dismiss it since it says "Windows XP Service Pack 2." However, we have a lot of the same symptoms, and since XP SP2 and Server 2003 SP1 include a lot of the same security changes it warrants further investigation.
One of the security changes in SP1 for Windows Server 2003 was to change the Logon Account used for RPC.
RPC use to log on as Local System and now uses an account with less privileges: Network Service.

The article states that this issue occurs if the SERVICE account is missing from the policy setting "Impersonate a client after authentication"

We can see if SERVICE is missing from this policy by performing the following steps:

1. Open up Local Security Policy in order to see what the effective settings are:

Start, Run, secpol.msc

2. Expand Local Policies, User Rights Assignment and then open up "Impersonate a client after authentication"

At minimum the following should be listed: Administrators and SERVICE

The problem that I have seen recently happens when someone decides to change the "Impersonate a client after authentication" user right in group policy. Typically how it goes is they decide to lockdown their servers, and only give specific accounts certain privileges. However, after incorrectly removing the SERVICE account from this privilege the server loses all network connectivity. Fortunately this problem doesn't show up until after a reboot. (You have an opportunity to identify that the problem exists before causing a major outage of all servers in a large OU.)

The fix is simple for the servers that haven't been restarted:

1. Correct the policy and then force group policy to be reapplied. (gpupdate /force)

(To correct the policy: just add SERVICE and Administrators to this policy setting in addition to the other ones defined)

If you have already rebooted the servers after applying the incorrect policy settings they will not be corrected by just simply changing the policy back since they have already lost network access. (unless the policy change was made locally to begin with)

1. Export the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RpcSs

2. In the services snap-in: Change Remote Procedure Call (RPC) to start up with the Local System account instead of Network Service, and then reboot

3. At this point the majority of the services should be started and we should now have network access. Ensure that the offending group policy has been corrected with the proper accounts, force group policy to apply, (gpupdate /force) and then reboot.

4. Change the logon account for Remote Procedure Call (RPC) service back to Network Service by importing the reg file that you exported in step one, and then reboot. Alternatively: navigate to the following reg key and then reboot

Technorati tags: Active Directory, Cluster, Windows Server 2003

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RpcSs

Change the ObjectName value from LocalSystem to: NT Authority\NetworkService

For more information regarding this security setting see article on Technet: SeImpersonatePrivilege
I have commented KB 269229 to reflect the requirement for SERVICE to be included in this User Right.

Please let me know if you like the format of this post or if you have any questions.

Until next time.

Thanks,

Justin Turner
This posting is provided "AS IS" with no warranties, and confers no rights.

Active Directory User and Group Restore...

justintu — Thu, 07 Dec 2006 13:04:41 GMT

Problem: John was awakened from a dead sleep by the Benny Hill theme ring tone he had just downloaded to his cell phone earlier that evening----one of his junior Network Admins had accidentally deleted one of their larger account OU's. In the blink of an eye five thousand user objects had been removed from AD. John was horrified to hear that his company's CEO was among one of the many deleted accounts. He took a second to regain his composure. No problem, he thought to himself, they ran system state backups nightly, and had practiced restoring them just in case they ran into problems like this one.

They successfully restored the system state backup of one of their DC's and issued the ntdsutil "Authoritative Restore" command for that particular OU. After about two hours of frantic work everything appeared to be back to normal.

The help desk calls started rolling in the next day. Many users were reporting that they were being denied access to resources that they once had access to. It appeared that the user's group membership was out of whack. After some investigation, John discovered that while some DC's had correct group membership for the affected users, many others did not...

For my first technical post I thought I would go over one of the more common Active Directory issues that I see: AD user and group restore.

This problem manifests itself in any numbers of ways, but most commonly someone just accidentally deletes an OU.

The procedure itself is fairly straightforward, (the recovery process is very well documented) but there are some peculiarities that you sometimes run into during the process.

Here is a simplified list of steps required to recover users along with their group membership:

---------------------------------------------------------------------------------

*** Steps summarized from KB 840001 *** This article is updated regularly. Please consult the article for the latest information.

Step 1

Determine if the deletion has already replicated to all of your Global Catalog DC's. If there is a latent GC/DC that has not processed any part of the deletion, then disconnect it from the network or disable inbound replication with the repadmin.exe command. (repadmin /options dc_name +DISABLE_INBOUND_REPL)

If there isn't a latent DC, then you will have to find a recent system state backup from a GC/DC from the domain where the deletion occurred.

Step 2

Reboot the recovery DC into DS restore mode and perform an authoritative restore (or if you did not find a latent DC, restore the system state and then perform the authoritative restore)

Step 3

If you did not have to restore the system state, reboot normally. Reboot with the network cable disconnected if you did restore the system state, and then disable inbound replication with the repadmin command.

Initiate outbound replication to all dc's in the domain (and gc's in the forest) with the repadmin /syncall command:

repadmin /syncall dc_name /d /e /P YourDomainDN (such as dc=microsoft,dc=com)

/d: identifies servers by distinguished name in messages.

/e: Enterprise; includes partners in all sites.

/P Pushes changes outward from the home server.

*Here is where you have to be careful, and why it is important to fully understand all of the commands that you run in a production environment.

Some documentation states to use the /A option as well with the repadmin command. The /A option tells repadmin to synchronize all directory partitions that are held on the server. This would be fine to use in a small environment, but probably not a good idea if you have a lot of DC's, or have DC's with slow WAN links. If you are just restoring users and groups then we just need to synchronize the domain partition.

Step 4

Wait for end to end replication to occur, and then proceed with fixing up group membership:

(here it gets a little tricky)

If all groups are LVR enabled then group membership was corrected when you issued the authoritative restore command. Information on LVR is available at the bottom of this post.

Method 1: If you performed the authoritative restore on a DC with 2003 SP1 then it created one or more ldif files that you can import with ldifde.exe to correct group membership. (ldifde -i -k -f filename.ldf -s Recovery_DCname)

* Be sure to specify the -s option along with the name of the name of the recovery dc name or it may connect to another DC and the command will fail.

Method 2: If this is a Windows 2000 DC or a Server 2003 DC without SP1 then you need to correct group membership manually. We have a utility called groupadd.exe that can aid you using this method.

Method 3: If you don't mind rolling back group membership to the time that the system state was created, you can boot back into DS restore mode and issue another authoritative restore command. (here you need to issue the command against the container that your groups exist in)

Step 5

Outbound replicate again via repadmin, and then re-enable inbound replication.

repadmin /options dc_name -DISABLE_INBOUND_REPL

Finally perform another system state backup, and you are done.

---------------------------------------------------------------------------------

If for some reason you do not have a good system state backup or a latent dc, you can manually "undelete" individual users on Server 2003.

There are steps in KB 840001 describing how this is done with ldp.exe, but alternatively you can use adrestore.exe.

Please note that only a limited set of attributes are restored with the user:

SID, ObjectGUID, LastKnownParent, SAMAccountName

Beginning with SP1, SidHistory is also retained.

Information on LVR (Linked Value Replication)

---------------------------------------------------------------------------------

The following information was taken from a Server 2003 class I attended. I couldn't find a good online source for it.

"In Windows 2000, the smallest unit of data that can be replicated is an attribute. In the case of multi-valued attributes, such as the "member" attribute on a group, a change to one value of the multi-value prompts replication of the entire attribute including all of the unchanged values as well" With LVR enabled groups only changes made to the individual values of multi-valued attributes get replicated.

LVR is enabled on the switch to Windows Server 2003 Forest functional mode or 2003 Interim mode.

"Existing groups are not immediately converted to LVR. Instead, the groups are converted only as group membership changes. Individual members are added or deleted using LVR, and the old style attribute is no longer replicated. When new groups are created, all new members are stored and replicated using LVR "

In order to determine if LVR is enabled use the following repadmin command: repadmin /bind %servername% then check to see if "Linked_Value_Replication" states "Yes"

---------------------------------------------------------------------------------

Wow, sorry for the long post. My intent with this post was to just provide a broad overview of the process, and throw in some of the useful tips that I have discovered along the way during countless calls.

Please let me know if you like or dislike the style presented with this post, and let me know if you have any questions.