Active Directory, Cluster and other fun stuff... : Windows 2000

AD User and Group Restore Webcast

justintu — Thu, 05 Apr 2007 08:32:00 GMT

Some time ago I did a webcast presentation on Active Directory User and Group Restore.

I've included the link for those of you that may have missed it.

Check out the on-demand presentation here:

http://www.msusapartnerreadiness.com/WS_abstract.asp?eid=15004864

(Unfortunately registration is required, but that takes only a few seconds)

Let me know if you would like to see more like this one.

Thanks!

Active Directory Forest Recovery...

justintu — Thu, 18 Jan 2007 15:56:00 GMT

The helpdesk phone had been ringing incessantly all day. Many people throughout the AD forest were unable to login to their respective domains. It seems that accounts throughout the forest had somehow been deleted. John, tired from having been up all night watching "White and Nerdy", was called in to help identify what was going on. Fortunately he had recently enabled auditing for account deletions due to a recent problem that he had. After some serious filtering he was able to find the following event in the Security event log:

Event Type: Success Audit
Event Source: Security
Event Category: Account Management
Event ID: 630
Date: 1/17/2007
Time: 12:30:44 AM
User: Contoso\JuniorAdmin
Computer: DisgruntledXP
Description:
User Account Deleted:
Target Account Name: JustinTurner
Target Domain: Contoso
Target AccountID: Justin Turner []DEL:3f4567f2-f90b-493e-81a3-dcfc75596cd7
Caller User Name: JuniorAdmin
Caller Domain: Contoso

This was a little offsetting to say the least. "JuniorAdmin" was the name of the account for one of his Junior Network Administrators that they just fired for getting them into that last mess. He quickly disabled the account, and then attempted to identify what kind of mess they were in now. His heart sank into his stomach when he discovered that JuniorAdmin was a member of the Schema and Enterprise Admins security groups...

I had planned on providing an in-depth discussion about forest recovery, and then realized that there is already more than enough information on this topic. Since I have already advertised this, I will go ahead and provide what I hope will serve as a good general overview, and then point you to a few good resources for the process. There is now a Server 2003 specific forest recovery whitepaper, but the process is unchanged from Windows 2000. There are some additional server 2003 specific goodies added however. (like repadmin /removelingeringobjects)

Before we dive right into the process I want to point out a couple of reasons for why you might have to perform an Active Directory forest recovery.

There are a few reasons that I won't mention, but the two most common I see are:

1. The security of your directory has been compromised either through virus, hacker, or disgruntled employee.

2. A change was made to the schema which needs to be undone.

This really is a big deal, and is not something you want to jump straight to without first consulting Microsoft PSS/CSS/EPS/Platforms Support. (we've had so many different names, I don't remember the current one :-) The team you would be dealing with for this particular issue would be Platforms Directory Services. We want to try to determine what caused the forest failure, and also to ensure that a forest recovery is the best recovery option. An entire forest recovery is obviously one of the last steps you would want to try, so it really is best to explore all other recovery options first.

The five hundred thousand foot overview of the process is:

1. Recover one dc from the forest root domain first from backup.

2. Recover one dc from each of the remaining domains from backup.

3. Restore additional DC's by promoting them via dcpromo.

What follows is a general overview of the process that is outlined in both the Windows 2000 and Server 2003 forest recovery whitepapers referenced earlier. Please reference the particular whitepaper for the specific steps.

There are three major stages of a forest recovery:

Pre-recovery, Recovery, and Post Recovery

Pre-Recovery:

1. Determine the current forest structure/topology

2. Find one trusted backup to use per domain

3. Shutdown, and disconnect if possible, all DC's in the forest

Recovery:

1. Isolate the server, (unplug network cable) and perform a system state restore (ensure you choose the Advanced option to perform a Primary restore of Sysvol) Only choose this option for the first DC in a domain.

2. Verify DC was successfully restored after rebooting

3. Configure DNS

4. Disable Global Catalog (if enabled)

5. Raise RID pool by 100,000

6. Seize FSMO roles

7. Perform metadata cleanup of all other DC's in the forest root domain (also delete DC computer objects for dc's that will not be restored from backup in this domain)

8. Reset machine account twice

9. Reset the krbtgt account password twice

10. Reset all trust passwords twice

11. Restore the first DC in each of the remaining domains from backup (perform Recovery steps 1-10 to recover one dc in each of the remaining domains)

As you restore each DC, you will want to point them to the recovered forest root DC for DNS.

12. Connect the restored DC's back to the network (prior to performing this step ensure that no old dc's are still online)

13. Perform a full replica set sync of AD

14. Enable forest root dc as a GC

15. Seize schema master on forest root dc (if the schema master wasn't the dc that was restored)

16. Recover additional DC's in each of the domains using dcpromo

Post-Recovery:

1. Revert forest back to original DNS configuration

2. Redistribute FSMO roles

3. Enable additional Global catalog servers

4. Get a good system state backup from at least two dc's in each domain

As you can see, this is a very lengthy process. The whitepaper walks you through each step in detail. There is a good index in the paper that has step by step instructions for every single process as well.

Finally I just want to expand on a couple of the items listed above.

Some considerations to take when identifying which DC's to restore:

You will only be restoring one DC per domain. The recovery process will go much quicker if the restored DC was a DNS server, and was not a GC at the time the backup was taken. For some of you this may be an easy choice as you may only be able to find one good backup. I find that when it comes to these situations, many have trouble locating a decent system state backup. (but maybe my view is skewed because the customers that have tested their disaster recovery plan don't call us?) Additionally the process will go by quicker if the DC that you restore in the forest root domain was the Domain Naming and or Schema master. Selecting one that was a RID master will also help. If you are unable to locate a backup from one of these FSMO masters then you will just need to seize the role after the server is restored. To help you out with this there is a cool repadmin command that shows you the last time a dc's system state was backed up: repadmin /showbackup DCName

Don't try to shortcut this process by leaving out steps:

For example: When it says to shutdown and/or disconnect each dc. Do exactly that. We want to ensure that a restored dc does not replicate in bad data from a dc that we forgot to (or couldn't) shutdown. So at the very least ensure that you have your servers that you are restoring disconnected from the network. Also ensure that you reset each of the passwords listed twice. Ensure that you are very thorough with your metadata cleanup stage. Otherwise you will have a not so fun time troubleshooting why your DC's aren't replicating.

There is a typo several times in both whitepapers that greatly changes the meaning of the step:

"Delete server objects and computer objects for all domain controllers in the forest root domain that you are restoring from backup..."

This should read "...that you aren't restoring from backup" I will attempt to get this changed in the whitepapers.

Repadmin is your friend:

There are a few steps where you will use various repadmin commands. Learning repadmin syntax ahead of time will aid in the process. It is also very useful for performing day-to-day AD operations as well.

Some options that you will need to use:

/showbackup

/syncall

/showreps

/options

You may also end up having to use /add, /sync, and /removelingeringobjects as well. However, if you follow the step where it says not to restore a DC that was a GC (or just uncheck that after the restore) then you shouldn't have to worry about lingering objects.

Well that's all I have to say about that. :-) I'll add more later if I think of something else that I left out.

Post any comments or questions you have about this or any other topic that I have blogged about.

Up next: Cluster service failure troubleshooting

Thanks for reading!

Justin

Technorati tags: Active Directory, AD, Server 2003, Disaster Recovery

Missing or corrupt Systemced - part 2

justintu — Thu, 21 Dec 2006 14:46:00 GMT

****EDIT Hey Guys, I goofed on this post: This post discusses a utility used during the course of a Microsoft support call. It is not available to send to customers, and is not available for download as I had originally thought.

The version posted on the download site does not contain the same functionality referenced here. If you email me through the blog I will do my best to help out. Due to my tremendous workload my response may be delayed. If this is an urgent matter then you may want to consider opening up a paid incident with Microsoft Support: http://support.microsoft.com/ ****

This is part 2 of my earlier post on the whole "missing or corrupt system hive" issue. Okay, so we have a copy of the bloated/corrupt registry hive. Now what do we do with it? Chkreg.exe is your friend. Chkreg is a command line utility that you can use to repair a corrupt registry hive. You can also use it to just display registry key size. The majority of the issues that I see are not due to a corrupt system hive, so I use chkreg to help me identify what is taking up all of the hive size.

The ability to view registry key size wasn't added until a later version of chkreg than what is available at Microsoft.com.

The main version that you will find is actually used along with the XP Setup disks. In that version it is placed on disk 6, and after you boot to the recovery console it automatically attempts to repair the system hive. This version does not let you run it from the GUI. You will get this message if you try: "chkreg.exe application cannot be run in Win32 mode."

I thought the newer version was available on our site, unfortunately it looks like you have to call us in order to get this special version of chkreg. With this version of chkreg you get the /S, /O, and /D options.

/S Displays space usage for the bin. When bin is not specified, displays usage for the entire hive.

/O Ordered by size

/D Dump subkeys

I typically put the bloated hive in a folder such as c:\temp, and so my command would be:

chkreg.exe /F c:\temp\system /S /O /D >regbloat.txt

This will output the keys listed largest to smallest to a file called regbloat.txt

Here is an example of two such bloated keys from the txt file:

Size Subkeys

552027 ControlSet002\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

547031 ControlSet001\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

In this example, the same key in both ControlSet keys are causing the registry size problem. This is a known issue that occurs when you have the Spooler service disabled on a Terminal Server.

I remove the bloated keys, and then run chkreg again, but this time with the /C switch to compress the hive. The last step is to swap the hive back out via recovery console in order to boot off of it.

There is a utility that you can use to correct the problem called scrubber.exe, but it only corrects the issue if it is due to the issue mentioned here: KB 277222

Tune in next time when I will discuss: Active Directory Forest recovery or something else equally exciting. :)

Thanks for viewing!

Justin

Technorati tags: Cluster, Setup, Windows 2000

Missing or corrupt Systemced... What's that?

justintu — Sat, 16 Dec 2006 16:33:00 GMT

Technorati tags: Cluster, Setup, Boot, Windows 2000

Part 1 of 2

The on-call pager went off at two in the morning. John rushed in to discover that one of their main Windows 2000 file and print servers was sitting at a black screen with the following error displayed:

Windows 2000 could not start because the following file is missing or corrupt:

\WINNT\SYSTEM32\CONFIG\SYSTEMced

Well was it missing or was it corrupt!? He booted the server into Recovery Console, and went to the location mentioned in the error. It appeared to be missing---he couldn't find a file called "Systemced" anywhere...

There really isn't a file called SYSTEMced. The error message has just overwritten the message that normally appears there during system boot: "For troubleshooting and advanced startup options for Windows 2000, press F8."

Here is the message that is normally displayed at this point in the boot process: (notice that the ced from SYSTEMced is actually the last part of the word "advanced")

The method of recovery for this issue is actually documented fairly well here: KB 269075 here: KB 323148 and here: KB 302829 There are several other articles that describe various methods of correcting the problem, but these cover the basic steps required.

Usually when I see the issue it is because the system hive is too large to load into memory. In Windows 2000 (and NT 4) we are limited to 16 MB of memory at boot time. You will likely first run into this problem when the system hive reaches just a little over 10 MB. Thankfully this memory limitation has been greatly increased in Server 2003.

Essentially what you do is boot using an alternate system hive, and then either restore the hive from backup, (in the case of a corrupted hive) or clean up space in the system hive if the boot failure is caused by the system hive being too large.

This is a very common problem. I've seen it three times in the past week. (and countless times in the last few years) Some customers have this problem so often that they have a process in place to check the size of the system hive before they reboot a server. (you know who you are ;-) ) Hopefully with this and the next post, I can convince some of you to correct the problem that causes the bloated hive in the first place so that you never have to see this error on reboot.

In the next post I will go over the chkreg.exe utility that I use to correct this problem, and ways to prevent it from happening in the future.

Stay tuned...

Justin