Active Directory, Cluster and other fun stuff... : Setup

Missing or corrupt Systemced - part 2

justintu — Thu, 21 Dec 2006 14:46:00 GMT

****EDIT Hey Guys, I goofed on this post: This post discusses a utility used during the course of a Microsoft support call. It is not available to send to customers, and is not available for download as I had originally thought.

The version posted on the download site does not contain the same functionality referenced here. If you email me through the blog I will do my best to help out. Due to my tremendous workload my response may be delayed. If this is an urgent matter then you may want to consider opening up a paid incident with Microsoft Support: http://support.microsoft.com/ ****

This is part 2 of my earlier post on the whole "missing or corrupt system hive" issue. Okay, so we have a copy of the bloated/corrupt registry hive. Now what do we do with it? Chkreg.exe is your friend. Chkreg is a command line utility that you can use to repair a corrupt registry hive. You can also use it to just display registry key size. The majority of the issues that I see are not due to a corrupt system hive, so I use chkreg to help me identify what is taking up all of the hive size.

The ability to view registry key size wasn't added until a later version of chkreg than what is available at Microsoft.com.

The main version that you will find is actually used along with the XP Setup disks. In that version it is placed on disk 6, and after you boot to the recovery console it automatically attempts to repair the system hive. This version does not let you run it from the GUI. You will get this message if you try: "chkreg.exe application cannot be run in Win32 mode."

I thought the newer version was available on our site, unfortunately it looks like you have to call us in order to get this special version of chkreg. With this version of chkreg you get the /S, /O, and /D options.

/S Displays space usage for the bin. When bin is not specified, displays usage for the entire hive.

/O Ordered by size

/D Dump subkeys

I typically put the bloated hive in a folder such as c:\temp, and so my command would be:

chkreg.exe /F c:\temp\system /S /O /D >regbloat.txt

This will output the keys listed largest to smallest to a file called regbloat.txt

Here is an example of two such bloated keys from the txt file:

Size Subkeys

552027 ControlSet002\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

547031 ControlSet001\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

In this example, the same key in both ControlSet keys are causing the registry size problem. This is a known issue that occurs when you have the Spooler service disabled on a Terminal Server.

I remove the bloated keys, and then run chkreg again, but this time with the /C switch to compress the hive. The last step is to swap the hive back out via recovery console in order to boot off of it.

There is a utility that you can use to correct the problem called scrubber.exe, but it only corrects the issue if it is due to the issue mentioned here: KB 277222

Tune in next time when I will discuss: Active Directory Forest recovery or something else equally exciting. :)

Thanks for viewing!

Justin

Technorati tags: Cluster, Setup, Windows 2000

Cluster service failure update...

justintu — Thu, 21 Dec 2006 09:52:00 GMT

Just a quick note to say that they did update KB 269229 with my comment about requiring the SERVICE account to be included in the "Impersonate client after authentication" user right. (reference this post for background info)

From the article:

"Note If you create a Group Policy setting to update the Impersonate a client after authentication rights policy setting, make sure that the Cluster service account is listed in the policy setting in addition to the Local Administrators group and the account that is called SERVICE."

It is still easy to overlook this in the article, so I don't anticipate and end to these issues. If any of you find this requirement missing from other MSFT documentation then please comment the article, or post a comment here and I will get it corrected.

Thanks,

Justin

Technorati tags: Cluster, Active Directory

Missing or corrupt Systemced... What's that?

justintu — Sat, 16 Dec 2006 16:33:00 GMT

Technorati tags: Cluster, Setup, Boot, Windows 2000

Part 1 of 2

The on-call pager went off at two in the morning. John rushed in to discover that one of their main Windows 2000 file and print servers was sitting at a black screen with the following error displayed:

Windows 2000 could not start because the following file is missing or corrupt:

\WINNT\SYSTEM32\CONFIG\SYSTEMced

Well was it missing or was it corrupt!? He booted the server into Recovery Console, and went to the location mentioned in the error. It appeared to be missing---he couldn't find a file called "Systemced" anywhere...

There really isn't a file called SYSTEMced. The error message has just overwritten the message that normally appears there during system boot: "For troubleshooting and advanced startup options for Windows 2000, press F8."

Here is the message that is normally displayed at this point in the boot process: (notice that the ced from SYSTEMced is actually the last part of the word "advanced")

The method of recovery for this issue is actually documented fairly well here: KB 269075 here: KB 323148 and here: KB 302829 There are several other articles that describe various methods of correcting the problem, but these cover the basic steps required.

Usually when I see the issue it is because the system hive is too large to load into memory. In Windows 2000 (and NT 4) we are limited to 16 MB of memory at boot time. You will likely first run into this problem when the system hive reaches just a little over 10 MB. Thankfully this memory limitation has been greatly increased in Server 2003.

Essentially what you do is boot using an alternate system hive, and then either restore the hive from backup, (in the case of a corrupted hive) or clean up space in the system hive if the boot failure is caused by the system hive being too large.

This is a very common problem. I've seen it three times in the past week. (and countless times in the last few years) Some customers have this problem so often that they have a process in place to check the size of the system hive before they reboot a server. (you know who you are ;-) ) Hopefully with this and the next post, I can convince some of you to correct the problem that causes the bloated hive in the first place so that you never have to see this error on reboot.

In the next post I will go over the chkreg.exe utility that I use to correct this problem, and ways to prevent it from happening in the future.

Stay tuned...

Justin