Active Directory, Cluster and other fun stuff... : Cluster

Windows Server 2K8 Reviewers Guide

justintu — Tue, 25 Mar 2008 06:32:00 GMT

Hi Everybody!

This one comes at the request of several customers. Many of you out there are trying to determine which version of Server 2008 you will deploy. For most it comes down to deciding between two of the five major versions: Server 2008 Standard edition and Server 2008 Enterprise edition. Given the amount of features included in the OS and all of the different versions we shipped, trying to determine what version includes what feature can be confusing.

Fortunately we have released a very detailed treatise on this very subject in the form of a 247 page document appropriately titled, Windows Server 2008 Reviewers Guide.

Inside you will find a pretty thorough support matrix and technical nuggets like:

In Standard Edition you are limited to one standalone DFS Namespace. (DFS Root) This limit does not apply to domain-based DFS implementations.
Cross-File Replication for DFS-R is not available in the Standard or Web editions.
Server Core is available in all editions except for Itanium.
Hyper-V is included in Enterprise, Datacenter and Standard editions as long as you don't buy the version that say "without Hyper-V"
TS Licensing in Windows Server 2008 now allows you to track per-user CALs
Still no support for Cluster (failover) in Standard Edition, but you can now have 16 nodes with the Enterprise and Datacenter editions (8 with Itanium)

Here are some screen snags taken right from the guide:

Enjoy!

Missing or corrupt Systemced - part 2

justintu — Thu, 21 Dec 2006 14:46:00 GMT

****EDIT Hey Guys, I goofed on this post: This post discusses a utility used during the course of a Microsoft support call. It is not available to send to customers, and is not available for download as I had originally thought.

The version posted on the download site does not contain the same functionality referenced here. If you email me through the blog I will do my best to help out. Due to my tremendous workload my response may be delayed. If this is an urgent matter then you may want to consider opening up a paid incident with Microsoft Support: http://support.microsoft.com/ ****

This is part 2 of my earlier post on the whole "missing or corrupt system hive" issue. Okay, so we have a copy of the bloated/corrupt registry hive. Now what do we do with it? Chkreg.exe is your friend. Chkreg is a command line utility that you can use to repair a corrupt registry hive. You can also use it to just display registry key size. The majority of the issues that I see are not due to a corrupt system hive, so I use chkreg to help me identify what is taking up all of the hive size.

The ability to view registry key size wasn't added until a later version of chkreg than what is available at Microsoft.com.

The main version that you will find is actually used along with the XP Setup disks. In that version it is placed on disk 6, and after you boot to the recovery console it automatically attempts to repair the system hive. This version does not let you run it from the GUI. You will get this message if you try: "chkreg.exe application cannot be run in Win32 mode."

I thought the newer version was available on our site, unfortunately it looks like you have to call us in order to get this special version of chkreg. With this version of chkreg you get the /S, /O, and /D options.

/S Displays space usage for the bin. When bin is not specified, displays usage for the entire hive.

/O Ordered by size

/D Dump subkeys

I typically put the bloated hive in a folder such as c:\temp, and so my command would be:

chkreg.exe /F c:\temp\system /S /O /D >regbloat.txt

This will output the keys listed largest to smallest to a file called regbloat.txt

Here is an example of two such bloated keys from the txt file:

Size Subkeys

552027 ControlSet002\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

547031 ControlSet001\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\##?#Root#RDPDR#0000#{28d78fad-5a12-11d1-ae5b-0000f803a8c2}

In this example, the same key in both ControlSet keys are causing the registry size problem. This is a known issue that occurs when you have the Spooler service disabled on a Terminal Server.

I remove the bloated keys, and then run chkreg again, but this time with the /C switch to compress the hive. The last step is to swap the hive back out via recovery console in order to boot off of it.

There is a utility that you can use to correct the problem called scrubber.exe, but it only corrects the issue if it is due to the issue mentioned here: KB 277222

Tune in next time when I will discuss: Active Directory Forest recovery or something else equally exciting. :)

Thanks for viewing!

Justin

Technorati tags: Cluster, Setup, Windows 2000

Cluster service failure update...

justintu — Thu, 21 Dec 2006 09:52:00 GMT

Just a quick note to say that they did update KB 269229 with my comment about requiring the SERVICE account to be included in the "Impersonate client after authentication" user right. (reference this post for background info)

From the article:

"Note If you create a Group Policy setting to update the Impersonate a client after authentication rights policy setting, make sure that the Cluster service account is listed in the policy setting in addition to the Local Administrators group and the account that is called SERVICE."

It is still easy to overlook this in the article, so I don't anticipate and end to these issues. If any of you find this requirement missing from other MSFT documentation then please comment the article, or post a comment here and I will get it corrected.

Thanks,

Justin

Technorati tags: Cluster, Active Directory

Missing or corrupt Systemced... What's that?

justintu — Sat, 16 Dec 2006 16:33:00 GMT

Technorati tags: Cluster, Setup, Boot, Windows 2000

Part 1 of 2

The on-call pager went off at two in the morning. John rushed in to discover that one of their main Windows 2000 file and print servers was sitting at a black screen with the following error displayed:

Windows 2000 could not start because the following file is missing or corrupt:

\WINNT\SYSTEM32\CONFIG\SYSTEMced

Well was it missing or was it corrupt!? He booted the server into Recovery Console, and went to the location mentioned in the error. It appeared to be missing---he couldn't find a file called "Systemced" anywhere...

There really isn't a file called SYSTEMced. The error message has just overwritten the message that normally appears there during system boot: "For troubleshooting and advanced startup options for Windows 2000, press F8."

Here is the message that is normally displayed at this point in the boot process: (notice that the ced from SYSTEMced is actually the last part of the word "advanced")

The method of recovery for this issue is actually documented fairly well here: KB 269075 here: KB 323148 and here: KB 302829 There are several other articles that describe various methods of correcting the problem, but these cover the basic steps required.

Usually when I see the issue it is because the system hive is too large to load into memory. In Windows 2000 (and NT 4) we are limited to 16 MB of memory at boot time. You will likely first run into this problem when the system hive reaches just a little over 10 MB. Thankfully this memory limitation has been greatly increased in Server 2003.

Essentially what you do is boot using an alternate system hive, and then either restore the hive from backup, (in the case of a corrupted hive) or clean up space in the system hive if the boot failure is caused by the system hive being too large.

This is a very common problem. I've seen it three times in the past week. (and countless times in the last few years) Some customers have this problem so often that they have a process in place to check the size of the system hive before they reboot a server. (you know who you are ;-) ) Hopefully with this and the next post, I can convince some of you to correct the problem that causes the bloated hive in the first place so that you never have to see this error on reboot.

In the next post I will go over the chkreg.exe utility that I use to correct this problem, and ways to prevent it from happening in the future.

Stay tuned...

Justin

Cluster service failure after AD lockdown...

justintu — Thu, 14 Dec 2006 14:22:00 GMT

Users were unable to connect to their shares. John discovered that the Cluster service wasn't started, and that any attempts to start it resulted in an error 1068. He attempted to ping the virtual server's IP address and it returned a "request timed out" message. He got the same error when trying to ping the cluster node's public adapter.

When he got to the node he found the Cluster service in a Starting state. He soon discovered that he had no network connectivity to or from either Cluster node, and that their network cards were missing from "Network Connections" The only changes made to the network were just a few minor group policy settings to lock down permissions a bit. Maybe that had something to do with this? It looked like it was going to be a long night...

This is another fairly common problem. This is not really just a Cluster problem, but that is usually how it is presented to me. Of course if networking is not functional, then Cluster isn't going to work either. :) I have worked at least three of these issues in the last two months, and thought it warranted discussion since there isn't a public KB article on this particular scenario yet. I hope to fully document every error encountered here, so that others may find this post when they run into this situation. (KB articles sometimes take a while to get published)

System event log:

SAM event ID: 12291 "SAM failed to start the TCP/IP or SPX/IPX listening thread"
IPSec event ID: 4292 "The IPSec driver has entered Block mode."
DfsSvc event ID: 14523 "DFS could not contact any DC for Domain DFS operations."

Application event log:

EventSystem event ID: 4609 "The COM+ Event System detected a bad return code during its internal processing. HRESULT was 80004015 from line 142 of d:\nt\com\complus\src\events\tier2\service.cpp."

Other problems discovered with this node:

The Com+ Event System, Network Connections and Shell Hardware Detection services were in a Starting state.

The following services failed to start:

Cluster Service: Error 1068: The dependency service or group failed to start.
File Replication: Error 1068: The dependency service or group failed to start.
---dependencies opens up a window titled "Service Dependencies" and the message is: Wind32: Access is denied.
IPSEC Services: Error 1899: The endpoint mapper database entry could not be created.
System Event Notification: Error 1068: The dependency service or group failed to start.
--trying to view the dependencies on the server returns the following message: Win32: Access is denied
Task Scheduler: "The endpoint mapper database could not be loaded"

We have three services failing with "the dependency service or group failed to start."
When we try to view the dependencies we get an access denied message.

Let's look in the registry to see what each of these services depend on:

Cluster service:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ClusSvc
DependOnService:

ClusNet
RpcSs
W32Time
NetMan

File Replication:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs
DependOnService:

EventLog
RpcSs
EventSystem

System Event Notification:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SENS
DependOnService:

EventSystem

So the common dependencies are RpcSs and EventSystem

RpcSs is the Remote Procedure Call (RPC) service, and EventSystem is the Com+ Event System service. We know from earlier that Com+ Event System is one of the services stuck in a Starting state, so that is why the File Replication and System Event Notification services haven't started. One of the other dependencies for the Cluster service is NetMan, which is the Network Connections service. Network Connections is also one of the services stuck in a Starting state.

So now the real question is: Why are the Com+ Event System and Network Connections services not starting?

If we view the dependencies for these two services, we just find RpcSs listed. So it all boils down to RPC. However, the Remote Procedure Call (RPC) service is actually started.

If you do a search in the knowledge base on these errors, you are likely to come across this article:

909444 Systems that have changed the default Access Control List permissions on the %windir%\registration directory may experience various problems after you install the Microsoft Security Bulletin MS05-051 for COM+ and MS DTC

This discusses changes made by a hotfix that would cause these problems. The fix is to correct NTFS permissions on the %SystemRoot%\Registration directory. However the permissions here are the same as in the article.

You may also come across this one:

916254 COM+-related events may be logged in Event Viewer when you install Windows XP Service Pack 2 and join the computer to a domain

Most would come across this second article and instantly dismiss it since it says "Windows XP Service Pack 2." However, we have a lot of the same symptoms, and since XP SP2 and Server 2003 SP1 include a lot of the same security changes it warrants further investigation.
One of the security changes in SP1 for Windows Server 2003 was to change the Logon Account used for RPC.
RPC use to log on as Local System and now uses an account with less privileges: Network Service.

The article states that this issue occurs if the SERVICE account is missing from the policy setting "Impersonate a client after authentication"

We can see if SERVICE is missing from this policy by performing the following steps:

1. Open up Local Security Policy in order to see what the effective settings are:

Start, Run, secpol.msc

2. Expand Local Policies, User Rights Assignment and then open up "Impersonate a client after authentication"

At minimum the following should be listed: Administrators and SERVICE

The problem that I have seen recently happens when someone decides to change the "Impersonate a client after authentication" user right in group policy. Typically how it goes is they decide to lockdown their servers, and only give specific accounts certain privileges. However, after incorrectly removing the SERVICE account from this privilege the server loses all network connectivity. Fortunately this problem doesn't show up until after a reboot. (You have an opportunity to identify that the problem exists before causing a major outage of all servers in a large OU.)

The fix is simple for the servers that haven't been restarted:

1. Correct the policy and then force group policy to be reapplied. (gpupdate /force)

(To correct the policy: just add SERVICE and Administrators to this policy setting in addition to the other ones defined)

If you have already rebooted the servers after applying the incorrect policy settings they will not be corrected by just simply changing the policy back since they have already lost network access. (unless the policy change was made locally to begin with)

1. Export the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RpcSs

2. In the services snap-in: Change Remote Procedure Call (RPC) to start up with the Local System account instead of Network Service, and then reboot

3. At this point the majority of the services should be started and we should now have network access. Ensure that the offending group policy has been corrected with the proper accounts, force group policy to apply, (gpupdate /force) and then reboot.

4. Change the logon account for Remote Procedure Call (RPC) service back to Network Service by importing the reg file that you exported in step one, and then reboot. Alternatively: navigate to the following reg key and then reboot

Technorati tags: Active Directory, Cluster, Windows Server 2003

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RpcSs

Change the ObjectName value from LocalSystem to: NT Authority\NetworkService

For more information regarding this security setting see article on Technet: SeImpersonatePrivilege
I have commented KB 269229 to reflect the requirement for SERVICE to be included in this User Right.

Please let me know if you like the format of this post or if you have any questions.

Until next time.

Thanks,

Justin Turner
This posting is provided "AS IS" with no warranties, and confers no rights.