Evan Dodds - Microsoft Exchange Server Blog : Exchange Clustering

KB.921181 and "File Share Witness"

evand — Thu, 06 Jul 2006 17:35:00 GMT

KB out yesterday (KB.921181) covers the introduction and details about the new “File Share Witness” quorum model. This is an improvement to the original MNS design that allows a read/write file share to serve as the “voter” for quorum status, eliminating the need for a third (or, at least, “odd-number”) node as voter. This new feature helps pave the way to more simplified and effective MNS clusters for use by the Cluster Continuous Replication feature of Exchange 2007 (pre-release documentation, but this section looks pretty complete).

This feature is released as a public hotfix (linked from the KB) to either Windows 2003 SP1 or Windows 2003 R2.

KB.867624 replaced

evand — Mon, 03 Apr 2006 04:58:00 GMT

Looking for KB.867624? It’s been replaced and retired (and removed)...

I had previously blogged about the “Upgrading to E2k3 SP2 on a cluster” process, and in that posting I had referenced the KB.867624 article about how to upgrade to E2k3 SP1 (pointing out that the process is essentially unchanged for SP2).

But, someone brought to my attention the other day that this KB article has recently been removed. In fact, I dug around a bit and discovered that the various E2k3 Service-pack-on-a-cluster details have been rolled into the Clustered Server section of KB.328839, rendering the KB.867624 redundant.

Yes, it would be nice if the old link had a redirect or some sort of notice of the removal. But, well, evidently that’s now how these things work. Send your comments through this link.

Virtual Server Host Clustering Step-by-Step

evand — Fri, 11 Nov 2005 01:37:00 GMT

A Step-by-step guide for using MSCS clusters to host Virtual Server guests was just released: http://www.microsoft.com/downloads/details.aspx?FamilyID=09cc042b-154f-4eba-a548-89282d6eb1b3&displaylang=en

Here’s the overview:

Overview

This document provides an introduction to the methods and concepts of Virtual Server host clustering. With Virtual Server host clustering, you can provide a wide variety of services through a small number of physical servers and, at the same time, maintain availability of the services you provide. If one server requires scheduled or unscheduled downtime, another server is ready to quickly begin supporting services. Users experience minimal disruptions in service.
Virtual Server host clustering is a way of combining Microsoft® Virtual Server 2005 R2 with the server cluster feature in Microsoft Windows Server™ 2003. This document describes a simple configuration in which you use Microsoft Virtual Server 2005 R2 to configure one guest operating system, and configure a server cluster that has two servers (nodes), either of which can support the guest if the other server is down. You can create this configuration and then, by carefully following the pattern of the configuration, develop a host cluster with additional guests or additional nodes.

Upgrading to E2k3 SP2 cluster

evand — Sun, 30 Oct 2005 18:51:00 GMT

There are a couple of important things I wanted to point out about the upgrade to E2k3 Sp2 on a cluster:

Same process as the upgrade to E2k3 Sp1
Fixes broken virtual directories and other upgrade snafus
IMF still not supported for use on cluster

Ok, more detail on these three, as I’m sure the one-sentence-explanation is not helpful…

Same process as the upgrade to E2k3 Sp1 – Here I am referring to KB.867624. This KB was written for the upgrade from E2k3 RTM -> E2k3 SP1, and it applies just the same for the upgrade from E2k3 RTM -> E2k3 SP2 or the upgrade from E2k3 SP1 -> E2k3 SP2. You’ll need to do the “Upgrade Exchange Virtual Server” bit again, ensuring you’re executing it from the upgraded node and with the EVS partially-online also on the upgraded node. It’s a bit complicated, but the steps should work just fine if you follow them carefully.

Fixes broken virtual directories and other upgrade snafus – Here I’m talking about the Outlook Mobile Access (OMA) and Recovery Storage Group (RSG) problems that some of you may have run into (and perhaps not even realized). These “new to E2k3” functions will be unavailable in E2k3 Sp1 if the upgrade wasn’t done correctly to E2k3 Sp1. Sp2 upgrade closes all the loopholes and ensures that this part of the upgrade is finally done and the functionality becomes available.

Here’s some more detail. The correct way to do the upgrade from E2k SP3 -> E2k3 SP1 is this high-level set of steps:

Upgrade the node(s) to E2k3 RTM
Do the “Upgrade Exchange Virtual Server” step at E2k3 RTM
Upgrade the node(s) to E2k3 SP1
Do the “Upgrade Exchange Virtual Server” step at E2k3 SP1

In that case, the server goes E2k SP3–>E2k3 RTM->E2k3 SP1

However, if you skip step #2 (ie - apply the E2k3 RTM binaries, then the E2k3 SP1 binaries, and only THEN do the Upgrade Exchange Virtual Server step), some of this “new to E2k3 RTM” functionality will be inaccessible.

Whew, that was a mouthful explanation. Anyways, the end summary is that E2k3 Sp2 “Upgrade Exchange Virtual Server” step goes back and ensures that in all cases the E2k3 RTM “Upgrade Exchange Virtual Server” actions were done, fixing anyone who did the abbreviated upgrade steps (skipping step #2).

IMF still not supported for use on cluster – Yes, it’s included as an integral part of E2k3 Sp2. But the “transport” portion of IMF (the thing you used to have to manually install on the SMTP gateway servers) is still not designed for clustered back-end servers. Store action behavior (moving stuff to the Junk E-mail folder automatically based on SCL) is still supported there, of course, as it’s a back-end server function. But the “stamping an SCL onto the message” part is not. This still makes sense though — do you really want to directly expose your “high availability email cluster” to the Internet? Any high-availability installation of Exchange should definitely have separated Internet gateway functionality, where the IMF will work just fine.

"Access Denied" creating an Exchange 2003 System Attendant resource

evand — Sun, 30 Oct 2005 18:22:00 GMT

KB.906906 talks about another problem you may run into while creating a system attendant resource for Exchange 2003 cluster, relating to Remote Registry access/permissions.

This is really just pulling out the “Symptom #4” part of KB.329229 as its own KB article, but it’s still worth pointing out in case you run into it!

Exploit for MSDTC bug in MS05-051

evand — Thu, 13 Oct 2005 17:55:00 GMT

Techweb reports that there’s an exploit out for the MSDTC security fix in MS05–051.

So, if you’re running an Exchange cluster on Windows 2000 server, you’ll want to get this patch applied ASAP! See the security bulletin for more details, and for info on the status of other Windows versions.

Exchange Standby Clusters are now supported

evand — Sat, 08 Oct 2005 01:48:00 GMT

Great news! The long-awaited ability to use “standby” Exchange clusters is finally completed testing and documentation with its addition to the Disaster Recovery Operations Guide: http://www.microsoft.com/technet/prodtechnol/exchange/guides/DROpsGuide/2493c2d6-618c-4c49-9cb1-fff556926707.mspx

This is one of those topics that has come up again and again over the years, and there was never any good (supported) solution for it. Well, over the past few months the Exchange team went through all the requisite testing and documentation, and now this is a fully-supported DR solution!

What does this mean? It means you can now “recover” your clustered Exchange 2003 EVS to a totally different cluster server (potentially in a different physical or AD site, with different IP, etc). See the updated DR Ops guide for more details.

Clustered Exchange setup tidbit

evand — Wed, 07 Sep 2005 17:10:00 GMT

This question came up yesterday on a list I read, so I figured it might be of interest out here.

Do you remember the “Setup Shall Install the Clustered Version of Exchange” pop-up in Exchange 2000? This was the pop-up you’d get when installing the binaries for Exchange 2000 onto a server that was joined to an MSCS cluster, and if you ever saw it, this wording probably stuck with you because it sounded a little strange.

Well, have you seen this pop-up in Exchange 2003? Nope. You haven’t. Although, you might still have seen a pop-up. Or you might not have!

In Exchange 2000, you ALWAYS saw this pop-up when you were doing the install into a cluster.

In Exchange 2003, this is changed. Now there are three possibilities:

Not running on cluster – logs “This is not a clustered machine” to the Exchange Server Setup Progress.Log file. No pop-up.
Running on 1 or 2 node cluster – logs “Microsoft Exchange Setup has detected that you are running in a cluster environment. See http://www.microsoft.com/exchange for updated information on cluster configuration recommendations.” into the Exchange Server Setup Progress.Log file. No pop-up.
Running on >2 node cluster (ie, such as when adding a 3rd Exchange node) – logs to file just like #2 above, but also pops up the message with an OK button for acknowledgement.

Evicted your Exchange cluster node?

evand — Mon, 05 Sep 2005 21:19:00 GMT

Jim McBee recounts a recent experience with this, and how forgetting to do a “setup /reinstall” on the cluster node led to Exchange thinking it was a beta/evaluation version of Exchange.

Excellent point to make: if you ever evict one of the (Exchange binary) nodes from the Exchange cluster, make sure you either reload the whole OS from scratch — including Exchange binaries — or, simply be sure to do “Setup /Reinstall” against the node after it’s been joined back to the cluster. When you evict the node with Exchange still on it, Exchange gets broken. The way to fix it is to force Exchange to run back through its setup.

Large Exchange 2003 clusters and disabling the MTA

evand — Mon, 05 Sep 2005 20:35:00 GMT

Big revision recently to KB.810489. This is the KB that talks about the MTA function and how it can or can’t be disabled in various scenarios.

The core change is that if you’ve got super-large Exchange clusters (>4 nodes) in a native-mode administrative group (and do don’t have MTA-dependent foreign connectors, etc), you may want to disable the MTA service. You can do this on non-clusters too, of course, but it’s really more interesting in the large cluster case where all of the EVS in the cluster utilize a single MTA and may lead to event log errors and performance bottle-necks. Disabling the MTA can’t be done in Exchange 2000, as move-mailbox is still dependent on the MTA… but in Exchange 2003 it’s now possible and supported to disable/remove the MTA! Have a look at the KB for the details.

This is a good companion KB article to the KB.899302 article I mentioned earlier.

Update Sep 8, 2005: Mark has blogged with a bunch more detail about "disabling the MTA" scenarios. His post covers more than just the cluster case, if you're interested.

The MTA and Exchange 2003 clusters with > 4 nodes

evand — Mon, 15 Aug 2005 17:10:00 GMT

This is a bit of a specific topic, as it only affects people who are trying to use the MTA in their cluster (for mail delivery back to 5.5 in mixed-mode, or to foreign systems not connected with SMTP, etc) *AND* have >4 nodes in their cluster.

There’s a new KB out that will help you with some performance tuning in this scenario: http://support.microsoft.com/kb/899302. This is a KB that is near and dear to my heart, as I was the original case owner on a group of these issues before I moved over to the product group.

In any case, what happens is that if you have more than 50 MDB databases sharing a single MTA, some scalability settings in the MTA are not set correctly by default and need to be increased. How do you get 50 MDB databases sharing a single MTA? Well, in an Exchange 2000/2003 cluster, there is but a single MTA for the entire cluster. And in Exchange 2003, you can create up to 8 nodes (up to 7 Exchange Virtual Servers) in that same, single-MTA cluster. Each EVS can then host up to 20 MDBs. So, in theory, you could load up an Exchange 2003 cluster with 140 MDBs.

Don’t do this. It’s a bit like RAID5 disks. If you have 8 disks and you set aside a single disk as parity for RAID5, and then you proceed to fill up to 100% capacity the usable disks… you’re sunk if you suffer any physical disk loss in a 7+1 configuration. You don’t want to spread yourself that thin!

Anyways, back to the topic at hand: you may find yourself getting event 3121 logged in your application log. This is telling you that you’ve hit the maximum # of associations for the MTA (ie – too many stores are trying to talk to the MTA at once). The KB article walks you through the one — and only one — registry change to make in this case. Don’t make other changes to the MTA settings in the registry, or set the reg key to some higher value than recommended by the KB.

With that registry key in place, you’ll be able to exceed the 50 MDB limit and increase the scalability for these very-large Exchange 2003 clusters.

Exchange clustering and firewalls

evand — Thu, 11 Aug 2005 17:56:00 GMT

Rod posts today about the strange (and sometimes frustrating) behavior where outbound SMTP connections from an Exchange cluster originate from the host IP address rather than from the virtualized IP resource associated with the Exchange Virtual Server. Great post and this is a common question on the newsgroups, etc.

Scott Landry also talked about the technical reasons behind this SMTP/network behavior in some detail (it has to do with networking APIs and OS routing tables) as “Myth 4” in his SMTP Virtual Server Myths post.

Cluster hotfix for Exchange 2000

evand — Sat, 06 Aug 2005 03:05:00 GMT

KB article came out the other day on this sort of obscure scenario, but interesting to know about in any case: KB.896617: “The Exchange Information Store service stops responding when the Exchange Information Store service is shutting down or the Exchange Cluster server is performing a failover process”.

Couple of things to note:

This only affects Exchange 2000 (it’s an Exchange 2000 post-SP3 hotfix)
… which means it does NOT happen on Exchange 2003.
It doesn’t STRICTLY affect clusters, but it’s probably most common there due to the way failover works

As with any hotfix, determine whether you need to apply it before doing so and don’t bother to apply it if you don’t need it. You’ll probably know if you’re hitting this problem on your server.

Clustered Exchange Nodes as AD domain controllers

evand — Mon, 25 Jul 2005 22:09:00 GMT

I blogged about KB.898364 briefly when it was created, but over this past weekend there was a flurry of newsgroup posts about putting clustered Exchange onto cluster nodes that are also domain controllers. Several responses referenced KB.281662 (how to cluster AD domain controllers), which didn’t very clearly specify not to do this with Exchange.

That’s been rectified. The More Information section of KB.281662 now clearly states that Exchange is not supported on cluster nodes which are configured as domain controllers:

Note Exchange 2000 and Exchange Server 2003 are not supported in a clustered configuration where the cluster nodes are domain controllers.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

898634 (http://support.microsoft.com/kb/898634/) Active Directory domain controllers are not supported as Exchange Server cluster nodes

Hope it helps!

MSDTC and Exchange clusters - redux

evand — Mon, 04 Jul 2005 19:51:00 GMT

Still catching up on my queue of blog posts, I think that all of the documentation around MSDTC and Exchange clusters has now been updated (see earlier posts for details).

Here’s what I’ve got:

KB.301600 has been updated to detail Exchange 2003 not requiring a dedicated group for MSDTC and that network DTC is not required for Exchange
KB.312316 has been updated to detail Exchange 2000 not requiring a dedicated group for MSDTC.
The Exchange 2003 Deployment guide “Deploying Exchange Server 2003 in a Cluster” chapter has been updated for MSDTC and a handful of other miscellaneous touch-ups.

Please let me know if you know of any other places it is still detailing the old recommendations.