EDIT 1/21/2010: Aded the note mentioning that the prerequisite check rule for this condition has been added to Exchange 2007 SP2.
We've recently encountered a small handful of customers who reported mailflow problems shortly after running Exchange 2007's /prepareAD, or after they installed their first Exchange 2007 role (which auto-launches the pre-requisite prepareAD process). Through determining root cause, we found that this problem might affect a larger set of our customers that have (or have ever attempted) e-mail domain name sharing and use the "forward all mail with unresolved recipients to host" option.
Some of the Exchange 2000/2003 mailflow problem symptoms include:
Why does this happen?
The e-mail domain name that users primarily use was ambiguously nonauthoritative. /PrepareAD does not expect this condition when enumerating recipient policies, and attempts to "fix" the Exchange 2000/2003 mis-configuration by making e-mail domain(s) consistently non-authoritative on recipient policies. A few other things occur behind the scenes, but eventually mail will queue-up. If you only have a handful of recipient policies, here's how to determine whether or not you are at risk of a mailflow outage:
First, navigate in Exchange 2000/2003's system manager, and pull-up the recipient policies. Here is a screenshot of the recipient policies in the Contoso lab organization. Since we know that "contoso.com" is present on multiple recipient policies, we need to check each occurrence for the authoritative setting.
In the site01 policy, we can see that there is an occurrence of contoso.com. When you double-click (or edit) the contoso.com entry, the popup-dialog box on the right indicates that it is authoritative. I've also included an ldifde output to illustrate how the GUI maps to the raw data from Active Directory.
In the site08 policy, we can see that contoso.com doesn't have the checkbox, and thus is nonauthoritative. Since this setting differs from the instance of contoso.com within the site01 policy, prepareAD would detect this mis-configuration, and subsequently converts any authoritative occurrences of contoso.com to non-authoritative. The recipient policy change(s) would eventually cause all Exchange servers in the organization to remove contoso.com from their own metabases, and transport on such servers are in an "in-between" state until IIS is restarted.
Notice that the screenshots show recipient policies which are appropriately-named for this blog. However, in real-world environments, they are rarely named descriptively. Another challenge is that not all of the policy objects (such as pure mailbox-manager policies) will have visible e-mail settings, in which case you would need to temporarily enable the "e-mail addresses" property sheet to expose the addresses written to those policies when they were created (which could have been years ago). Lastly, an admin would be prone to error if he/she were to check hundreds of policies in a large environment. Therefore, Marc Nivens took our requirements and wrote a read-only script to assist you in detecting ambiguously authoritative e-mail domain names. Seeing that you might not have any Exchange 2007 servers in the org yet, all you need is a machine that has Powershell 1.0 installed, and you can copy the script to run it. (Do note that you will need to change your execution policy. If you don't feel comfortable changing your execution policy, you can always author your own script by creating a .ps1 file with notepad and paste-in the contents of our code.) After invoking the script in this lab environment, the contoso.com domain (among others) was found to be ambiguously authoritative:
What do you do?
If you have not yet executed prepareAD: As you've seen above, the corrective action is to check to see if there's an e-mail domain name that exists on multiple recipient policies. If so, double-click on each occurrence of that domain to make sure the checkbox for "This organization is responsible..." is either consistently checked or unchecked across all occurrences of that domain. If you intended for all occurrences of the e-mail domain name to be nonauthoritative, you need to uncheck the checkbox from all occurrences of that domain. On the other hand, if you intended for all occurrences of the e-mail domain name to be authoritative, you need to check the checkbox from all occurrences of that domain. Any changes to the "This organization is responsible..." checkbox (i.e. msexchnonauthoritativedomains) do not affect address generation, and thus will not produce the pop-up dialog requesting to update all recipient e-mail addresses. You can use the attached PowerShell script for detection, but the corrective steps are manual. The PowerShell script may be used again to double-check your corrections after AD has replicated.
If you have already executed prepareAD and are experiencing a mailflow issue with one of your e-mail domain names: Restart IIS on all servers in the Exchange organization. This is because the metabase on all servers exists in an in-between state that doesn't fully recognize the route change. In some cases, restarting IIS will alleviate the mailflow outage, but it doesn't necessarily resolve the issue if you truly intended for the domain to be authoritative. Additionally, Exchange System Manager will not accurately reflect the nonauthoritative state of the e-mail domain names. This is due to /prepareAD having populated a lowercase value of "smtp" into msexchnonauthoritativedomains. If the gatewayproxy attribute has the uppercase "SMTP" then ESM doesn't see the case match, and will misleadingly render the domain as "authoritative" in the GUI. The chart below illustrates the before-and-after changes:
ESM GUI "This Exchange organization is responsible..."
Default policy (authoritative for contoso.com)
Site08 policy (nonauthoritative for contoso.com)
Site01 policy (authoritative for contoso.com)
Additionally, the metabase shows a domain key for contoso.com with routeaction=32.
During /PrepareAD execution, the recipient policies are changed at the same time the following entries are logged to the file ExchangeSetup.log:
[9/2/2008 12:24:32 PM]  [WARNING] The SMTP address template 'SMTP:@contoso.com' is invalid because it references a domain that is not an accepted domain.
[9/2/2008 12:24:32 PM]  [WARNING] Found 'contoso.com' in NonAuthoritativeDomains but did not expect it. Run Set-EmailAddressPolicy to correct this.
After SP1 /prepareAD succeeds, the changes (in red) are apparent:
At this point, if you dump the IIS metabase on any Exchange server in the organization, you will no longer see the local route; contoso.com actually gets removed from the metabases. And since gatewayproxy is capitalized whilst msexchnonauthoritativedomains is not, the ESM UI will mis-lead administrators to believe the e-mail domain names are authoritative.
Long-term solution if you have already run /PrepareAD: You will need to edit your recipient policies to re-write the authoritative/nonauthoritative status for each occurrence of that shared e-mail domain name, and restart IIS on all servers after AD replication has completed. If you wish to make the domain authoritative, you will need to uncheck the checkbox next to "This organization is responsible...", hit apply, and re-check it, and hit "apply" to correct any lowercasing issues in the raw data to force ESM to reflect an accurate checkbox. For reference, article 321721 mentions two methods for namespace sharing and recommends against configuring SMTP virtual servers having been configured with "forward all unresolved mail to this host".
What if I already ran /prepareAD and didn't notice a problem? Check your ExchangeSetup.log on the machine where you ran /prepareAD. If you skipped /prepareAD, you need to open the exchangesetup.log file on your first Exchange 2007 server role). Search for the existence of "in NonAuthoritativeDomains but did not expect it" (without quotes) and make a note of the referenced e-mail domain(s) preceding occurrences of that text string. As long as you're aware of the ESM GUI display issue described above, you can ignore this entire post if:
In conclusion, if an e-mail domain name is intended to be non-authoritative, then make sure it's non-authoritative on ALL recipient policies (even mailbox-manager-only policies). The opposite also holds true: If an e-mail domain name is intended to be authoritative, make sure it's consistently authoritative on all policies. So any e-mail domain name that is ambiguously non-authoritative (i.e. authoritative on one policy and non-authoritative on the other) will cause prepareAD to modify the authoritative setting (msexchnonauthoritativedomains) on recipient policies.
Hope this has been helpful.
NOTE: We have added a prerequisite check for this condition and shipped it in Exchange 2007 Service Pack 2. So if you are starting with Exchange 2007 SP2, you should not have a problem described in this blog post but will be alerted to address it beforehand.
You can download the script from the following link.
Note: this script is not officially supported by Microsoft. Please see the script for more details.
- Vincent Yim