Public Folder Replication Troubleshooting - Part 4: Exchange Server 2007/2010 tips
Original post on US blog: January 11, 2008
Two years ago, I posted a three-part series on troubleshooting public folder replication. Part 1 discussed the replication of new data, Part 2 discussed the replication of existing data, and Part 3 discussed the replica deletion process and some common problems we saw with Exchange 2003. With this post, I want to update the series for Exchange 2007.
In Exchange 2007, public folder replication works basically the same way it always has. The troubleshooting steps in the first three parts of the series all still apply. However, the admin tools have changed and the common problems we see with Exchange 2007 are a little different, so that's what I want to cover here.
Changes in the Admin Tools
The event log is still your best tool for narrowing down a replication problem to a particular point of failure. In Part 1, I suggested turning up logging on Replication Incoming and Replication Outgoing to Maximum. That still applies, except that with Exchange 2007 you'll be using the Set-EventLogLevel cmdlet to set "MSExchangeIS\9001 Public\Replication Incoming Messages" and "MSExchangeIS\9001 Public\Replication Outgoing Messages" to the Expert level.
In Part 2, I described how to use the Synchronize Hierarchy and Synchronize Content options in ESM to force a status message and to timeout all outstanding backfill entries. You can still do this in Exchange 2007 via the Update-PublicFolderHierarchy and Update-PublicFolder cmdlets. These are also available in Sp1's public folder management tool, appearing as Update Hierarchy when the Public Folders root is selected and Update Content when a particular public folder is selected. Because you can use these from the command line, they are a lot more flexible than the Exchange 2003 options. For instance, it's now very simple to time out backfill entries for every folder that has a replica on your Exchange 2007 server with a simple "Get-PublicFolderStatistics | Update-PublicFolder" command. That wasn't possible in Exchange 2003 without a lot of clicking.
In Part 3, I described how to use the Public Folder Instances view to see if the deletion of a replica has completed. In Exchange 2007, you use the Get-PublicFolderStatistics command to see that same information.
Common Problems in Exchange 2007
The most common symptom I've seen so far is a store driver failure. For instance, a backfill response will be sent to an Exchange 2007 server, but if you look at the application log on the 2007 side you never see the incoming replication event. Message tracking shows that the replication message got to the hub transport server and then failed in the store driver.
This can happen for a number of reasons, and fortunately it usually is not that hard to troubleshoot. Your best troubleshooting approach in this case is to use the Pipeline Tracing and Content Conversion Tracing available on the hub transport server. If you run "Get-TransportServer | fl" you'll see a few settings related to this:
PipelineTracingEnabled : False
ContentConversionTracingEnabled : False
PipelineTracingPath : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing
PipelineTracingSenderAddress : SERVER01-IS@contoso.com
To find out why your backfill response is failing in the store driver, set the PipelineTracingSenderAddress to match the SMTP address of the public folder store that's sending the backfill response. Then set ContentConversionTracingEnabled to $true and PipelineTracingEnabled to $true, and reproduce the problem. After doing so, have a look in the folder specified by the PipelineTracingPath. You should find a subfolder called ContentConversionTracing, and an InboundFailures folder within that. In the InboundFailures folder, you'll have an EML file containing the replication message itself and a TXT file containing some information on the failure. The top of the TXT file will often give you a useful clue as to the reason for the failure.
For instance, in a few cases we've seen this output in the TXT file:
Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00020329-0000-0000-c000-000000000046}:'Keywords'] Categories
Error = Element 0 in the multivalue property is invalid..
In this case it's complaining about the Categories property. This will happen if the public folder in question contains items on which the Categories property is set to be blank - such as with a single space - instead of truly being empty. You can see this by going into Outlook and choosing to view items by Category. You should find that there are two different sets of items with "None". To correct the problem, simply clear the Category on all of the "None" items using Outlook. You may have to set them to some other category and then set them back to None. Once the Categories property is truly clear, you'll only have one set of items that show "None", and the items will replicate successfully.
Another example:
Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00062004-0000-0000-c000-000000000046}:0x8092] Email2AddrType
Error = Email2AddrType is too long: maximum length is 9, actual length is 35..
In this case its flagging the Email2AddrType property as a problem. We discovered that it had somehow been populated with the full email address on certain contacts, instead of containing only the address type that it normally should, such as 'SMTP' or 'EX'. Fixing that property allowed the items to replicate.
Sometimes the store driver will fail in a manner where it does not identify a specific problem property, such as in this output:
Microsoft.Exchange.Data.Storage.ConversionFailedException: Message content has become corrupted. ---> Microsoft.Exchange.Data.Storage.ConversionFailedException: Content conversion failed due to corrupt TNEF (violation status: 0x00000800)
This is what the failure looks like when you hit the problem described in KB 936000. Applying the fix to the Exchange 2003 server that's generating the replication message will correct that problem.
The important thing to take away from this is that Exchange 2007 does a lot of property validation to keep bad data from getting into the store. Although that's a good thing, it can keep public folder data from replicating from your Exchange 2003 servers until problems with the content are corrected on the 2003 server. ContentConversionTracing can help you identify these problems, and will often point you to the exact property causing the problem.
There's one more common problem you can identify with ContentConversionTracing, but it isn't caused by any actual problem with the content. The TXT file in the InboundFailures folder will look like this:
Microsoft.Exchange.Data.Storage.ConversionFailedException: The content conversion limit has been exceeded.
at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItem(MimePromotionFlags promotionFlags)
at Microsoft.Exchange.Data.Storage.ItemConversion.InternalConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags promotionFlags, Boolean isStreamToStream)
at Microsoft.Exchange.Data.Storage.ItemConversion.ConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options)
InboundConversionOptions:
- preferredCharset: iso-8859-1
- trustAsciiCharsets: True
- isSenderTrusted: False
- imceaResolveableDomain: contoso.com
- preserveReportBody: False
- clearCategories: True
- userADSession:
- recipientCache: Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache
- clientSubmittedSecurely: False
- serverSubmittedSecurely: False
ConversionLimits:
- maxMimeTextHeaderLength: 2000
- maxMimeSubjectLength: 255
- maxSize: 2147483647
- maxMimeRecipients: 12288
- maxRecipientPropertyLength: 1000
- maxBodyPartsTotal: 250
- maxEmbeddedMessageDepth: 30
- exemptPFReplicationMessages: True
First, notice the top line says, "The content conversion limit has been exceeded." Normally a public folder replication message is exempt from size limits and such, so why would this message be failing in this way? Notice that "isSenderTrusted" is False. This means the message came over an SMTP connection that was not authenticated. The sending server either failed to authenticate, or never even tried. This is very similar to what I described in the Common Problems section of Part 3, where an authentication failure would cause the XEXCH50 verb to fail in Exchange 2003. Because the sending server did not authenticate, the Exchange 2007 server applies the normal size limits to the message. If this is a content replication message with more than 250 attachments (not uncommon for a content backfill response), then it fails because it exceeds the limit. This often happens because an administrator has created a second Receive Connector which does not allow authentication, but has it configured such that it listens on the internal IP rather than the external IP. If that's not the cause, you may need to use a Netmon capture to identify the problem as I described in Part 3.
Conclusion
That should cover everything you need to know to narrow down public folder replication problems in an environment with Exchange 2007. All the old troubleshooting steps still apply; we just have different administrative tools and a different set of problems. Hope this helps!