Exchange Search Fails After Installing Exchange 2007 SP3

Symptoms

1. After installing Exchange Server 2007 Service Pack 3, in some cases, you may find that searches in Outlook online mode or OWA may not return any results for any new email that has arrived after installing Exchange 2007 SP3 for any users with mailboxes on the affected database(s).

2.  If you run ResetSearchIndex.ps1 on the indexing for the affected database(s), searches in online mode or OWA may not return any results for any users with mailboxes on the affected database(s).

3. If you run ResetSearchIndex.ps1 on the indexing for the affected database(s), if you check the CatalogData folder(s) that correspond to the affected database(s), you may find that either no .CI files are generated at all or only only a small number .CI files are generated and these files are only a few KB each.

Cause

The ultimate reason why this does not happen with Exchange 2007 Service Pack 2 or earlier, but does occur in some cases after installing Exchange 2007 Service Pack 3 is unknown.  However, the immediate cause for the content indexing files failure to generate is explained in detail below.

1. When the Microsoft .NET Framework 2.0 loads a managed assembly, the managed assembly calls the CryptoAPI function to verify the Authenticode signature on the assembly files to generate publisher evidence for the managed assembly.

2. The CryptoAPI function checks a Certificate Revocation List (CRL) that is available at https://crl.microsoft.com . This action requires an Internet connection.

3. If the Internet connection is blocked, the outgoing HTTP requests may be dropped. Therefore, an error message is not returned. This problem may also occur if DNS cannot resolve https://crl.microsoft.com . This long delay causes the CRL check to time out.

4.  The Micrsoft Search Indexer service (Microsoft.Exchange.Search.ExSearch.exe) is a managed assembly that is loaded by Microsoft .NET Framework 2.0. After installing Exchange 2007 SP3, if the Exchange 2007 SP3 server or DNS cannot resolve https://crl.microsoft.com for any reason, this affects the Microsoft Search Indexer service in such a way that any new email fails to be indexed. If DNS or the Exchange 2007 SP3 server cannot resolve https://crl.microsoft.com for a long enough time, the cached CRLs will expire.

5. MSSearch 3.0  and MSSearch 3.1 is hardcoded to call the “WINTRUST!WinVerifyTrust” to verify modules when they load into process space.

6. When the service starts up, VerifySignature is called when the configuration blob is created via SetProperty().

7. Also, when a search thread is created from the thread pool via WorkItemThread(), VerifySignature is called each time.

8. VerifySignature will then call WinVerifyTrust(). This is the API that actual performs the CRL check.

9. WINTRUST will ultimately use the WINHTTP API to make the internet connection.

Resolution

NOTE: The following is the official resolution from Exchange Development:

1. Add the following entry in the HOSTS file on the Exchange 2007 SP3 server:

127.0.0.1 crl.microsoft.com

2. Save the HOSTS file.

3. Stop the Microsoft Search Indexer Service.

4. Stop the Microsoft Search (Exchange) service.

5. Rename the CatalogData folder(s) for the database having the issue – it should be in the same folder as the database(s) with the issue.  If there is not enough available room, then either save the CatalogData folder to another location or delete the CatalogData folder.

6. Start the Microsoft Search (Exchange) service.

7. Start the Microsoft Search Indexer Service.

8. Wait for Event ID 110 in the application log to signify that creating the new index has finished.

9. Check the CatalogData folder(s) to see if .CI files are now being generated.

10. Check searches in Outlook online mode and OWA to see if search now works.

More Information

1. When MSSearch performs the CRL check, it tries to access crl.microsoft.com, and will wait 15 seconds (tbc) for a response. Furthermore, if there are proxy servers configured (especially bad proxy entries), MSSearch attempts to access the CRL URL via each proxy address. Adding the 127.0.0.1 crl.microsoft.com entry to the HOSTS file forces the server to check against itself, which immediately responds saying that the CRL is not available. This bypasses the timeout period.

2. Click on the following links for more information on WinVerifyTrust and WinHTTP:

WinVerifyTrust Function
https://msdn.microsoft.com/en-us/library/aa388208(VS.85).aspx

About WinHTTP
https://msdn2.microsoft.com/en-us/library/aa382925.aspx

Using WinHTTP tools
https://msdn2.microsoft.com/en-us/library/aa384084.aspx

Bob Want, Senior Support Escalation Engineer, Enterprise Communications Services, Microsoft