Exchange Search trouble shooting Guide Exchange 2010

  • Article

There have been many requests for the full trouble shooting steps for Exchange 2010 search and at what point should you add more servers to your environment to help deal with the search load. Here are the steps that we typically do when we receive a support call.

Please note that with OWA, Outlook online mode and E-discovery that there is a 180 second timeout and this value cannot be modified .  

  1. Make sure the Internet proxy is disabled on the mailbox server

    1. Check to see if the proxy is running on the server

      1. From an administrative privileged command line run: netsh winhttp show proxy

    2. Disable the proxy if present

      1. From an administrative privileged command prompt run netsh winhttp reset proxy

  2. If The Test-ExchangeSearch cmdlet fails; modify the MSFTEFD.EXE.CONFIG file as described in:  https://support.microsoft.com/kb/2837276

  3. If you see any 9877 events in the application log; or  a QUERY_E_TOCOMPLEX in the ExTra trace apply the fix rom the following KB:  https://support.microsoft.com/kb/2616127

    1. Ignore steps 4-10

    2. Navigate to HKEY_Local_Machine\Software\Microsoft\ExchangeServer\v14\ExSearch

    3. Right click and select New\DWORD (32-bit) value

    4. The name should be MaxRestrictionNodes

    5. Value should be 1500000 (Decimal) (Up this to 2000000; theoretically this can be increased more but it can cause performance issues so never set it higher than two million)

  4. Disable and re-enable the content indexes for each database, one at a time

    1. Get-MailboxDatabase | FL *index*

    2. Verify -IndexEnabled was set to true

    3. Run the below command to set it to false

      1. Set-Mailboxdatabase DBName -IndexEnabled $false

    4. Rename catalog for the DB

    5. start the Search service

    6. Run the below command with True switch

      1. Set-Mailboxdatabase DBName -IndexEnabled $True

    7. Restart the Search service.

  5. Install Filter pack 2.0 and the latest updates and register them

    1. Filter Pack 2.0: https://www.microsoft.com/en-us/download/details.aspx?id=17062

    2. SP2 for Filter Pack 2.0: https://support.microsoft.com/kb/2687447

    3. Register Filter Pack: https://technet.microsoft.com/en-us/library/ff354976(v=exchg.80).aspx

  6. Make sure all third-party filters such as the Adobe pdf filter are up to date

  7. Move an affected mailbox to another database

  8. Enable Outlook cache mode for all users especially high item/large mailbox users.

  9. Rebuild your content indexes.  One at a time.  If this is completed after the steps above, we should have maximized performance with search and patched it to the newest updates.

If you are still experiencing search issues at this point, there are some extra steps that support will have you run.

There is a good chance that the timeouts are because of store throttling with search queries and performance issues from the server. Here is how we determine from the support side.

 

  1. Turn up diagnostic logging

    1. set-eventloglevel "msexchange search Indexer\General" -level expert

    2. set-eventloglevel "msexchange search indexer\configuration" -level expert

  2. Remove all third-party tools from the MB server

  3. Check high-item counts in mailboxes

  4. Check for resource constraints on the MB server such as processor, memory, disk latency, etc., using ExPerfWiz

  5. Monitor the following counters in perfmon

    1. \MSExchange Search Indices(*)\Average Time to Process Batches

    2. \MSExchange Search Indices(*)\Document Indexing Rate

    3. \MSExchange Search Indices(*)\Indexing Slow

    4. \MSExchange Search Indices(*)\Number of Documents Successfully Indexed

    5. \MSExchange Search Indices(*)\Number of Documents That Failed During Indexing

    6. \MSExchange Search Indices(*)\Number of Failed Mailboxes

    7. \MSExchange Search Indices\Number of Mailboxes Left to Crawl

  6. Collect ExTra trace using with following components and tags

    1. Exchange 2010

      1. ExchangeSearch/*

      2. Store/tagCIPSBrief

      3. Store/tagCIPSConditions

      4. Store/tagCIPSParams

      5. Store/tagCIPSVerbose

      6. Store/tagCIStateMachine

      7. Store/tagQnode

      8. Store/tagQroot

      9. Store/tagQueryCi

      10. Store/tagQueryCiDump

      11. Store/tagRestrict

      12. Store/tagSearch

      13. Store/tagViewCleanup

      14. InfoWorker.Common/Search

  7. Disable the signature verification with DIP.exe

    1. Make sure you STOP and DISABLE the following service on all their mailbox servers: WinHTTP Web Proxy Auto-Discovery

    2. Use the dip.exe tool on all the mailbox servers with a search problem.

    3. Place a copy of dip.exe in the Exchange V14 subdirectory (Exchange 2010)

    4. Open an elevated command prompt in the V14 subdirectory (Exchange 2010)

    5. Run the following command: dip /ADMIN_SETBOOL g FTE_VerifySignature 0

    6. Restart the Microsoft Search Indexer Service and Microsoft Search (Exchange) services

    7. https://blogs.msdn.com/b/roopeshpattan/archive/2009/11/14/search-index-fails-for-all-users-in-a-particular-database-in-exchange-2007.aspx

  8. Capture a procdump of store while reproducing the issue

     

    If we are seeing that batches are not processing after 30 minutes do to server performance issues, we may add in these keys.

  1. Modify the registry to increase the indexing time out

    1. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\ExSearch

    2. Create two DWORD, decimal values

    3. DatabaseCacheTtlInSeconds.  Default is 30 minutes, increase to an hour: 1800

    4. BatchTimeoutInSeconds, default is 1 hour, 1.5 hourr, : 5400

       

       

       

       

      AT this point after following all these additional steps, it is time to build out the environment to add more servers to disperse the load. We always build out and environment and not up.  It is more efficient having a few smaller servers than one large server when it comes to server load.

Tony Geraghty

SR Support Escalation Engineer

Terrance Brennan

Support Escalation Engineer