Context :

You noticed more than 300 system messages stuck in some HUB transport servers queues with error code : 432 4.3.2 STOREDRV.Deliver ;Recipient thread limit exceeded

This is a known issue from Exchange 2010 SP1, which implements a default limitation on delivery threads, to avoid mail storms or Public Folder replication storms that could take Hub servers down (note that Public Folders are used by Outlook 2003 clients for system messages).

The solution is to slightly increase this default limitation to stay away from this issue with Public Folders replication. Below is the procedure to do this (EdgeTransport.exe.config is located on your \Exchange\V14\Bin folder on your servers)

Hub transport queue building up due to public folder replication messages.

Symptoms

Hub transport queue building up due to public folder replication messages. 
Last Error: 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

Cause

Exchange 2010 SP1 implements delivery thread limits so a hand full of recipients does not cause a disruption on the hub server. Example given, Journal mailbox or public folders.

Resolution

To resolve the issue we added the following settings to edgetransport.exe.config. 
<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

The above information is explained in details in the following article as well:

http://blogs.technet.com/b/exchange/archive/2011/04/11/store-driver-fault-isolation-improvements-in-exchange-2010-sp1.aspx

And confirming the action plan above, here is the relevant part of the article:

- There are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is:

432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

The two scenarios are:

- Journaling

- Public Folders replication

In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). But it can be enough to paralyze or slow down the HUB servers.

Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. The solution is to incrementing both values up one, as follows:

<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

=> In general, Microsoft recommends to stay below 3 for RecipientThreadLimit and below 4 for MaxMAilboxDeliveryPerMdbConnections to avoid performance issues, but you may go upper depending on your hardware performance.

First, a reminder of the theory:

• Exchange store is comprised of 3 services:

• msexchangerepl.exe = Replication Service Process
• Microsoft.Exchange.Store.Service.exe = Store Service Process (or Controller)
• Microsoft.Exchange.Store.Worker.exe = Store Worker Process (1 per DB)

•Algorithm will allocate total ESE cache available for all store worker processes based on physical RAM

– ~25% of total memory allocated to ESE cache

•ESE cache allocated to each database (store worker process) based on number of local database copies and MaximumActiveDatabases configuration

– Static amount of ESE cache allocated to passive and active database copies

•Passive database allocates 20% of max ESE cache target used for active database

– Store worker process will only use max cache target when operating as active

•Max cache target computed at service process startup

– Restart service process when adding/removing copies or changing maximum active database configuration

Example/illustration:

1- Let’s say your server has 64GB RAM

2- Exchange 2013 store will calculate a max cache target

25% x 64GB RAM = 16GB

That means that potentially, Exchange Store can allocate and use 16GB RAM maximum.

That would be if your server is hosting active databases only: as stated above, for passive databases, Exchange will allocate less memory.

See the point 3- below:

3- Say you will host 10 databases on your server.

The Max cache target computed at service process startup (Microsoft.Exchange.Store.Service.exe) will be evenly distributed between each database : 16GB / 10 = 1.6GB.

For databases that are active on this server, Exchange Store will use 1.6GB cache for these databases.

But for the passive ones on this server, it will use only 20% of the 1.6GB that the Store service that is 327.68MB

So for 5 Passive databases and 5 Active databases, the RAM usage for the Store processes will be

( Max allocated cache per DB x nb active DBs ) + 20% x (Max allocated cache per DB x nb of passive DBs)

( 1.6 x 5 ) + 20% x ( 1.6 x 5 ) = 9.6GB

If you activate all 10 databases on this server, then the RAM usage will then just be :

1.6 x 10 = 16GB.

First configure Autoreseed for your DAG(s):

http://technet.microsoft.com/en-us/library/jj619303.aspx

Second, here is the process, explained by Scott Schnoll:

AutoReseed is designed to automatically restore database redundancy after a disk failure by using spare disks that have been provisioned on the system.

CU1 includes numerous fixes to AutoReseed, including fixes for issues around AutoReseed not detecting spare disks correctly and AutoReseed not using detected spare disks.  In addition, the following enhancements have been made to AutoReseed:

• GetCopyStatus now has a new field 'ExchangeVolumeMountPoint', which shows the mount point of the database volume under C:\ExchangeVolumes (or a custom folder if you are not using the default setting of C:\ExchangeVolumes).  This is useful information to know because in a configuration that uses multiple disks per volume, the LogicalDisk performance counters show up as the first mount point (which would be the one under C:\ExchangeVolumes) instead of as the database path like they used to in Exchange 2010 with a single disk per volume.
• We now have better internal tracking around mount paths and the ExchangeVolume path.
• The limits for AutoReseed have been increased from 4 databases per volume in Exchange 2013 RTM to 8 databases per volume in CU1.
• AutoReseed properties have been added to Active Directory that allow you to enable and disable automatic reseeding and the DiskReclaimer function (which formats Exchange volumes). The two new properties are:
  • AutoDagAutoReseedEnabled - Setting AutoDagAutoReseedEnabled to false turns off AutoReseed (including automatic resume, sparing, and in-place reseeds).
  • AutoDagDiskReclaimerEnabled - Setting AutoDagDiskReclaimerEnabled to false turns off the DiskReclaimer, which formats exchange volumes. The default setting is true, and it only tries to format volumes mounted under C:\ExchangeVolumes.
• The unused AutoDagFailedVolumesRootFolderPath property was also removed from the DAG object.

As a result of these and other changes, the workflow for AutoReseed in CU1 has changed.  The primary input condition for the AutoReseed workflow is still a database copy that is in an Failed and Suspended (F&S) state for 15 consecutive minutes.  When that condition is detected, the following AutoReseed workflow is initiated:

1. The system will first try to resume the database copy up to 3 times, with 5 minute sleeps in between each try.  Sometimes, after an F&S database copy is resumed, the copy remains in a Failed state. This can happen for a variety of reasons, so this first step is designed to handle all such cases; AutoReseed will automatically suspend a database copy that has been Failed for 10 consecutive minutes to keep the workflow running. If the suspend and resume actions don’t result in a healthy database copy, the workflow continues.
2. Next, AutoReseed will perform a variety of pre-requisite checks. For example, it will verify that a spare disk is available, that the database and its log files are configured on the same volume, and in the appropriate locations that match the required naming conventions. In a configuration that uses multiple databases per volume, AutoReseed will also verify that all database copies on the volume are in an F&S state.
3. Next, AutoReseed will attempt to assign a spare volume up to 5 times, with 1 hour sleeps in between each try.
4. Once a spare has been assigned, AutoReseed will perform an InPlaceSeed operation using the SafeDeleteExistingFiles seeding switch.  If one or more database files exists, AutoReseed will wait for 2 days before in-place reseeding (based on the LastWriteTime of the database file).  This provides an administrator with an opportunity to preserve data, if needed.  AutoReseed will attempt a seeding operation up to 5 times, with 1 hour sleeps in between each try.

Once all retries are exhausted, the workflow stops.  If, after 3 days, the database copy is still F&S, the workflow state is reset and it starts again from Step 1.  This reset/resume behavior is useful (and intentional) since it can take a few days to replace a failed disk, controller, etc..

Complete article here: https://blogs.technet.com/b/scottschnoll/archive/2013/04/02/high-availability-changes-in-exchange-server-2013-cumulative-update-1.aspx

by Steve Goodman

Introducing Load Balancing in Exchange Server 2013 (Part 1)

http://www.msexchange.org/articles-tutorials/exchange-server-2013/high-availability-recovery/introducing-load-balancing-exchange-server-2013-part1.html

Thanks Steve !

What are CTS and IMS as referred in the Exchange 2013 architecture slides ?

(XSO = Exchange Server Object – structure i.e. properties, attributes, methods associated with the the Exchange Server Object into the AD for example)

CTS = Content Transformation Services + IMS = Interaction Management Services both part of the FAST search engine, acquired by Microsoft years ago.

http://blog.comperiosearch.com/blog/2010/11/10/fast-search-internet-sites/

In summary:

- CTS (referred on the E2013 slides) = Workflow that prepares the DATA (data side) to be indexed

- IMS (not referred above but good to know)= Workflow that processes the searches requests and that sends the search results back to the user.

- XSO = Exchange Server Object

Sam.