Context :

You noticed more than 300 system messages stuck in some HUB transport servers queues with error code : 432 4.3.2 STOREDRV.Deliver ;Recipient thread limit exceeded

This is a known issue from Exchange 2010 SP1, which implements a default limitation on delivery threads, to avoid mail storms or Public Folder replication storms that could take Hub servers down (note that Public Folders are used by Outlook 2003 clients for system messages).

The solution is to slightly increase this default limitation to stay away from this issue with Public Folders replication. Below is the procedure to do this (EdgeTransport.exe.config is located on your \Exchange\V14\Bin folder on your servers)

Hub transport queue building up due to public folder replication messages.

Symptoms

Hub transport queue building up due to public folder replication messages. 
Last Error: 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

Cause

Exchange 2010 SP1 implements delivery thread limits so a hand full of recipients does not cause a disruption on the hub server. Example given, Journal mailbox or public folders.

Resolution

To resolve the issue we added the following settings to edgetransport.exe.config. 
<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

The above information is explained in details in the following article as well:

http://blogs.technet.com/b/exchange/archive/2011/04/11/store-driver-fault-isolation-improvements-in-exchange-2010-sp1.aspx

And confirming the action plan above, here is the relevant part of the article:

- There are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is:

432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

The two scenarios are:

- Journaling

- Public Folders replication

In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). But it can be enough to paralyze or slow down the HUB servers.

Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. The solution is to incrementing both values up one, as follows:

<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

=> In general, Microsoft recommends to stay below 3 for RecipientThreadLimit and below 4 for MaxMAilboxDeliveryPerMdbConnections to avoid performance issues, but you may go upper depending on your hardware performance.

First, a reminder of the theory:

• Exchange store is comprised of 3 services:

• msexchangerepl.exe = Replication Service Process
• Microsoft.Exchange.Store.Service.exe = Store Service Process (or Controller)
• Microsoft.Exchange.Store.Worker.exe = Store Worker Process (1 per DB)

•Algorithm will allocate total ESE cache available for all store worker processes based on physical RAM

– ~25% of total memory allocated to ESE cache

•ESE cache allocated to each database (store worker process) based on number of local database copies and MaximumActiveDatabases configuration

– Static amount of ESE cache allocated to passive and active database copies

•Passive database allocates 20% of max ESE cache target used for active database

– Store worker process will only use max cache target when operating as active

•Max cache target computed at service process startup

– Restart service process when adding/removing copies or changing maximum active database configuration

Example/illustration:

1- Let’s say your server has 64GB RAM

2- Exchange 2013 store will calculate a max cache target

25% x 64GB RAM = 16GB

That means that potentially, Exchange Store can allocate and use 16GB RAM maximum.

That would be if your server is hosting active databases only: as stated above, for passive databases, Exchange will allocate less memory.

See the point 3- below:

3- Say you will host 10 databases on your server.

The Max cache target computed at service process startup (Microsoft.Exchange.Store.Service.exe) will be evenly distributed between each database : 16GB / 10 = 1.6GB.

For databases that are active on this server, Exchange Store will use 1.6GB cache for these databases.

But for the passive ones on this server, it will use only 20% of the 1.6GB that the Store service that is 327.68MB

So for 5 Passive databases and 5 Active databases, the RAM usage for the Store processes will be

( Max allocated cache per DB x nb active DBs ) + 20% x (Max allocated cache per DB x nb of passive DBs)

( 1.6 x 5 ) + 20% x ( 1.6 x 5 ) = 9.6GB

If you activate all 10 databases on this server, then the RAM usage will then just be :

1.6 x 10 = 16GB.

First configure Autoreseed for your DAG(s):

http://technet.microsoft.com/en-us/library/jj619303.aspx

Second, here is the process, explained by Scott Schnoll:

AutoReseed is designed to automatically restore database redundancy after a disk failure by using spare disks that have been provisioned on the system.

CU1 includes numerous fixes to AutoReseed, including fixes for issues around AutoReseed not detecting spare disks correctly and AutoReseed not using detected spare disks.  In addition, the following enhancements have been made to AutoReseed:

• GetCopyStatus now has a new field 'ExchangeVolumeMountPoint', which shows the mount point of the database volume under C:\ExchangeVolumes (or a custom folder if you are not using the default setting of C:\ExchangeVolumes).  This is useful information to know because in a configuration that uses multiple disks per volume, the LogicalDisk performance counters show up as the first mount point (which would be the one under C:\ExchangeVolumes) instead of as the database path like they used to in Exchange 2010 with a single disk per volume.
• We now have better internal tracking around mount paths and the ExchangeVolume path.
• The limits for AutoReseed have been increased from 4 databases per volume in Exchange 2013 RTM to 8 databases per volume in CU1.
• AutoReseed properties have been added to Active Directory that allow you to enable and disable automatic reseeding and the DiskReclaimer function (which formats Exchange volumes). The two new properties are:
  • AutoDagAutoReseedEnabled - Setting AutoDagAutoReseedEnabled to false turns off AutoReseed (including automatic resume, sparing, and in-place reseeds).
  • AutoDagDiskReclaimerEnabled - Setting AutoDagDiskReclaimerEnabled to false turns off the DiskReclaimer, which formats exchange volumes. The default setting is true, and it only tries to format volumes mounted under C:\ExchangeVolumes.
• The unused AutoDagFailedVolumesRootFolderPath property was also removed from the DAG object.

As a result of these and other changes, the workflow for AutoReseed in CU1 has changed.  The primary input condition for the AutoReseed workflow is still a database copy that is in an Failed and Suspended (F&S) state for 15 consecutive minutes.  When that condition is detected, the following AutoReseed workflow is initiated:

1. The system will first try to resume the database copy up to 3 times, with 5 minute sleeps in between each try.  Sometimes, after an F&S database copy is resumed, the copy remains in a Failed state. This can happen for a variety of reasons, so this first step is designed to handle all such cases; AutoReseed will automatically suspend a database copy that has been Failed for 10 consecutive minutes to keep the workflow running. If the suspend and resume actions don’t result in a healthy database copy, the workflow continues.
2. Next, AutoReseed will perform a variety of pre-requisite checks. For example, it will verify that a spare disk is available, that the database and its log files are configured on the same volume, and in the appropriate locations that match the required naming conventions. In a configuration that uses multiple databases per volume, AutoReseed will also verify that all database copies on the volume are in an F&S state.
3. Next, AutoReseed will attempt to assign a spare volume up to 5 times, with 1 hour sleeps in between each try.
4. Once a spare has been assigned, AutoReseed will perform an InPlaceSeed operation using the SafeDeleteExistingFiles seeding switch.  If one or more database files exists, AutoReseed will wait for 2 days before in-place reseeding (based on the LastWriteTime of the database file).  This provides an administrator with an opportunity to preserve data, if needed.  AutoReseed will attempt a seeding operation up to 5 times, with 1 hour sleeps in between each try.

Once all retries are exhausted, the workflow stops.  If, after 3 days, the database copy is still F&S, the workflow state is reset and it starts again from Step 1.  This reset/resume behavior is useful (and intentional) since it can take a few days to replace a failed disk, controller, etc..

Complete article here: https://blogs.technet.com/b/scottschnoll/archive/2013/04/02/high-availability-changes-in-exchange-server-2013-cumulative-update-1.aspx

by Steve Goodman

Introducing Load Balancing in Exchange Server 2013 (Part 1)

http://www.msexchange.org/articles-tutorials/exchange-server-2013/high-availability-recovery/introducing-load-balancing-exchange-server-2013-part1.html

Thanks Steve !

What are CTS and IMS as referred in the Exchange 2013 architecture slides ?

(XSO = Exchange Server Object – structure i.e. properties, attributes, methods associated with the the Exchange Server Object into the AD for example)

CTS = Content Transformation Services + IMS = Interaction Management Services both part of the FAST search engine, acquired by Microsoft years ago.

http://blog.comperiosearch.com/blog/2010/11/10/fast-search-internet-sites/

In summary:

- CTS (referred on the E2013 slides) = Workflow that prepares the DATA (data side) to be indexed

- IMS (not referred above but good to know)= Workflow that processes the searches requests and that sends the search results back to the user.

- XSO = Exchange Server Object

Sam.

Here is what I put together some time ago around internal Microsoft discussions to highlight the key differences between FOPE and EOP:

http://technet.microsoft.com/en-us/library/jj200686(v=exchg.150).aspx

http://www.microsoft.com/online/legal/v2/?docid=25

4.3.1

Out-of-memory or out-of-disk space condition on the Exchange server. Potentially also means out-of-file handles on IIS.

4.3.2

Message deleted from a queue by the administrator via the Queue Viewer interface in Exchange System Manager.

4.4.1

Host not responding. Check network connectivity. If problem persists, an NDR will be issued.

4.4.2

Connection dropped. Possible temporary network problems.

4.4.6

Maximum hop count for a message has been exceeded. Check the message address, DNS address, and SMTP virtual servers to make sure that nothing is causing the message to loop.

4.4.7

Message expired. Message wait time in queue exceeds limit, potentially due to remote server being unavailable.

5.0.0

Generic message for no route is available to deliver a message or failure. If it is an outbound SMTP message, make sure that an address space is available and have proper routing groups listed.

5.1.0

Message categorizer failures. Check the destination addresses and resend the message. Forcing rebuild of Recipient Update Service (RUS) may resolve the issue.

5.1.1

Recipient could not be resolved. Check the destination addresses and resend the message. Potentially e-mail account no longer exists on the destination server.

5.1.3

Bad address.

5.1.4

Duplicate SMTP address. Use LDIFDE or script to locate duplicate and update as appropriate.

5.2.1

Local mail system rejected message, “over size” message. Check the recipient’s limits.

5.2.3

Message too large. Potentially the recipient mailbox is disabled due to exceeding mailbox limit.

5.3.3

The remote server has run out of disk space to queue messages, possible SMTP protocol error.

5.3.5

Message loopback detected.

5.4.0

Authoritative host not found. Check message and DNS to ensure proper entry. Potential error in smarthost entry or SMTP name lookup failure.

5.4.4

No route found to next hop. Make sure connectors are configured correctly and address spaces exist for the message type

5.4.6

Categorizer problems with recipient. Recipient may have alternate recipient specified looping back to self.

5.4.8

Looping condition detected. Server trying to forward the message to itself. Check smarthost configuration, FQDN name, DNS host and MX records, and recipient policies.

5.5.0

Generic SMTP protocol error.

5.5.2

SMTP protocol error for receiving out of sequence SMTP protocol command verbs. Possible to low disk space/memory of remote server.

5.5.3

Too many recipients in the message. Reduce number of recipients in message and resend.

5.7.1

Access denied. Sender may not have permission to send message to the recipient. Possible unauthorized SMTP relay attempt from SMTP client.

Source:

Prerequisites

We’ll use the ExTra.exe and ProcDump Tools to take traces that will capture the issue.

To take memory dump of Store.exe and EdgeTransport.exe for deep code level analysis.

Download ProcDump: http://technet.microsoft.com/en-US/sysinternals/dd996900

The instructions

Scenario

Mails get stuck in some servers, not always the same, but the common point between the servers is that it’s always queues which point to the same database (messages for users in a particular database).

Event Log analysis does not show any issues, event with Event Logging set to “Expert” for MSExchange Transport and MSExchangeIS store delivery components.

We need to take traces that will capture what Exchange is trying to do, and that will enable Microsoft to tell what object is blocking the queue(s).

Traces

Traces: take both ExTra traces and Dumps from Store and Transport (or other process if it’s for a different scenario)

These will be analyzed by an accredited Microsoft Engineer.

1 of 2 > Extra.exe

Launch ExTra.exe (available by default on Exchange 2010 and Exchange 2013) and choose the “Trace Control” option

Then click on “Set manual trace tags” on the next screen:

Then select the components we with to trace (for our scenario example, Transport and StoreDriver)

and click on the below “Start Tracing” link.

To stop the tracing, when the issue will occur and when the Dumps below will be taken, click on “Stop Tracing now”

2 of 2> process memory dump - NOTE : do not copy/paste the below lines, just retype them as for some reason it takes invisible special characters:

• On one HUB, run this from the ProcDump directory:

procdump.exe -ma edgetransport.exe –n 3 –s 15 -accepteula c:\dumps

• On the Mailbox Server, run this from the ProcDump directory - NOTE the "-mp" parameter instead of the "-ma", because otherwise we'll get 10s GBs of data that we don't necessarily need for store.exe debug:

procdump.exe -mp store.exe –n 3 –s 15 -accepteula c:\dumps

Note: you may not want to place the dump output files in the C:\ drive, you can specify any path you want replacing c:\dumps by any other directory.

More great and very useful information on this link :

Procdump: How to PROPERLY gather dump (.dmp) files for crashes and hangs, CPU spikes, etc, including gathering PERF data, for Exchange issues – by Kris Waters, Premier Field Engineer in Microsoft US.

http://blogs.technet.com/b/kristinw/archive/2012/10/03/procdump-how-to-properly-gather-dump-dmp-files-for-crashes-and-hangs.aspx

Sam.

Hey,

If you ever encounter some messages that takes time to be delivered from HUB servers to Mailbox servers (we call this step "Mapi Submission" or "MAPI Delivery") or even from HUB to another HUB in a remote site, and your network don't have bandwidth or latency or general reliability or other configuration issues such as wrong MTU, wrong speed set on equipments, then check for the Windows TCP Chimney and RSS settings.

These are part of what we call the "Network Scalability Pack" (http://technet.microsoft.com/en-us/network/bb545631.aspx) and their primary use is to offload the CPU from processing network packets, offloading it down to the network card. It's supposed to remove network processing load from the CPU and give the work to the network card instead.

In the past, when installed on Windows 2003 servers, we had issues with this setting being used with Exchange (see this : http://support.microsoft.com/kb/948496), and we recommended to deactivate these TCP chimney, RSS and sometimes network DMA features as well. But experience showed that we can encounter sporadic issues as well on Windows 2008/Exchange 2007 and 2010 (especially for HUB submission queues) under certain circumstances due to these settings; these circumstances can be the network card driver TCP offload features not very compatible with how Exchange uses the network, or simply the network card driver being too old to handle the TCP offloading features... hard to tell exactly why anyways until we do a full raw debugging, which I didn't do.

Anyways, when we encountered network issues on Windows 2008/Exchange 2007/2010 and we also troubleshooted the issue thoroughly to ensure that were not related to bandwidth , latency , general reliability , other configuration issues such as wrong MTU, wrong speed set on equipments,bad network driver, etc..., disabling TCP chimney and RSS magically solved our issues; AND, no reboot is theorically necessary after changing these using netsh (but as we always say, when in doubt, reboot).

• To check for the current TCP Chimney, RSS and NetDMA settings do :

Netsh int tcp show global

For Windows 2008 R2 it will give you the following output-ish:

PS C:\Users\Administrator.CONTOSO> netsh int tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : automatic
NetDMA State                        : enabled
Direct Cache Acess (DCA)            : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : ctcp
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled

• To print out network cards that actually have the TCP Chimney functionnality activated :

Netsh int tcp show chimneystats

On the below example, you see Under the “Supp” column the value “No”, meaning that TCP Chimney is not supported or not enabled on the Network card driver settings, otherwise you’d see “Yes” with some values on “TMax” and “PMax”.

PS C:\Users\Administrator.CONTOSO> netsh int tcp show chimneystats

Idx  Supp  Atmpt    TMax    PMax     Olc              Failure Reason
---  ----  -----    ----    ----     ---              --------------
  11    No  -n/a-       0       0       0                       -n/a-

  Idx           - Interface (NIC) index used by the system
  Supp          - Interface (NIC) supports TCP chimney offload
  Atmpt         - System has attempted TCP connection offload
  TMax          - Offload capacity advertized by the NIC
  PMax          - Offload capacity observed by the system
  Olc           - Number of currently offloaded connections
  FailureReason - Reason why last attempt to offload a connection failed

Use `netsh int tcp show chimneystats <Idx>' for more details.

• To disable TCP Chimney and RSS:

netsh int tcp set global chimney=disabled

netsh int tcp set global rss=disabled

For details and reference about the above commands, see http://support.microsoft.com/kb/951037

The StartDagMaintenance.ps1 script pauses servers cluster node and blocks chosen servers and databases from being mounted. Here is what it does in the below example, for en environment that has 3 or more Servers with 3 or more database copies per database:

Here is the logic used by the script (big thanks to Allan Wang – US PFE for putting the below together !):

• SERVER_1 (who is already in maintenance) was chosen because

• 1/ the activation preference is set to “1”,

AND

• 2/ it has the following conditions: “Status=Healthy, ContentIndexing=Healthy, CopyQueueLength=0, ReplayQueueLength=0”

VERBOSE: [01:45:10.133 UTC] Move-DagMasterCopy: Entering: `$db=DB10, `$srcServer=SERVER_2, `$preferredTarget=

VERBOSE: [01:45:10.180 UTC] Test-DagTargetCopy: Testing move criteria for DB10\SERVER_1, with `$Lossless=True and

`$CICheck=False ...

VERBOSE: [01:45:10.196 UTC] Test-DagTargetCopy: Name='RED02\RED-MLT-1', Status='Healthy', CIStatus='Healthy',

CopyQueueLength=0, ReplayQueueLength=0

VERBOSE: [01:45:10.196 UTC] Test-DagTargetCopy: Leaving (returning 'True')

The suggestion made to administrators is to suspend the database after the server is put into maintenance so that the logic would most likely skip it (Overall status will change to “Suspended” then the script won’t try to mount databases on it)

*********** Below example put together by Bernard Chouinard, Canadian independant Consultant *********

• Start the Dag maintenance on the first server with the dag maintenance script (StartDagMaintenance.ps1 –ServerName <First_Server_Name>)

• Then suspend databases on this same first server with the below script:

#Get database on the first server

$dbases = Get-MailboxDatabaseCopyStatus -Server servername

#Suspend the database copy, this is so when the second server is put into maintenance mode it does not select the first server to move databases to

$dbases | %{if ((Get-MailboxDatabase $($_.DatabaseName)).ReplicationType -eq 'Remote'){Suspend-MailboxDatabaseCopy $($_.Name) -SuspendComment 'Serverbeing patched and rebooted' -Confirm:$false}}

• Start the Dag maintenance on the second server with the dag maintenance script (StartDagMaintenance.ps1 –ServerName <SECOND_Server_Name>) – NOTE: databases will be on a third server, but the script won’t try to mount databases on the FIRST server as it’s suspended now.

• Then resume (“un-suspend”) databases on the first server – NOTE: they will continue to replicate, but still be blocked from activation as we ran StartDagMaintenance.ps1 earlier…

#Resume the database copy on the first server

$dbases | %{if ((Get-MailboxDatabase $($_.DatabaseName)).ReplicationType -eq 'Remote'){Resume-MailboxDatabaseCopy $($_.Name) -Confirm:$false}}

****************** Example end – Thank you very much Bernard ! *****************

Or, if you are a scripting warrior, you can add an “IF” statement to test the DatabaseCopyAutoActivationPolicy  parameter and/or the database status (as the StartDagMaintenance.ps1 script also set individual databases as suspended for activation only).

Sam.

Users on Exchange 2010 who try to update or create a Distribution List may get the following error message:

"Changes to the public group membership cannot be saved. You do not have sufficient permission to perform this operation on this object.”

Usually, it’s because they don’t have the permissions (reminder: RBAC only for Exchange 2010 users) to create or modify security groups).

To assign these permissions to a security group in which you add users (recommended) or to a single user:

1 – First create a security group

2 – All the DL owners should be member of this group ( This way you do not have to assign permissions to individual users, it will be easier to manage permissions Only for the required users, )

3 – Assign permissions to security group (recommended)

• New-managementroleassignment –role “Security group creation and membership” –securitygroup “DL Owners”

or to assign the roles directly to user

• New-managementroleassignment –role “Security group creation and membership” –User “username”

5 – Add all the users who needs to manage DL’s to the security group

6 – Wait for AD Replication

7 – now users will be able to manage Mail enabled Security DL using outlook

8 – Any helpdesk users can use Exchange management shell and manage DL membership

9 - logout and login to their outlooks and try to change the Security group membership

NOTE : If you configured Outlook 2010 or earlier to directly use a GC (using the “ClosestGC” or the “DS Server” registry key) as per http://support.microsoft.com/kb/319206 , users won’t be able to manage DL memberships or creation as this will bypass RBAC check and use AD permissions instead – it’s not recommended you give users direct AD permissions – Also remember that the registry keys described in the above article (KB 319206) works, but is NOT SUPPORTED when mailboxes are in Exchange 2010/2013 (one example why is the bypass of RBAC permissions for AD related operations like DL management). So please only use these for troubleshooting purposes.

Sam.

Choice of two useful Tools to monitor AD replication.

• First is Active Directory Replication Status Tool.
• Second is REPADMIN.

> Examples below are run in an AD Forest named “Contoso.ca”, composed of 1 domain and 2 DCs (GC2K7 and E2010) with no errors.

Repadmin gives something similar to:

PS C:\Users\Administrator> repadmin /showrepl * /csv

showrepl_COLUMNS,Destination DSA Site,Destination DSA,Naming Context,Source DSA Site,Source DSA,Transport Type,Number of

Failures,Last Failure Time,Last Success Time,Last Failure Status

showrepl_INFO,SITE1,GC2K7,"DC=Contoso,DC=ca",SITE1,E2010,RPC,0,0,2013-09-23 11:48:03,0

showrepl_INFO,SITE1,GC2K7,"CN=Configuration,DC=Contoso,DC=ca",SITE1,E2010,RPC,0,0,2013-09-23 11:48:03,0

showrepl_INFO,SITE1,GC2K7,"CN=Schema,CN=Configuration,DC=Contoso,DC=ca",SITE1,E2010,RPC,0,0,2013-09-23 11:48:03,0

showrepl_INFO,SITE1,GC2K7,"DC=DomainDnsZones,DC=Contoso,DC=ca",SITE1,E2010,RPC,0,0,2013-09-23 11:59:12,0

showrepl_INFO,SITE1,GC2K7,"DC=ForestDnsZones,DC=Contoso,DC=ca",SITE1,E2010,RPC,0,0,2013-09-23 11:48:03,0

showrepl_INFO,SITE1,E2010,"DC=Contoso,DC=ca",SITE1,GC2K7,RPC,0,0,2013-09-23 12:01:53,0

showrepl_INFO,SITE1,E2010,"CN=Configuration,DC=Contoso,DC=ca",SITE1,GC2K7,RPC,0,0,2013-09-23 11:57:33,0

showrepl_INFO,SITE1,E2010,"CN=Schema,CN=Configuration,DC=Contoso,DC=ca",SITE1,GC2K7,RPC,0,0,2013-09-23 11:57:33,0

showrepl_INFO,SITE1,E2010,"DC=DomainDnsZones,DC=Contoso,DC=ca",SITE1,GC2K7,RPC,0,0,2013-09-23 11:59:27,0

showrepl_INFO,SITE1,E2010,"DC=ForestDnsZones,DC=Contoso,DC=ca",SITE1,GC2K7,RPC,0,0,2013-09-23 11:57:33,0

ADRST gives the following :

Exportable as an Excel table as well:

Sam.

Just a reminder for me (Extrace.exe is an internal tool used for deep debugging).

extrace.exe -c -v ExchangeDebugTraces.etl >TraceOut.csv

We have to use the Microsoft.Exchange.Diagnostics.dll matching the Exchange build where the trace come from. To check your Exchange server build, checkout the following link:

http://social.technet.microsoft.com/wiki/contents/articles/240.exchange-server-and-update-rollups-build-numbers.aspx

Sam.

First, as my colleague Rhoderick wrote in a nice and easy-to-read “Troubleshooting Content Indexing” article, we can use the Troubleshoot-CI.ps1 script to try detect other potential issues such as deadlocks (threads blocked and search is waiting on these to continue to index), corruption (not likely here as the state is “Healthy” for the indexes), stall (usually a one time issue that a restart of the Exchange Search service solves), backlog.

In case the above troubleshooter is not identifying any issues, we can proceed to the below steps.

First, check that any File-Level anti-virus exclude the scan of any Exchange Index files (as well as DB files, and other Exchange files).

Then, rebuild the symbolic links for each language that the SP may have striped out, and reset the index:

-> Open Exchange Management Shell by right-clicking the shortcut and selecting “Run as Administrator”.

-> Navigate to the Scripts folder from Exchange Management Shell.

-> Run the command, ".\Repair-ExchangeSearchSymlinks.ps1" and check that it is completing successfully.

-> Run the command, ".\ResetSearchIndex.ps1 -force "Mailbox Database" and check that it’s completing successfully.

Then disable TCP Chimney, RSS and NetDMA that may be a cause for this issue as well

-> Open Command Prompt and run the following NetSH commands:

Disable TCP Chimney (the functionnality to offload the network packet processing from the CPU to the Network Card)
=> netsh int tcp set global chimney=disabled

Disable RSS
=> netsh int tcp set global rss=disabled

Disable NetDMA (Network Direct Memory Access)
=> Opened Registry Editor and navigated to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
=> Right click Parameters and add a new DWORD (32-bit) EnableTCPA and set the value to 0.

Information for TCP, RSS, NetDMA/EnableTCPA can be found on http://support.microsoft.com/kb/951037/en-us

Optionnally, you can also disable IPV6 as well:
=> Open Registry Editor and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
=> Right click Parameters and add a new DWORD (32-bit) DisabledComponents and set the value to 0xffffffff in Hexadecimal.

More information on http://technet.microsoft.com/en-us/network/cc987595.aspx

Finally, reboot the Exchange server to apply the above changes.

I recommend you test this procedure on a Lab first to get familiar with it, and to validate it in your environment before implementing these into production.

Sam.

Want to pull out statistics from Exchange such as :

· messages sent per mailbox per day

· messages received per mailbox per day

· average message size

… And more if you want ?

Use Powershell and Excel now !

I- If you want to pull data from Exchange 2003 to 2007 Exchange versions, then you can use Exchange Profile Analyzer (EPA):

This tool uses WebDAV to interrogate the mailboxes and generates user profile data.

II- If you want to pull data from Exchange 2007/2010 servers, then you have to use Powershell to pull out these data

You cannot use Exchange Profile Analyzer on Exchange 2010 (you still can on Exchange 2007) as in Exchange 2010 and after, we gave up the WebDav protocol.

But we now use Powershell to query and ask the Exchange 2010 (and 2007) message tracking logs to pull out the same data as we had on EPA.

You can use the following TechNet script:

Exchange 2007/2010 Email stats

http://gallery.technet.microsoft.com/scriptcenter/bb94b422-eb9e-4c53-a454-f7da6ddfb5d6

And here is how to interpret data using Excel functions. Quick and easy:

User Profile Analysis for Exchange Server 2010

http://blogs.technet.com/b/neiljohn/archive/2011/08/09/user-profile-analysis-for-exchange-server-2010.aspx

Thanks very much to mjolinor and Neil Johnson for their scripts and explanations !

I- Default WMI memory quotas for Windows 2008 R2:

Memory Per Host:  536870912
Handles Per Host:  4096
Memory All Hosts:  1073741824
Process Limit All Hosts:  32
Threads Per Host:  256

To check what is the current ones on any Windows 2003 and up servers, you can use the following script, and pipe one or more servers on the function from the following script:

#Disclaimer : The below script takes a file with server names as pipeline input.

#You can write a better function with more time with parameters, error control, etc... 

 #    

#How to use :     

#1- run the script to load the function    

#2- create a text file with server names for which we want to check the WMI quotas    

#3- Launch Get-Content C:\temp\ServerNames.txt | Check-AllWMIQuotas    

Function Check-AllWMIQuotas {

      Begin {    

      Write-Host "Beginning to process servers from the pipeline"    

      }    

      Process {    

      #Load the __ProviderHostQuotaConfiguration class from the root namespace    

      $objWMI=get-wmiobject -ComputerName $_ -Namespace root –Class __ProviderHostQuotaConfiguration       

      Write-Host "----------- Checking Server $_ ------------"    

      Write-host "Memory Per Host: "$objWMI.MemoryPerHost            

      Write-Host "Handles Per Host: "$objWMI.HandlesPerHost    

      Write-Host "Memory All Hosts: "$objWMI.MemoryAllHosts    

      Write-Host "Process Limit All Hosts: "$objWMI.ProcessLimitAllHosts

      Write-Host "Threads Per Host: "$objWMI.ThreadsPerHost    

      }     

      End {    

      Write-Host Finished !    

      }     

}    

II- Either monitor WMIPrsve process using Perfmon (Process\Private Bytes of all WMI processes (WmiPrvSE*)) to see if memory is skyrocketting and/or take a dump of the WmiPrvSE process to see which application is using all WMI memory from the allocated quota.

See http://blogs.technet.com/b/kevinholman/archive/2010/06/09/wmi-leaks-memory-on-server-2008-r2-monitored-agents.aspx for some détails.

III- Or if it appears an application needs more WMI memory than allowed by the default quotas, raise the quotas.

Usually doubling the “Memory Per Host” property is enough. But I usually try first by doubling all properties.

You can either use WBEMTEST to change the WMI quota, or use Powershell along with GWMI (or Get-WMIObject) to change these on several servers if needed.

Here is a sample script :

#Disclaimer : The below script takes a file with server names as pipeline input.   

#You can do a better function with more time with parameters, error control, etc...

#How to use :

#1- run the script to load the function (copy/paste in a .PS1 file or copy/paste it directly on a Powershell window)

#2- create a text file with server names for which we want to change the WMI quotas   

#3- Launch Get-Content C:\temp\ServerNames.txt | Double-AllWMIQuotas

  Function Double-AllWMIQuotas {    

      Begin {    

      Write-Host "Beginning to process servers from the pipeline"    

      Write-Host "Storing Windows 2008 R2 default"    

      $defaultMemoryPerHost=536870912    

      $defaultHandlesPerHost=4096    

      $defaultMemoryAllHosts=1073741824    

      $defaultProcessLimitAllHosts=32    

      $defaultThreadsPerHost=256    

      } 

      Process {

      #Load the __ProviderHostQuotaConfiguration class from the root namespace    

      $objWMI=get-wmiobject -ComputerName $_ -Namespace root -Class __ProviderHostQuotaConfiguration    

     #Usually, the below line should be enough, but I recommend to double all values    

      $objWMI.MemoryPerHost=$defaultMemoryPerHost*2    

      #Again, usually the below lines are optional but best practices to be sure we won’t have any Out Of Memory error.    

      $objWMI.HandlesPerHost=$defaultHandlesPerHost*2    

      $objWMI.MemoryAllHosts=$defaultMemoryAllHosts*2    

      $objWMI.ProcessLimitAllHosts=$defaultProcessLimitAllHosts*2    

      $objWMI.ThreadsPerHost=$defaultThreadsPerHost*2 

      #In WMI programming, we have to push the above changes to be applied to the object    

      $objWMI.put()    

      } 

      End {

      Write-Host Finished !

      }  

}

Then, as instructed on the comments on the above scripts, either use a Get-content Servers_List.txt | Double-AllWMIQuotas or use “Server1”,”Server2”,”Server3” | Double-AllWMIQuotas

Here is how to test it on a user first.

1> Using powershell, do a $objMailbox = Get-Mailbox Some_Test_User to get the user we will assign the Out Of Office to.

2> Now define your text (best is to do it in HTML) and store it on 2 variables $InternalMsg and $ExternalMsg like in the example below:

$InternalMsg = "-- Internal Message -- <BR><BR>Hi, <BR><BR>I am not available until next Monday.<BR><BR>For any emergency, please contact my backup John at 613-555-6789.<BR><BR>Jack Doe"

$ExternalMsg = "-- External Message -- <BR><BR>Hi, <BR><BR>I am not available until next Monday.<BR><BR>For any emergency, please contact my backup John at 613-555-6789.<BR><BR>Jack Doe"

3> Now, it’s time to witness the effect of the OOF application on the user’s mailbox.

Open either an Outlook or an Outlook Web App session and check the Out Of Office status, which should be deactivated, with no messages configured (greyed anyways) or some older messages configured :

4> Launch the commandlet against the test user by piping the $ObjMailbox variable we set above with the test user mailbox (UserTest1 in my example):

$objMailbox | Set-MailboxAutoReplyConfiguration -AutoReplyState enabled -ExternalAudience all -InternalMessage $InternalMsg -ExternalMessage $ExternalMsg

5> Check in Outlook or OWA that the message is now set to “Send automatic replies” and also that we have the correct message as defined in the script, for both internal and External OOF messages :

Once you verified it, to apply the OOF to the remainder of the mailboxes, or in my example to all mailboxes containing the string “User”, define your $objMailbox variable with a Get-Mailbox *User* for example, and reapply the powershell command:

$objMailbox | Set-MailboxAutoReplyConfiguration -AutoReplyState enabled -ExternalAudience all -InternalMessage $InternalMsg -ExternalMessage $ExternalMsg

Note: To deactivate the OOF using Powershell on one or more users (depending how many mailboxes you have on the $ObjMailbox variable above), user the following command:

$objMailbox | Set-MailboxAutoReplyConfiguration -AutoReplyState disabled

Did you ever have users complaining that meetings didn’t show up for some attendees ? Or a boss scheduling a meeting, and the meeting disappears from his calendar ?

Note: if the Calendar Repair Assistant has to do too many repairs, there might be an application that is constantly corrupting the calendar(s).

This can be:

- An Outlook add-on

- a misconfigured BlackBerry Server

- file level antivirus scanning the OST and/or the TMP files on the Outlook working directory (on the users’s profile Appdata/Roaming directory)

- too many delegates on a single calendar, thus creating conflicts and resulting on meetings not being updated or created

…

Remeber ISINGEG ? The Exchange tool to repair logical corruptions in databases ? Now this tool is over, replaced by a commandlet in Exchange 2010+ which does not need to dismount databases !

When you install patches, RollUps, Update Intérims, Cumulative Updates, use the Powershell scripts that are available by default on all Exchange 2010/2013 installation folders !

Easy and safe !

Sam.

Since Exchange 2010 SP1, you can enable DAC mode (Datacenter Activation Coordination) even if you have a DAG on a single AD site. Lots of people are not aware that DAC mode not only helps preventing split brain scenario using a smart and simple mechanism (“hello I’m there and I’ve mounted databases” bit, and boot times), but also add 3 commandlets to make DR switchover / failover easier !

Note: This is also valid for Echange 2013

More info about DAC mode, check the differences, not so much different from Exchange 2010 SP1 to Exchange 2013 – YaY, less new things to learn !

DAC mode for Exchange 2013

DAC mode for Exchange 2010

Even if you have your CAS Array already established, double check the RPCClientAccessServer attribute on all your databases ! Experience show that on 80% cases, customers still have the wrong value for some databases…

Hello all,

From now on, and before moving on to Exchange 2013, I will post a series of common day-to-day How-To procédures for Exchange 2007/2010.

Stay tuned,

Cheers,

Sam.