250 Hello

Random Musings on Exchange and Virtualization

Exchange 2010 Tweaks

Exchange 2010 Tweaks

  • Comments 2
  • Likes

Kris Waters, one of my great colleagues from the US, originally posted a really neat list of items which can mitigate issues found in a lot of large scale Exchange deployments.  Please take the time to review her post here. There are a lot of valuable pointers in her blog, so definitely check it out!

In addition there are a couple of other  items listed below that you may also want to review.

As Kris states, please ensure that you carefully review and test any items mentioned here prior to placing them into production!  As I like to say, some of these items follow the Captain Jack Sparrow words of wisdom – the pirate’s code is more what you’d call guidelines than actual rules…   In other words carefully consider each on its on merits and how it relates to your organisation!

Update 27-11-2013:  Added Logging section

Update 8-1-2014:   Added RPC Client Access detail to Logging section

Update 21-6-2014: Added additional cluster hotfix information

 

Cluster Hotfixes

This hotfix is strongly recommended for DAG servers, and has been for some time now.  This resolves several issues in Windows 2008 R2 SP1.  Exchange 2010 SP3 will prompt to install this update if it is not installed.  The is the GUI view, and the same message is also displayed using command line setup. 

Exchange 2010 SP3 Prompt To Install Cluster Update

 

Most Exchange admins will be aware of this issue, but what is sometimes then missed is the other base Cluster Hotfixes that are recommended by the cluster team.  For example:

Recommended hotfixes for Windows Server 2008-based server clusters

Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters

Recommended hotfixes and updates for Windows Server 2012-based failover clusters

 

Exchange 2010 is typically installed onto Windows 2008 R2, well at least most of the customers I visit do this, so looking at the 2008 R2 cluster updates in detail we see that in the general section there is an “interesting” hotfix contained in KB 2524478 The network location profile changes from "Domain" to "Public" in Windows 7 or in Windows Server 2008 R2.  This was something that you can find a previous post on here.  On a recent case I also saw this change once the server had been running for a wile.  This customer had various network issues that seemed to exacerbate this issue. 

 

Office Filter Pack

Office 2010 Filter pack SP2 is available

 

Networking

Make sure that the network card drivers and firmware are at the correct build level.  This can be a tricky one as you do not necessarily want to just stick on the latest available driver as it was just released.  By carefully testing and evaluating releases you can determine the appropriate builds in conjunction with your hardware vendor of choice.

One other item that is now critical is the firmware of the blade chassis.  Also make sure that its firmware and management components are also at the correct build level.

 

OS Updates

In addition to ensuring that the monthly security updates are installed we sometimes see issues with the following items and it can pay off to keep them in mind when troubleshooting:

  • TCPIP.sys
  • Ntoskrnl.exe
  • Storport.sys

An issue with modern Exchange servers installed on blades is that the blade can have 256GB of memory, but only 146GB drives installed locally so how to configure the pagefile?  Pagefile still needs to be RAM + 10 MB for performance and dumpfile reasons.  Where to put it then and preserve the ability to capture complete memory dumps?

Traditionally if you select the Complete memory dump option, you must have a paging file on the boot volume that is sufficient to hold all the physical RAM plus 1 megabyte (MB).  That does not work in the scenario above!

In Windows Vista, in Windows 7, in Windows Server 2008, and in Windows Server 2008 R2, this paging file can be on a partition that differs from the partition on which the operating system is installed as discussed in Overview of memory dump file options.  There is also another hotfix available that allows you to create a dump file even if you have no pagefile configured at all!  No Exchange admin should be doing this as Exchange requires the pagefile configuration mentioned above!

One note on storport that needs to be called out.  You must check with the storage vendor, especially when SAN storage is used, to ensure the storage vendor supports the version of storport.  The last thing you want is to have performance issues, call them for support and be told that you are in an unsupported position.  That will spoil your day in a hurry!

One interesting issue I did see was around very slow access to performance counters.  The underlying issue was with the Remote Registry service as it was leaking resources.  This is resolved with hotfix 2699780.

 

.NET Update

Hotfix 2497453 is required to resolve an issue with the .NET Framework.  This issue manifests itself when Exchange 2010 SP1 is installed due to the Free/Busy intercept mechanism which was introduced in Exchange 2010 SP1.  This issue is discussed here.

 

Exchange Service Pack

Exchange 2010 SP3 should be installed or you are in the planning stages to install it.  Exchange 2010 SP2 will move out of support on the 8th of April 2014.

Exchange Logging

Note that in Exchange 2010 not all logging is enabled by default.  So if an issue occurs you may need to enable logging and then wait for the issue to reoccur. 

IMAP Logging

Note that the log location must be set first, and then the logging can be enabled

Set-ImapSettings -Server Exch-1 -LogFileLocation D:\Logs\IMAP

Set-ImapSettings -Server Exch-1 -ProtocolLogEnabled $true

POP Logging

Note that the log location must be set first, and then the logging can be enabled

Set-PopSettings -Server  -LogFileLocation D:\Logs\POP

Set-PopSettings -Server <servername> -ProtocolLogEnabled $true

SMTP Receive Connector Logging


Note that the format is server\connector name.  In the below example Exch-1 is the server, and "Default Exch-1" is the default receive connector on that server.

    Set-ReceiveConnector "EXCH-1\Default EXCH-1"  -ProtocolLoggingLevel Verbose

SMTP Send Connector Logging

Set-SendConnector Interwebs -ProtocolLoggingLevel Verbose

SMTP Implicit Intra-Organisation Send Connector Logging

Set-TransportServer Exch-1 -IntraOrgConnectorProtocolLoggingLevel Verbose


 

RPC Client Access Logging

By default, throttling logging is disabled for the RPC client access service. Therefore, you will not see throttling information in the RPC Client Access logs. To enable throttling logging, follow these steps:

  1. Open the following file in a text editor, such as Notepad: C:\Program Files\Microsoft\Exchange Server\V14\Bin Microsoft.Exchange.RpcClientAccess.Service.exe.config
  2. In the file, locate the <add key="LoggingTag" value="ConnectDisconnect, Logon, Failures, ApplicationData, Warnings" /> section.
  3. Type Throttling in the comma-separated string. For example, type Throttling in the string that resembles the following: <add key="LoggingTag" value="ConnectDisconnect, Logon, Failures, ApplicationData, Warnings, Throttling" />.
    Save and then close the file.
  4. Restart the RPC Client Access service.

 

Cluster Log Wrap

Be aware of the wrapping issue with the Windows 2008/2008 R2 cluster log .ETL files

While we do NOT support directly manipulating the DAG’s underlying cluster, it is very useful to look at the cluster logs if there is an issue.  By ensuring that the cluster logs are sized correctly there is less risk of losing valuable troubleshooting data.   

The default cluster log is 100 MB.  In the examples below the new size is indicated by XXX. Size this so you have sufficient cluster log data retention. 

To modify using PowerShell

Set-ClusterLog –Size XXX

 

To Modify using cluster.exe

Cluster.exe LOG /Size:XXX

 

Cluster Log Generation

In the newer versions of Windows, the failover cluster human readable log is not present on disk and must be explicitly generated.  This is different from Windows 2003 and 2000 Failover Clustering where the readable log file was present without any intervention. 

This can be done via the command prompt or PowerShell.

Command Prompt

Cluster.exe LOG /GEN 

Look for the log on each cluster member in the local C:\Windows\Cluster\Reports folder. 

 

 Sometimes you may want to look at logs individually, but typically the command will look like this to dump the cluster log from all notes to a specified central directory so you do not have to manually pull them together:

Cluster.exe  LOG /GEN /COPY:<Directory>

 

Additional information can be found on TechNet. 

 

PowerShell

Get-ClusterLog 

Typically the command will look like this to dump the cluster log from all notes to a specified directory:

Get-ClusterLog –Destination ‘directory’

Additional information can be found on TechNet. 

 

Cluster Heartbeat

We do not recommend changing the default cluster inter and intra subnet heartbeat intervals as a means to resolve underlying network issues.  The network issue should be fixed.  Increasing the heartbeat settings merely masks the underlying issue.

To check what is currently set we can run:

cluster /cluster:<ClusterName> /prop

This will return the following entries:

CrossSubnetDelay               1000
CrossSubnetThreshold           5

SameSubnetDelay                1000
SameSubnetThreshold           5

Or in the land of PowerShell, we can use Get-Cluster to see the properties but make sure that the PowerShell module is loaded up first:

Import-Module FailoverClusters

Then we can run:

Get-Cluster | Format-List *

Note that there is an asterisk after the Format-List command.  That always gets me!!

 

Throttling Policy

Ensure that your users receive the appropriate throttling policy, and the same applies to service accounts!

The default throttling policy should remain unchanged, and you should create new throttling policies for each of the groups of users you wish to have different settings.

One other item worth mentioning is that some of the throttling infrastructure changes have gone unnoticed.  This is generally when I see folks running:

Set-Mailbox <mailboxname> –ThrottlingPolicy MyCustomPolicy

That’s great, but that is only a mailbox.  What about machines that need to interact with Exchange where the content is a computer object rather than a mailbox?

To enhance this the  Get-ThrottlingPolicyAssociation and Set-ThrottlingPolicyAssociation cmdlets were adding in Exchange 2010 SP1.  Use the Set-ThrottlingPolicyAssociation cmdlet to associate a throttling policy with a specific object. The object can be a user with a mailbox, a user without a mailbox, a contact, or a computer account

 

Change Mailbox Quarantine Duration

Update 9-4-2014:  Please see this other post for a more detailed discussion on quarantine.

The default time out for mailbox quarantine in 6 hours in Exchange 2010.  In the scenario where an exec’s mailbox gets quarantine at 09:00 local time then it will stay quarantined until 15:00 unless you take action.

This may not be acceptable for some organisations, and the default value can be changed.

The settings for the amount of failures that lead to quarantining a mailbox and also for the amount of time that a mailbox should stay quarantined are stored in the MailboxQuarantineCrashThreshold and MailboxQuarantineDurationInSeconds keys in:

HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIS\<Server Name>\Private-{db guid} subkey.

The default values for these keys are three failures for MailboxQuarantineCrashThreshold and 21,600 seconds (six hours) for MailboxQuarantineDurationInSeconds.

KB 2603736 discusses the issue.

 

SCOM Management Pack

I wish I had money for every time I asked if SCOM is monitoring Exchange and the reply is yes!  Then we find out that the MP was imported and nothing else was done.  That is not really what we need.  Make sure all the events that you are interested in are actually monitored.  Do not assume the default MP is all you need.

You will also find that overrides will be necessary to suppress items that are not relevant to your business.  For Example:

  • POP and IMAP are disabled by default in Exchange 2007 and 2010.  Yet the MP seeks to monitor them by default
  • You may not have an Internet accessible CAS servers due to an array of reasons, so external URLs may not be populated.  Those external monitors will need to be overridden to disable them.

 

CAS Namespaces

Be sure to set the CAS URLs as per design and not overlook any.

Also ensure that when new Exchange servers are deployed that their URLs are changed immediately to the correct values and not left at the default ones.

 

Outlook Configuration

Be sure to manage Outlook settings fully via GPO.  You have AD and GPOs so use them to your full advantage!  Do not let users create Personal Storage Tribbles (PSTs), lock those settings down!

 

Outlook Build Level

Ensure that Outlook is properly patched to mitigate any security issues, and also to provide fixes to issues!  Neglecting client maintenance will lead to end user impact and should not be over looked.

Cheers,

Rhoderick

 

Can You Help Us?  -- Yes !

If you would like to have Microsoft Premier Field Engineering (PFE) visit your company and assist with the topic(s) presented in this blog post, then please contact your Microsoft Premier Technical Account Manager (TAM) for more information on scheduling and our varied offerings!

If you are not currently benefiting from Microsoft Premier support and you’d like more information about Premier, please email the appropriate contact below, and tell them you how you got introduced!

US

Canada

For all other areas please use the US contact point.





Comments
  • Nicely done dude!

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Post Comment Fixer