Exchange Team Blog : Exchange 2010

Using Exchange Web Services to Apply a Personal Tag to a Custom Folder

Bharat Suneja [MSFT] — Mon, 20 May 2013 15:50:00 GMT

In Exchange 2010, we introduced Retention Tags, a Messaging Records Management (MRM) feature that allows you to manage email lifecycle. You can use retention policies to retain mailbox data for as long as it’s required to meet business or regulatory requirements, and delete items older than the specified period.

One of the design goals for MRM 2.0 was to simplify administration compared to Managed Folders, the MRM feature introduced in Exchange 2007, and allow users more flexibility. By applying a Personal Tag to a folder, users can have different retention settings apply to items in that folder than the default tag applied to the entire mailbox(known as a Default Policy Tag). Similarly, users can apply a different tag to a subfolder than the one applied to the parent folder. Users can also apply a Personal Tag to individual items, allowing them the freedom to organize messages based on their work habits and preference, rather than forcing them to move messages, based on the retention requirement, to an admin-controlled Managed Folder.

You can still use Managed Folders in Exchange 2010, but they’re not available in Exchange 2013.

For a comparison of Retention Tags with Managed Folders and migration details, see Migrate Managed Folders.

If you like the Managed Folders approach of being able to create a folder in the user’s mailbox and configure a retention setting for that folder, you can use Exchange Web Services (EWS) to accomplish something similar, with some caveats mentioned later in this post. You can write your own code or even a PowerShell script to create a folder in the user’s mailbox and apply a Personal Tag to it. There are scripts available on the interwebs, including some code samples on MSDN to accomplish this. For example:

Stamping Retention Policy Tag using EWS Managed API 1.1 from PowerShell (Exchange 2010)
5 Lesser Known Operations in Exchange Web Services on Exchange 2013, Exchange MVP Glen Scales’ post on the MVP Program Blog, which uses a simpler method to do this on Exchange 2013.

Note: The above scripts are examples for your reference. They’re not written or tested by the Exchange product group.

But is it supported?

We frequently get questions about whether this is supported by Microsoft. Short answer: Yes. Exchange Web Services (EWS) is a supported and documented API, which allows ISVs and customers to create custom solutions for Exchange.

When using EWS in your code or PowerShell script to apply a Personal Tag to a folder, it’s important to consider the following:

For Developers

EWS is meant for developers who can write custom code or scripts to extend Exchange’s functionality. As a developer, you must have a good understanding of the functionality available via the API and what you can do with it using your code/script.
Support for EWS API is offered through our Exchange Developer Support channels.

For IT Pros

If you’re an IT Pro writing your own code or scripts, you’re a developer too! Above applies to you.
If you’re an IT Pro using 3rd-party code or scripts, including the code samples & scripts available on MSDN, TechNet or elsewhere on the interwebs, we recommend that you follow the general best practices for using such code or scripts, including (but not limited to)the following:
- Do not use code/scripts from untrusted sources in a production environment.
- Understand what the script or code does. (This is easy for scripts – you can look at the source in a text editor.)
- Test the script or code thoroughly in a non-production environment, including all command-line options/parameters available in it, before installing or executing it in your production environment.
- Although it’s easy to change the PowerShell execution policy on your servers to allow unsigned scripts to execute, it’s recommended to allow only signed scripts in production environments. You can easily sign a script if it's unsigned, before running it in a production environment.

So should I do it?

If using EWS to apply a Personal Tag to custom folders helps you meet your business requirements, absolutely! However, do note and consider the following:

You’re replicating some of the functionality available via Managed Folders, but it doesn’t turn the folder into a Managed Folder.
Remember - it’s a Personal Tag! Users can remove the tag from the folder using Outlook or Outlook Web App.
If you have additional Personal Tags available in your environment, users can change the tag on the custom folder.
Users can tag individual items with a different Personal Tag. There is no way to enforce inheritance of retention tag if Personal Tags have been provisioned and available to the user.
Users can rename or delete custom folders. Unlike Managed Folders, which are protected from changes or deletion by users, custom folders created by users or by admin are just like any other (non-default) folder in the mailbox.

Provisioning custom folders with different retention settings (by applying Personal Tags) may help you meet your organization’s retention requirements. As an IT Pro, make sure you understand the above and follow the best practices.

Bharat Suneja

Use Exchange Web Services and PowerShell to Discover and Remove Direct Booking Settings

The Exchange Team — Fri, 10 May 2013 00:45:00 GMT

Prior to Exchange 2007, there were two primary methods of implementing automated resource scheduling – Direct Booking and the AutoAccept Agent (a store event sink released as a web download for Exchange 2003). In Exchange 2007, we changed how automated resource scheduling is implemented. The AutoAccept Agent is no longer supported, and the Direct Booking method, technically an Outlook function, has been replaced with server-side calendar booking function called the Resource Booking Attendant.

Note There are various terms associated with this new Resource Booking function, such as: Calendar Processing, Automatic Resource Booking, Calendar Attendant Processing, Automated Processing and Resource Booking Assistant. We will be using the “Resource Booking Attendant” nomenclature for this article.

While the Direct Booking method for resource scheduling can indeed work on Exchange Server 2007/2010/2013, we strongly recommend that you disable Direct Booking for resource mailboxes and use the Resource Booking Attendant instead. Specifically, we are referring to the “AutoAccept” Automated Processing feature of the Resource Booking Attendant, which can be enabled for a mailbox after it has been migrated to Exchange 2007 or later and upgraded to a Resource Mailbox.

Note The published resource mailbox upgrade guidance on TechNet specifies to disable Direct Booking in the resource mailbox while still on Exchange 2003, move the mailbox, and then enable the AutoAccept functionality via the Resource Booking Attendant. This order of steps can introduce an unnecessary amount of time where the resource mailbox may be without automated scheduling capabilities.

We are currently working to update that guidance to reflect moving the mailbox first, and only then proceed with disabling the Direct Booking functionality, after which the AutoAccept functionality via the Resource Booking Attendant can be immediately enabled. This will shorten the duration where the mailbox is without automated resource scheduling capabilities.

This conversion process to resource mailboxes utilizing the Resource Booking Attendant is sometimes an honest oversight or even deliberately ignored when migrating away from Exchange 2003 due to Direct Booking’s ability to continue to work with newer versions of Exchange, even Exchange Online. This will often result in resource mailboxes (or even user mailboxes!) with Direct Booking functionality remaining in place long after Exchange 2003 is ancient history in the environment.

Why not just leave Direct Booking enabled?

There are issues that can arise from leaving Direct Booking enabled, from simple administrative burden scenarios all the way to major calendaring issues. Additionally, Resource Booking Attendant offers advantages over Direct Booking functionality:

Direct Booking capabilities, technically an Outlook function, has been deprecated from the product as of Outlook 2013. It was already on the deprecation list in Outlook 2010 and required a registry modification to reintroduce the functionality.
Direct Booking and Resource Booking Attendant are conflicting technologies, and if simultaneously enabled, unexpected behavior in calendar processing and item consistency can occur.
Outlook Web App (as well as any non-MAPI clients, like Exchange ActiveSync (EAS) devices) cannot use Direct Booking for automated resource scheduling. This is especially relevant for Outlook Web App-only environments where the users do not have Microsoft Outlook as a mail client.
The Resource Booking Attendant AutoAccept functionality is a server-side solution, eliminating the need for client-side logic in order to automatically process meeting requests.

How do I check which mailboxes have Direct Booking Enabled?

How does one validate if Direct Booking settings are enabled on mailboxes in the organization, especially if mailboxes had previously been hosted on Exchange 2003?

Figure 1: Checking Direct Booking settings in Microsoft Outlook 2010

Unfortunately, the manual steps involve assigning permissions to all mailboxes, creating MAPI profiles for each mailbox, logging into each mailbox, checking Tools > Options > Calendar > Resource Scheduling, note which of the three Direct Booking checkboxes are checked, click OK/Cancel a few times, log out of mailbox. Whew! That can be a major undertaking even for a small to midsize company that has more than a handful of mailboxes! Having staff perform this type of activity manually can be a costly and tedious endeavor. Once you have discovered which mailboxes have the Direct Booking settings enabled, you would then have to repeat this entire process to disable these settings unless you removed them at the time of discovery.

Having an automated method to discover, track, and even disable Direct Booking settings would be nice right?

Look no further, we have the solution for you!

Using Exchange Web Services (EWS) and PowerShell, we can automate the discovery of Direct Booking settings that are enabled, track the results, and even disable them! We wrote Remove-DirectBooking.ps1, a sample script, to do exactly that and even more to aid in automating this manual effort.

After you've downloaded it, rename the file and remove the .txt extension.

IMPORTANT The previously uploaded script had the last line truncated to Stop-Tran (instead of Stop-Transcript). We've uploaded an updated version to TechNet Gallery. If you downloaded the previous version of the script, please download the updated version. Alternatively, you can open the previously downloaded version in Notepad or other text editor and correct the last line to Stop-Transcript.

Let’s break down the major tasks the PowerShell script does:

Uses EWS Application Impersonation to tap into a mailbox (or set of mailboxes) and read the three MAPI properties where the Direct Booking settings are stored. It does this by accessing the localfreebusy item sitting in the NON_IPM_SUBTREE\FreeBusy Data folder, which resides in the root of the Information Store in the mailbox. The three MAPI properties and their equivalent Outlook settings the script looks at are:
- 0x686d Automatically accept meeting requests and remove canceled meetings
- 0x686f Automatically decline meeting requests that conflict with an existing appointment or meeting
- 0x686e Automatically decline recurring meeting requests
These three properties contain Boolean values mirroring the Resource Scheduling checkboxes found in Outlook (see Figure 1 above).
For mailboxes where Direct Booking settings were detected, it checks for conflicts by determining if the mailbox also has Resource Booking Attendant enabled with AutomateProcessing set to AutoAccept.
Optionally, disables any enabled Direct Booking settings encountered.
Note It is important to understand that by default the script runs in a read-only mode. Additional command line switches are available to run the script to disable Direct Booking settings.
Writes a detailed runtime processing log to console and log file.
Creates a simple output text file containing a list of mailboxes that can be later leveraged as an input file to feed the script for disabling the Direct Booking functionality.
Creates a CSV file containing statistics of the list of mailboxes processed with detailed information, such as what was discovered, any errors encountered, and optionally what was disabled. This is useful for performing analysis in the discovery phase and can also be used as another source to create an input file to feed into the script for disabling the Direct Booking functionality.

Example Scenarios

Here are a couple of example scenarios that illustrate how to use the script to discover and remove enabled Direct Booking settings.

Scenario 1

You've recently migrated from Exchange 2003 to Exchange 2010 and would like to disable Direct Booking for your company’s conference room mailboxes as well as any user mailboxes that may have Direct Booking settings enabled. The administrator’s logged in account has Application Impersonation rights and the View-Only Recipients RBACrole assigned.

On a machine that has the Exchange management tools & the Exchange Web Services API 1.2 or greater installed, open the Exchange Management Shell, navigate to the folder containing the script, and run the script using the following syntax:
.\Remove-DirectBooking.ps1 –identity * -UseDefaultCredentials
The script will process all mailboxes in the organization with detailed logging sent to the shell on the console. Note, depending the number of mailboxes in the org, this may take some time to complete
When the script completes, open the Remove-DirectBooking_<timestamp>.txtfile in Notepad, which will contain list of mailboxes that have Direct Booking enabled:

Figure 2: Output file containing list of mailboxes with Direct Booking enabled
After reviewing the list, rerun the script with the InputFile parameter and the RemoveDirectBookingswitch:
.\Remove-DirectBooking.ps1 –InputFile ‘.\Remove-DirectBooking_<timestamp>.txt’ –UseDefaultCredentials -RemoveDirectBooking
The script will process all the mailboxes listed in the input file with detailed logging sent to the shell on the console. Because you specified the RemoveDirectBooking switch, it does not run in read-only mode and disables all currently enabled Direct Booking settings encountered.
When the script completes, you can check the status of the removal operation by checking the Remove-DirectBooking_<timestamp>.csv file. A column called Direct Booking Removed? will record if the removal was successful. You can also check the runtime processing log file RemoveDirectBooking_<timestamp>.logas well.

Figure 3: Reviewing runtime log file in Excel (see larger screeshot))

Note The Direct Booking Removed? column now shows Yes where applicable, but the three Direct Booking settings columns still show their various values as “Yes”; this is because we record those three values pre-removal. If you were to run the script again in read-only mode against the same input file, those columns would reflect a value of N/A since there would no longer be any Direct Booking settings enabled. The Resource Room?, AutoAccept Enabled?, and Conflict Detected all have a value of N/A regardless because they are not relevant when disabling the Direct Booking settings.

Scenario 2

You're an administrator who's new to an organization. You know that they migrated from Exchange 2003 to Exchange 2007 in the distant past and are currently in the process of implementing Exchange 2013, having already migrated some users to Exchange 2013. You have no idea what resources mailboxes or even user mailboxes may be using Direct Booking and would like to discover who has what Direct Booking settings enabled. You would then like to selectively choose which mailboxes to pilot for Direct Booking removal before taking action on the majority of found mailboxes.

Here's how you would accomplish this using the Remove-DirectBooking.ps1 script:

Obtain a service account that has Application Impersonation rights for all mailboxes in the org.
Ensure service account has at least Exchange View-Only Administrator role (2007) and at least have an RBAC Role Assignment of View Only Recipients (2010/2013).
On a machine that has the Exchange management tools & the Exchange Web Services API 1.2 or greater installed, preferably an Exchange 2013 server, open the Exchange Management Shell, navigate to the folder containing the script, and run the script using the following syntax:
.\Remove-DirectBooking.ps1 –Identity *
The script will prompt you for the domain credentials of the account you wish to use because no credentials were specified. Enter the service account’s credentials.
The script will process all mailboxes in the organization with detailed logging sent to the shell on the console. Note, depending the number of mailboxes in the org, this may take some time to complete.
When the script completes, open the Remove-DirectBooking_<timestamp>.csvin Excel, which will looks something like:

Figure 4: Reviewing the Remove-DirectBooking_<timestamp>.csv in Excel (see larger screeshot))
Filter or sort the table by the Direct Booking Enabled? column. This will provide a list that can be scrutinized to determine which mailboxes are to be piloted with Direct Booking removal, such as those that have conflicts with already having the Resource Booking Attendant’s Automated Processing set to AutoAccept (which you can also filter on using the AutoAccept Enabled? column).
Once the list has been reviewed and the targeted mailboxes isolated, simply copy their email addresses into a text file (one address per line), save the text file, and use it as the input source for the running the script to disable the Direct Booking settings:
.\Remove-DirectBooking.ps1 –InputFile ‘.\’ -RemoveDirectBooking
As before, the script will prompt you for the domain credentials of the account you wish to use. Enter the service account’s credentials.
The script will process all the mailboxes listed in the input file with detailed logging sent to the shell on the console. It will disable all enabled Direct Booking settings encountered.
Use the same validation steps at the end of the previous example to verify the removal was successful.

Script Options and Caveats

Please see the script’s help section (via “get-help .\remove-DirectBooking.ps1 -full”) for full information on all the available parameters. Here are some additional options that may be useful in certain scenarios:

EWSURL switch parameter By default, the script will attempt to retrieve the EWS URL for each mailbox via AutoDiscover. This is preferred, especially in complex multi-datacenter or hybrid Exchange Online/On-premises environments where different EWS URLs may be in play for any given mailbox depending on where it resides in the org. However, there may be times where one would want to supply an EWS URL manually, such as when AutoDiscover is having “issues”, or the response time for AutoDiscover requests is introducing delays in overall script execution (think very large quantity of number of mailbox identities to churn through) and the EWS URL is the same across the org, etc. In these situations, one can use the EWSURL parameter to feed the script a static EWS URL.
UseDefaultCredentials If the current user is the service account or perhaps simply has both the Impersonation and the necessary Exchange Admin rights per the script’s requirements and they don’t wish to be prompted to type in a credential (another great example is scheduling the script to run as a job for instance), you can use the UseDefaultCredentials to run the script under that security context.
RemoveDirectBooking By default, the script runs in read-only mode. In order to make changes and disable Direct Booking settings on the mailbox, you mus specify the RemoveDirectBooking switch.

The script does have several prerequisites and caveats to ensure proper operation and meaningful results:

Application Impersonation rights and minimum Exchange Admin rights must be used
- For Exchange 2007, see Configuring Exchange Impersonation (Exchange Web Services)
- For Exchange 2010/2013, see Configuring Exchange Impersonation (uses RBAC)
Exchange Web Services Managed API 1.2 or later must be installed on the machine running the script
Exchange management tools must be installed on the machine running the script
Script must be executed from within the Exchange Management Shell
The Shell session must have the appropriate execution policy to allow the script to be executed (by default, you can't execute unsigned scripts).
AutoDiscover must be configured correctly (unless the EWS URL is entered manually)
Exchange 2003-based mailboxes cannot be targeted due to lack of EWS capabilities
In an Exchange 2010/2013 environment that also has Exchange 2007 mailboxes present, the script should be executed from a machine running Exchange 2010/2013 management tools due to changes in the cmdlets in those versions

Summary

The discovery and removal of Direct Booking settings can be a tedious and costly process to perform manually, but you can avoid and automate it using current functions and features via PowerShell and EWS in Microsoft Exchange Server 2007, 2010, & 2013. With careful use, the Remove-DirectBooking.ps1 script can be a valuable tool to aid Exchange administrators in maintaining automated resource scheduling capabilities in their Microsoft Exchange environments.

Your feedback and comments are welcome.

Thank you to Brian Day and Nino Bilic for their guidance in content review, and to our customers (you know who you are) for piloting the script.

Seth Brandes & Dan Smith

Troubleshoot your Exchange 2010 database backup functionality with VSSTester script

The Exchange Team — Mon, 29 Apr 2013 15:38:00 GMT

Download

Frequently in support, we encounter several backup related calls for Exchange 2010 databases. A sample of common issues we hear from our customers are:

“My backup software is not able to take a successful snapshot of the databases”
“My backups have been failing for quite a while. I have several thousand log files consuming disk space and I will eventually run out of disk space”
“My backup software indicates that the backup is successful but at the end of my backup, logs do not truncate”
“The Exchange Writer /VSS writer is not in a stable state (state is listed as ‘Retryable‘, ’Waiting for completion‘ or ’Failed’)”
“We suspect that the Volume Shadow Copy Service (VSS) is failing on the server and hence there are no successful backups”

It is critical to understand how backups and log truncation work in Exchange 2010. If you haven't already done so, check out our three-part blog series by Jesse Tedoff on backups and log truncation in Exchange 2010, Everything You Need to Know About Exchange Backups*.

When troubleshooting backups in Exchange 2010 we are interested in two writers – the Exchange Information Store Writer (utilized for active copy backups) and the Exchange Replica Writer (utilized for passive copy backups). The writers are responsible for providing the metadata information for databases to the VSS Requestor (aka the backup software). The VSS Provider is the component that creates and maintains shadow copies. At the end of successful backups, when the Volume Shadow Copy Service signals backup is complete, the writers initiate post-backup steps which include updating the database header and performing log truncation. (For more details, see Exchange VSS Writers on MSDN.)

As explained above, it is the responsibility of the VSS Requestor to get metadata information from Exchange writers and at the end of successful backup, VSS service signals backup complete to the Exchange writers so the writers can perform post-backup operations.

The purpose of this blog is to discuss the VSSTester script, its functionality and how it can help diagnose backup problems.

What does the script to?

The script has two major functions:

Perform Diskshadow backup of a selected Exchange database so we can exercise the VSS framework in the system, so at the end of a successful snapshot, database header is updated and log files are truncated. We will discuss in detail what Diskshadow is and what it does.
The second function of this script is to collect diagnostic data. For backup cases, there is a lot of data that needs to be collected. To get the diagnostic data you may have to manually go to different places in the Exchange server and turn on logging. If that is not done correctly, we will miss getting crucial logs during the time of the issue. The script makes the data collection process much easier.

Script requirements

The current version of the script works only on Exchange 2010 servers.
The script needs to be run on the Exchange server that is experiencing backup issues. If you are having issues with passive copy backups, please go to the appropriate node in the DAG and run the script. For example: You may have Database A having copies on Server1, Server2 and Server3. Server1 hosts the active copy of the database. If backups of the active copy have previously failed run the script on Server1. Otherwise run script on whichever of the remaining servers has failed previously when backing up the passive copy.
Please ensure that you have enough space on the drive you save the configuration and output files. Exchange and VSS traces, Diagnostic logs can occupy up to several GBs of drive space depending on the time taken for taking backup. For example: Running the script in a lab environment consumed close to 25MB of drive space a minute.
The script is unsigned. On the server where you run the script you will have to set the execution policy to allow unsigned PowerShell scripts. Please see this for how to do this.

The script can be run on any DAG configuration. You can use this to troubleshoot Mailbox and Public folder database backup issues. Databases and log files can be on regular drives or mount points. Mix and match of the two will also work!

Let us discuss in detail the two main functionalities of the script.

Diskshadow functionality and how the script uses it

What is Diskshadow and why do we utilize it in VSSTester script?

Diskshadow.exe is a command line tool built in to Windows Server 2008 operating system family as well as Windows Server 2012. Diskshadow is an in-box VSS requestor. It is utilized to test the functionality provided by the Volume Shadow Copy Service (VSS). For more details on Diskshadow please visit:

http://technet.microsoft.com/en-us/library/ee221016(v=ws.10).aspx

http://blogs.technet.com/b/josebda/archive/2007/11/30/diskshadow-the-new-in-box-vss-requester-in-windows-server-2008.aspx

The best part about Diskshadow is that it includes a script mode for automating tasks. This feature of Diskshadow is utilized in the VSSTester. The shadow copy done by Diskshadow is a snapshot of the entire volume at a given point in time. This copy is read-only.

More details on how a shadow copy is created, please visit the following link: http://technet.microsoft.com/en-us/library/ee923636(v=ws.10).aspx

During the course of the blog post, I will be mentioning the term “Diskshadow backup”. It is very important to understand that the term “backup” is relative here. Diskshadow uses the VSS service and gets the appropriate writer to be utilized for the snapshot. The writer will provide the metadata information of database /log files to the Diskshadow. After which Diskshadow utilizes the VSS Provider to create a shadow copy.

After a successful shadow copy /snapshot of databases and log files, the VSS Provider signals an end-backup to Exchange writers. To Exchange this looks like a full backup has been performed on the database. The key to understand here is NO data is actually transferred to a device, tape etc. This is only a test! You will see events in the application logs that usually show up when you take a regular backup, but NO data is actually backed up. Diskshadow has simply run all the backup APIs through the backup process without transferring any data.

The VSS Provider will take a snapshot of all the databases and logs (if present) on the volume. We will be doing a mirrored snapshot of the entire volume at the point in time when Diskshadow was run. Anything that is on the volume will be part of the snapshot. During the Diskshadow backup, we will be utilizing either the Information store writer (for active copy backup) or the Replica Writer (Passive copy backup) to provide the metadata information for the database.

When you use the VSSTester script, it prompts you for a database to be selected to perform the Diskshadow backup. When we take a snapshot of the volume all other databases (if present on the same drive) will be part of the snapshot, but post-backup operations will happen only on the selected database. This is because we will be utilizing either the Information store Writer (Active Copy Backup) or the Replica Writer (Passive copy backup) that is associated with the selected database. DB headers get updated based on VSS Requestor interaction with the Exchange writer that was utilized, which in turn leads to log truncation. Hence the header of the selected database will be only updated and logs will be purged (only for that the selected database) without being backed up.

When would you be interested in utilizing this Diskshadow functionality of the script?

You would be interested to utilize this functionality in almost all scenarios that I discussed at the start of this blog post. In addition to those scenarios another one that is not related to backups sometimes arises:

“I had an unexpected high transactional log growth issue in my exchange 2010 environment and now I am on the verge of losing all disk space in the logs directory. I do not have the time to perform a backup to truncate logs and my goal is to safely remove all the log files”

In the scenario mentioned above (and, by the way, if you have that problem, please go here), Exchange administrators would like to avoid causing a service outage by dismounting the database, removing log files and remounting the database. Another downside to manually removing the log files is breaking replication if the database has replicas across Database Availability Group members.

If you are willing to forgo a backup of the log files you can use the Diskshadow functionality of the script to trigger the backup APIs and tell Exchange to truncate the log files. The truncation commands will replicate to the other database copies and purge log files there as well. If successful, the net result is that the database will not go offline for lack of disk space on the log drive, but you will not have the security of retaining those log files for a future restore.

A sample run of the VSSTester script (with Diskshadow functionality)

Let me demonstrate the Diskshadow functionality of the script.

The Script can be downloaded from TechNet gallery here.

The script initializes and gives us the following options.

We select the option 1 to test backup using the built-in Diskshadow function.

If the path does not exist, the script will create the folder for you.

We gather the server name and verify it is an Exchange 2010 server. The script will check for the VSS writer status on the local machine. If we detect, any of the writers are not in a “Stable” state, the script will exit. You will need to restart the service associated with the writer to get the writers to a stable state (The Replication service for the Replica Writer or the Information Store service for the Exchange Writer).

The script then gets a list of databases present on the local server and displays the database name, if database is mounted or not and what is the server that holds the active copy of the database. You will have to select the number of the database.

Note: If the user does not provide an input, the script will automatically select the last database in the list.

In my case, I selected database mdb5. The number to enter would be 8.

The next important check is ensuring that the database’s replicas (if present) are healthy. If we detect that one of the copies is not healthy, the script will exit mentioning that the database copies need to be in healthy status before running the script.

The script next detects the location of the database file and log files. We create the Diskshadow configuration file on the fly every time a database is selected. This configuration file is also saved to the location you had specified earlier (in the example screenshots of this blog c:\vsstesterlogs) to save the configuration and output files. In this case the log files are in a mount point and the database file is on a regular volume. The script will add the appropriate volumes to the disk shadow file.

The script will then prompt you to provide the drive letters to expose the snapshots. A common question that arises is, do I need to initialize the drive before I specify a drive letter? The answer is no!

You will be specifying a drive letter that is currently not in use, so Diskshadow will create a virtual drive and expose the snapshot. Remember, the virtual drive that exposes the shadow copy is a read-only volume. The shadow copy is a read only copy .If the database and logs are in the same mount point / drive only, one drive letter is required to expose the snapshot, otherwise you will need to provide two different drive letters. One for exposing database snapshot and another for log files.

When you select the option to perform the Diskshadow backup, the script will automatically collect Diagnostic logs, ExTRA traces and VSS traces. Also verbose logging is turned on for Diskshadow. Whatever activity the script does is also logged in to transcript log and saved in the output files directory (c:\vsstesterlogs in this example).

Note: If you are performing a passive copy backup, ExTRA tracing will also be turned on in the active node. At the end of the script, we turn off ExTRA tracing in the active node and it will be automatically moved to the passive node. The active node ETL will be placed in the logs folder you had specified at the start of the script. .

Now, the main Diskshadow function will execute.

In the screenshots below we have excluded all other writers on the system that are associated with all other databases on the node (that are mounted or be replicas) and we are ONLY utilizing the writer associated with the selected database. This node hosts the passive copy of the database MDB5. Hence, the writer utilized will be associated with the Replication service aka the Microsoft Exchange Replica Writer.

(please click on above two screenshots to see them)

From the screen shot below, you can see that VSS Provider has taken a successful snapshot of the database and signaled end backup to the replica writer.

Now that we performed a successful snapshot of the database and log files, all the logging that was turned on will be turned off. The log files will be consolidated in the logs folder that you specified earlier at the start of the script. The script checks the VSS writer status after the backup is complete.

When the snapshot operation is complete, you will be prompted for an option to either remove the snapshot or leave the snapshots exposed in Windows Explorer.

(click to view)

I selected the option to remove the snapshot; hence we will be invoking Diskshadow again to delete the snapshot created earlier.

Let us discuss in detail exposing and removing snapshot functionality:

Remove snapshots - The snapshots that were taken earlier (database or log files) will be exposed in the Windows explorer, if the snapshot operation was successful. In this script we expose the snapshots as a drive letter (that you had specified earlier). If you do not want to have a copy of the log files, you may chose this option and the snapshot will be deleted. All the logs that got purged after post-backup will be present in this read only volume and when this volume is removed they will be deleted forever.
Expose Snapshots – You may choose to have the snapshots exposed. Later, if you want to delete the snapshot, please do the following

Open Command prompt
Type in Diskshadow
Delete shadows exposed <volume>

Note: It is highly recommended to take a full backup of the database using your regular backup software after utilizing Diskshadow.

After this, the script collects the application and system logs. The script filters them to cover only the period you started the script to the present. The transcript log is also stopped. The logs will be saved as a text file and saved in the output folder you had specified earlier (c:\vsstesterlogs in this example).

The most reliable method to verify log truncation takes place is to get the log sequence before and after the backup. Hence, before running the script I ran eseutil/ml ENN (the log generation prefix associated with database).

Post-backup, when I ran the same command, and can see:

We can clearly see a difference in the start of the sequence, meaning log truncation has occurred for the database. One more verification that can be done is to check the database header. We can see that the database header got updated to the most recent time, where Diskshadow was run.

I ran the script; what have I accomplished?

If the script finished successfully:

We were able to successfully test and exercise the underlying VSS framework in the server. Volume Shadow Copy service was able to successfully identify and utilize the Exchange writers in the box
The Exchange writers are able to provide the metadata information to the VSS Requestor (Diskshadow)
VSS Provider was able to successfully create a snapshot /shadow copy
VSS successfully signaled the Exchange writers on backup complete
The Exchange writers were able to perform the post snapshot operations which included log truncation.

Let us now look in to the other major functionality of the script.

Enable logging to troubleshoot backup issues

Use this if you do not want to test backup using Diskshadow and you just want to collect diagnostic logs for troubleshooting backup issues.

You may collect the diagnostic logs and have them handy before calling Microsoft Support saving a lot of time in the support incident because you can provide the files at the beginning of the case.

This time we will be selecting option 2 to enable logging.

Selecting this option does the majority of the things that the script did earlier, EXCEPT Diskshadow of course!

After checking the writer status, you can select the database to backup. We will be enabling all the logging like before (Diagnostic Logging, ExTRA, VSS tracing). Remember that, even though you would still be selecting one database - diagnostic logging, ExTRA tracing, VSS tracing are not database specific and are turned on at the server level. When you are utilizing the script to troubleshoot backup issues you can select any one database on the server and it will turn on appropriate logging on the server.

After the logging is turned on and traces enabled, you will see:

(click to view)

Now you will need to start your regular backup. After the backup completes/fails, you will need to come back to the PowerShell window where you are running the script and use the “ENTER” key to terminate the data collection. The script then disables diagnostic logging and tracing that was turned up earlier. If needed it will copy diagnostic logs from the active node for that database copy as well.

The script will again check for writer status after the backup then collect the application and system logs. It will stop the transcript log as well.

At this point, in order to troubleshoot the issue, you can open a case with Microsoft Support and upload the logs.

I hope this script helps you in better understanding the core concepts in Exchange 2010 backups, thus helping you troubleshoot backup issues! You can utilize Diskshadow to test Volume Shadow Copy Service and also check if the Exchange writers are performing as intended. If Diskshadow completes successfully without any error and you are still experiencing issues with backup software, you may need to contact the backup vendor to further troubleshoot the issue.

Your feedback and comments are most welcome.

Special thanks to Michael Barta for his contribution to the script, Theo Browning and Jesse Tedoff for reviewing the content.

Muralidharan Natarajan

Exchange 2010 Database Availability Groups and Disk Sector Sizes

The Exchange Team — Wed, 24 Apr 2013 22:00:00 GMT

These days, some customers are deploying Exchange databases and log files on advanced format (4K) drives. Although these drives support a physical sector size of 4096, many vendors are emulating 512 byte sectors in order to maintain backwards compatibility with application and operating systems. This is known as 512 byte emulation (512e). Windows 2008 and Windows 2008 R2 support native 512 byte and 512 byte emulated advanced format drives. Windows 2012 supports drives of all sector sizes. The sector size presented to applications and the operating system, and how applications respond, directly affects data integrity and performance.

For more information on sector sizes see the following links:

Understanding the Impact of Large Sector Media for IT Pros
Advanced format (4K) disk compatibility update
KB 2510009 Microsoft support policy for 4K sector hard drives in Windows

When deploying an Exchange 2010 Database Availability Group (DAG), the sector sizes of the volumes hosting the databases and log files must be the same across all nodes within the DAG. This requirement is outlined in Understanding Storage Configuration.

Support requires that all copies of a database reside on the same physical disk type. For example, it is not a supported configuration to host one copy of a given database on a 512-byte sector disk and another copy of that same database on a 512e disk. Also be aware that 4-kilobyte (KB) sector disks are not supported for any version of Microsoft Exchange and 512e disks are not supported for any version of Exchange prior to Exchange Server 2010 SP1.

Recently, we have noted that some customers have experienced issues with log file replication and replay as the result of sector size mismatch. These issues occur when:

Storage drivers are upgraded resulting in the recognized sector size changing.
Storage firmware is upgraded resulting in the recognized sector size changing.
New storage is presented or existing storage is replaced with drives of a different sector size.

This mismatch can cause one or more database copies in a DAG to fail, as illustrated below. In my example environment, I have a three-member DAG with a single database that resides on a volume labeled Z that is replicated between each member.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 1 3/19/2013 10:27:50 AM Healthy
SectorTest\MBX-3 Healthy 0 1 3/19/2013 10:27:50 AM Healthy

If I use FSUTIL to query the Z volume on each DAG member, we can see that the volume currently has 512 logical bytes per sector and a 512 physical bytes per sector. Thus, the the volume is currently seen by the operating system as having a native 512 byte sector size.

On MBX-1:

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0x18d0bc1dd0bbfed6
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fb842c
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        EF486117-9094-11E2-BF55-00155D006BA1

On MBX-3:

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0x0ad44aafd44a9d37
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fad281
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0000
Mft Zone End   :                  0x00000000000cc820
RM Identifier:        B9B00E32-90B2-11E2-94E9-00155D006BA3

Effects of storage changes

But what happens if there is a change in the way storage is seen on MBX-3, so that the volume now reflects a 512e sector size. This can happen when upgrading storage drivers, upgrading firmware, or presenting new storage that implements advanced format storage.

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0x0ad44aafd44a9d37
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fad2e7
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       4096
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        B9B00E32-90B2-11E2-94E9-00155D006BA3

When reviewing the database copy status, notice that the copy assigned to MBX-3 has failed.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/19/2013 11:13:05 AM Healthy
SectorTest\MBX-3 Failed 0 8 3/19/2013 11:13:05 AM Healthy

The full details of the copy status of MBX-3 can be reviewed to display the detailed error:

[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\MBX-3 | fl

RunspaceId                       : 5f4bb58b-39fb-4e3e-b001-f8445890f80a
Identity                         : SectorTest\MBX-3
Name                             : SectorTest\MBX-3
DatabaseName                     : SectorTest
Status                           : Failed
MailboxServer                    : MBX-3
ActiveDatabaseCopy               : mbx-1
ActivationSuspended              : False
ActionInitiator                  : Service
ErrorMessage                     : The log copier was unable to continue processing for database 'SectorTest\MBX-3' because an error occurred on the target server: Continuous replication - block mode has been terminated. Error: the log file sector size does not match the current volume's sector size (-546) [HResult: 0x80131500]. The copier will automatically retry after a short delay.
ErrorEventId                     : 2152
ExtendedErrorInfo                :
SuspendComment                   :
SinglePageRestore                : 0
ContentIndexState                : Healthy
ContentIndexErrorMessage         :
CopyQueueLength                  : 0
ReplayQueueLength                : 7
LatestAvailableLogTime           : 3/19/2013 11:13:05 AM
LastCopyNotificationedLogTime    : 3/19/2013 11:13:05 AM
LastCopiedLogTime                : 3/19/2013 11:13:05 AM
LastInspectedLogTime             : 3/19/2013 11:13:05 AM
LastReplayedLogTime              : 3/19/2013 10:24:24 AM
LastLogGenerated                 : 53
LastLogCopyNotified              : 53
LastLogCopied                    : 53
LastLogInspected                 : 53
LastLogReplayed                  : 46
LogsReplayedSinceInstanceStart   : 0
LogsCopiedSinceInstanceStart     : 0
LatestFullBackupTime             :
LatestIncrementalBackupTime      :
LatestDifferentialBackupTime     :
LatestCopyBackupTime             :
SnapshotBackup                   :
SnapshotLatestFullBackup         :
SnapshotLatestIncrementalBackup :
SnapshotLatestDifferentialBackup :
SnapshotLatestCopyBackup         :
LogReplayQueueIncreasing         : False
LogCopyQueueIncreasing           : False
OutstandingDumpsterRequests      : {}
OutgoingConnections              :
IncomingLogCopyingNetwork        :
SeedingNetwork                   :
ActiveCopy                       : False

Using the Exchange Server Error Code Look-up tool (ERR.EXE), we can verify the definition of the error code –546.

D:\Utilities\ERR>err -546

# for decimal -546 / hex 0xfffffdde
JET_errLogSectorSizeMismatch esent98.h
# /* the log file sector size does not match the current
# volume's sector size */
# 1 matches found for "-546"

In addition, the Application event log may contain the following entries:

Log Name:      Application
Source:        MSExchangeRepl
Date:          3/19/2013 11:14:58 AM
Event ID:      2152
Task Category: Service
Level:         Error
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The log copier was unable to continue processing for database 'SectorTest\MBX-3' because an error occured on the target server: Continuous replication - block mode has been terminated. Error: the log file sector size does not match the current volume's sector size (-546) [HResult: 0x80131500]. The copier will automatically retry after a short delay.

The cause

Why does this issue occur?
Each log file records in the header the sector size of the disk where a log file was created. For example, this is the header of a log file on MBX-1 with a native 512 byte sector size:

Z:\SectorTest>eseutil /ml E0100000001.log

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
      Base name: E01
      Log file: E0100000001.log
      lGeneration: 1 (0x1)
      Checkpoint: (0x38,FFFF,FFFF)
      creation time: 03/19/2013 09:40:14
      prev gen time: 00/00/1900 00:00:00
      Format LGVersion: (7.3704.16.2)
      Engine LGVersion: (7.3704.16.2)
      Signature: Create time:03/19/2013 09:40:14 Rand:11019164 Computer:
      Env SystemPath: z:\SectorTest\
      Env LogFilePath: z:\SectorTest\
     Env Log Sec size: 512 (matches)
      Env (CircLog,Session,Opentbl,VerPage,Cursors,LogBufs,LogFile,Buffers)
          (    off,   1227, 61350, 16384, 61350,   2048,   2048, 44204)
      Using Reserved Log File: false
      Circular Logging Flag (current file): off
      Circular Logging Flag (past files): off
      Checkpoint at log creation time: (0x1,8,0)
      Last Lgpos: (0x1,A,0)
Number of database page references: 0
Integrity check passed for log file: E0100000001.log
Operation completed successfully in 0.62 seconds.

The sector size that is chosen is determined through one of two methods:

If the log stream is brand new, read the sector size from disk and utilize this sector size.
If the log stream already exists, use the sector size of the given log stream.

In theory, since the sector size of disks should not be changing across nodes and the sector size of all disks must match, this should not cause a problem. In our example, and in some customer environments, these sector sizes are actually changing. Since most of these databases already exist, the existing sector size of the log stream is utilized, which in turn causes a mismatch between DAG members.

When a mismatch occurs, the issue only prevents the successful use of block mode replication. It does not affect file mode replication. Block mode replication was introduced in Exchange 2010 Service Pack 1. For more information on block mode replication, see New High Availability and Site Resilience Functionality in Exchange 2010 SP1.

Why does this only affect block mode replication?
When a log file is addressed we reference locations within a log file based off a log file position. The log file position is a combination of the log generation, the sector, and offset within that sector. For example, in the previous header dump you can see the “Last LGPOS” is (0x1,A,0) – this just happens to be the last log file position within the log. Let us say we were creating a block for block mode replication within a log file generation 0x1A, sector 8, offset 1 – this would be reflected as an LGPOS of (0x1a,8,1). When this block is transmitted to a host with an advanced sector size disk, the log position would actually have to be translated. On an advanced format disk this same log position would be (0x1a,1,1). As you can see, it could create significant problems if incorrect positions within a log file were written to or read from.

The resolution

How do I go about correcting this condition?
To fix this condition, first ensure that the same sector sizes exist on all disks across all nodes that host Exchange data, and then reset the log stream.

The following steps can show you how to do this with minimal downtime.

Ensure that Exchange 2010 Service Pack 2 or later is installed on all DAG members.

Note: Exchange 2010 Service Pack 1 and earlier do not support 512e volumes).
Disable block mode replication on all hosts. This step requires restarting the replication service on each node. This will temporarily cause all copies to fail on passive nodes when the service is restarted on the active node. When the service is restarted on the passive node only passive copies on that node will enter a failed state. Databases that are mounted and client connections are not impacted by this activity. Block mode replication should remain disabled until all steps have been completed on all DAG members.
1. Launch registry editor.
2. Navigate to HKLM\Software\Microsoft\ExchangeServer\V14\Replay\Parameters
3. Right click in the parameters key and select New –> DWORD
4. The name for the DWORD is DisableGranularReplication
5. The value for the DWORD is 1
Restart the Microsoft Exchange Replication service on each member using the Shell: Restart-Service MSExchangeRepl
Validate that all copies of databases across DAG members are healthy at this time:

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/19/2013 12:28:34 PM Healthy
SectorTest\MBX-3 Healthy 0 0 3/19/2013 12:28:34 PM Healthy
Apply the appropriate hotfix for Windows Server 2008 or Windows Server 2008 R2 and Advanced Format Disks. Windows Server 2012 does not require a hotfix.
- Windows 2008 R2: KB 982018 An update that improves the compatibility of Windows 7 and Windows Server 2008 R2 with Advanced Format Disks is available
- Windows 2008: KB 2553708 A hotfix rollup that improves Windows Vista and Windows Server 2008 compatibility with Advanced Format disks
Repeat the procedure that caused the disk sector size to change. For example, if the issue arose as a result of upgrading drivers and firmware on a host utilize your maintenance mode procedures to complete the driver and firmware upgrade on all hosts.

Note: If your installation does not allow for you to use the same sector sizes across all DAG members, then the implementation is not supported.
Utilize FSUTIL to ensure that the sector sizes match across all hosts for the log and database volumes.

On MBX-1:

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0x18d0bc1dd0bbfed6
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fac6e6
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       4096
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        EF486117-9094-11E2-BF55-00155D006BA1

On MBX-2

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0xfa6a794c6a790723
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fac86f
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       4096
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        5F18A2FC-909E-11E2-8599-00155D006BA2

On MBX-3

C:\>fsutil fsinfo ntfsinfo z:

NTFS Volume Serial Number :       0x0ad44aafd44a9d37
Version :                         3.1
Number Sectors :                  0x000000000fdfe7ff
Total Clusters :                  0x0000000001fbfcff
Free Clusters :                  0x0000000001fabfd6
Total Reserved :                  0x0000000000000000
Bytes Per Sector :               512
Bytes Per Physical Sector :       4096
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        B9B00E32-90B2-11E2-94E9-00155D006BA3

At this point, the DAG should be stable, and replication should be occurring as expected between databases using file mode. In order to restore block mode replication and fully recognize the new disk sector sizes, the log stream must be reset.

IMPORTANT: Please note the following about resetting the log stream:

The log stream must be fully reset on all database copies.
All lagged database copies must be replayed to current log.
If backups are utilized as a recovery method this will introduce a gap in the log file sequence preventing a full roll forward recovery from the last backup point.

You can use the following steps to reset the log stream:

Validate the existence of a replay queue:

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/19/2013 1:34:37 PM Healthy
SectorTest\MBX-3 Healthy 0 138 3/19/2013 1:34:37 PM Healthy
Set the replay and truncation lag times values to 0 on all database copies. This will ensure that logs replay to current while allowing the databases to remain online. In this example, MBX-3 is a lagged copy database. When the configuration change is detected, log replay will occur allowing the lagged copy to eventually catch up. Note that depending on the replay lag time, this could take several hours before proceeding to next steps.

[PS] C:\>Set-MailboxDatabaseCopy SectorTest\MBX-3 -ReplayLagTime 0.0:0:0 -TruncationLagTime 0.0:0:0

Validate that the replay queue has caught up and is near zero.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/19/2013 1:34:37 PM Healthy
SectorTest\MBX-3 Healthy 0 0 3/19/2013 1:34:37 PM Healthy
Dismount the database.

CAUTION: Dismounting the database will cause a client interruption, which will continue until the database is mounted.

[PS] C:\>Dismount-Database SectorTest

Confirm
Are you sure you want to perform this action?
Dismounting database "SectorTest". This may result in reduced availability for mailboxes in the database.
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): y
[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\*
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Dismounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/25/2013 5:41:54 AM Healthy
SectorTest\MBX-3 Healthy 0 0 3/25/2013 5:41:54 AM Healthy
On each DAG member hosting a database copy, open a command prompt and navigate to the log file directory. Execute eseutil /r ENN to perform a soft recovery. This step is necessary to ensure that all log files are played into all copies.

Z:\SectorTest>eseutil /r e01

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating RECOVERY mode...
    Logfile base name: e01
            Log files: <current directory>
         System files: <current directory>
Performing soft recovery...
                      Restore Status (% complete)
          0    10   20   30   40   50   60   70   80   90 100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................
Operation completed successfully in 0.203 seconds.
On each DAG member hosting a database copy open a command prompt and navigate to the database directory. Execute eseutil /mh <EDB> against the database to dump the header. You must validate that the following information is correct on all database copies:
- All copies of the database show in clean shutdown.
- All copies of the database show the same last detach information.
- All copies of the database show the same last consistent information.
Here is example output of a full /mh dump followed by a comparison of the data across our three sample copies.

Z:\SectorTest>eseutil /mh SectorTest.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
         Database: SectorTest.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x010f4400
Actual Checksum: 0x010f4400
Fields:
        File Type: Database
         Checksum: 0x10f4400
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,17
Engine ulVersion: 0x620,17
Created ulVersion: 0x620,17
     DB Signature: Create time:03/19/2013 09:40:15 Rand:11009066 Computer:
         cbDbPage: 32768
           dbtime: 601018 (0x92bba)
State: Clean Shutdown
     Log Required: 0-0 (0x0-0x0)
    Log Committed: 0-0 (0x0-0x0)
   Log Recovering: 0 (0x0)
GenMax Creation: 00/00/1900 00:00:00
         Shadowed: Yes
       Last Objid: 3350
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
Old Repair Count: 0
Last Consistent: (0x138,3FB,1A4) 03/19/2013 13:44:11
      Last Attach: (0x111,9,86) 03/19/2013 13:42:29
Last Detach: (0x138,3FB,1A4) 03/19/2013 13:44:11
             Dbid: 1
    Log Signature: Create time:03/19/2013 09:40:14 Rand:11019164 Computer:
       OS Version: (6.1.7601 SP 1 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

Last checksum finish Date: 03/19/2013 13:11:36
Current checksum start Date: 00/00/1900 00:00:00
      Current checksum page: 0

Operation completed successfully in 0.47 seconds.

MBX-1:

State: Clean Shutdown
Last Consistent: (0x138,3FB,1A4) 03/19/2013 13:44:11
Last Detach: (0x138,3FB,1A4) 03/19/2013 13:44:11

MBX-2:

State: Clean Shutdown
Last Consistent: (0x138,3FB,1A4) 03/19/2013 13:44:12
Last Detach: (0x138,3FB,1A4) 03/19/2013 13:44:12

MBX-3:

State: Clean Shutdown
Last Consistent: (0x138,3FB,1A4) 03/19/2013 13:44:13
Last Detach: (0x138,3FB,1A4) 03/19/2013 13:44:13

In this case, the values match across all copies so further steps can be performed.

If the values do not match across copies for any reason, do not continue and please contact Microsoft support.
Reset the log file generation for the database.

Note: Use Get-MailboxDatabaseCopyStatus to record database locations and status prior to performing this activity.

Locate the log file directory for each ACTIVE (DISMOUNTED) database. Remove all log files from this directory first. Failure to remove log files from the ACTIVE (DISMOUNTED) database may result in the Replication service recopying log files, a failure of this procedure, and subsequent need to reseed all database copies.

IMPORTANT: If log files are located in the same location as the database and catalog data folder, take precautions to not remove the database or the catalog data folder.

In our example MBX-1 hosts the ACTIVE (DISMOUNTED) copy.

[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\*

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Dismounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/25/2013 5:41:54 AM Healthy
SectorTest\MBX-3 Healthy 0 0 3/25/2013 5:41:54 AM Healthy

Locate the log file directory for each PASSIVE database. Remove all log files from this directory. Failure to remove all log files could result in this procedure failing, and the need to reseed this or all database copies. If log files are located in the same location as the database and catalog data folder take precautions to not remove the database or the catalog data folder.

In our example MBX-2 and MBX-3 host the passive database copies.

[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\*

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Dismounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/25/2013 5:41:54 AM Healthy
SectorTest\MBX-3 Healthy 0 0 3/25/2013 5:41:54 AM Healthy
Mount the database using Mount-Database <DBNAME>, and verify it has mounted.

[PS] C:\>Mount-Database SectorTest
[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 1 3/25/2013 5:57:28 AM Healthy
SectorTest\MBX-3 Healthy 0 1 3/25/2013 5:57:28 AM Healthy
Suspend and resume all passive database copies.

Note: The error on suspending the active database copy is expected.

[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\* | Suspend-MailboxDatabaseCopy

The suspend operation can't proceed because database 'SectorTest' on Exchange Mailbox server 'MBX-1' is the active mailbox database copy.
    + CategoryInfo          : InvalidOperation: (SectorTest\MBX-1:DatabaseCopyIdParameter) [Suspend-MailboxDatabaseCopy], InvalidOperationException
    + FullyQualifiedErrorId : 5083D28B,Microsoft.Exchange.Management.SystemConfigurationTasks.SuspendDatabaseCopy
    + PSComputerName        : mbx-1.exchange.msft

Note: The error on resuming the active database copy is expected.

[PS] C:\>Get-MailboxDatabaseCopyStatus SectorTest\* | Resume-MailboxDatabaseCopy

WARNING: The Resume operation won't have an effect on database replication because database 'SectorTest' hosted on server 'MBX-1' is the active mailbox database.
Validate replication health.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Mounted 0 0 Healthy
SectorTest\MBX-2 Healthy 0 0 3/19/2013 1:56:12 PM Healthy
SectorTest\MBX-3 Healthy 0 0 3/19/2013 1:56:12 PM Healthy
Using Set-MailboxDatabaseCopy, reconfigure any replay lag or truncation lag time on the database copy. This example implements a 7 day replay lag time.

set-mailboxdatabasecopy –identity SectorTest\MBX-3 –replayLagTime 7.0:0:0
Repeat the previous steps for all databases in the DAG including those databases that have a single copy.

IMPORTANT: DO NOT proceed to the next step until all databases have been reset.
Enable block mode replication. Using registry editor navigate to HKLM \Software\Microsoft\ExchangeServer \V14 \Replay, and then remove the DisableGranularReplication DWORD value.
Restart the replication service on each DAG member.

Restart-Service MSExchangeREPL
Validate database health using Get-MailboxDatabaseCopyStatus.

[PS] C:\>Get-MailboxDatabaseCopyStatus *

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State
---- ------ --------- ----------- -------------------- ------------
SectorTest\MBX-1 Healthy 0 0 3/19/2013 2:25:56 PM Healthy
SectorTest\MBX-2 Mounted 0 0 Healthy
SectorTest\MBX-3 Healthy 0 230 3/19/2013 2:25:56 PM Healthy
Dump the header of a log file and verify that the new sector size is reflected in the log file stream. To do this, open a command prompt and navigate to the log file directory for the database on the active node. Run eseutil /ml against any log within the directory, and verify that the sector size reflects 4096 and (matches).

Z:\SectorTest>eseutil /ml E0100000001.log

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
      Base name: E01
      Log file: E0100000001.log
      lGeneration: 1 (0x1)
      Checkpoint: (0x17B,FFFF,FFFF)
      creation time: 03/19/2013 13:56:11
      prev gen time: 00/00/1900 00:00:00
      Format LGVersion: (7.3704.16.2)
      Engine LGVersion: (7.3704.16.2)
      Signature: Create time:03/19/2013 13:56:11 Rand:2996669 Computer:
      Env SystemPath: z:\SectorTest\
      Env LogFilePath: z:\SectorTest\
     Env Log Sec size: 4096 (matches)
      Env (CircLog,Session,Opentbl,VerPage,Cursors,LogBufs,LogFile,Buffers)
          (    off,   1227, 61350, 16384, 61350,   2048,    256, 44204)
      Using Reserved Log File: false
      Circular Logging Flag (current file): off
      Circular Logging Flag (past files): off
      Checkpoint at log creation time: (0x1,1,0)
      Last Lgpos: (0x1,2,0)
Number of database page references: 0
Integrity check passed for log file: E0100000001.log
Operation completed successfully in 0.250 seconds.

If the above steps have been completed successfully, and the log file sequence recognizes a 4096 sector size, then this issue has been resolved.

This guidance was validated in the following configurations:

Windows 2008 R2 Enterprise with Exchange 2010 Service Pack 2
Windows 2008 R2 Enterprise with Exchange 2010 Service Pack 3
Windows 2008 SP2 Enterprise with Exchange 2010 Service Pack 3
Windows 2012 Datacenter with Exchange 2010 Service Pack 3

Tim McMichael

Troubleshooting Rapid Growth in Databases and Transaction Log Files in Exchange Server 2007 and 2010

The Exchange Team — Thu, 18 Apr 2013 17:58:45 GMT

A few years back, a very detailed blog post was released on Troubleshooting Exchange 2007 Store Log/Database growth issues.

We wanted to revisit this topic with Exchange 2010 in mind. While the troubleshooting steps needed are virtually the same, we thought it would be useful to condense the steps a bit, make a few updates and provide links to a few newer KB articles.

The below list of steps is a walkthrough of an approach that would likely be used when calling Microsoft Support for assistance with this issue. It also provides some insight as to what we are looking for and why. It is not a complete list of every possible troubleshooting step, as some causes are simply not seen quite as much as others.

Another thing to note is that the steps are commonly used when we are seeing “rapid” growth, or unexpected growth in the database file on disk, or the amount of transaction logs getting generated. An example of this is when an Administrator notes a transaction log file drive is close to running out of space, but had several GB free the day before. When looking through historical records kept, the Administrator notes that approx. 2 to 3 GBs of logs have been backed up daily for several months, but we are currently generating 2 to 3 GBs of logs per hour. This is obviously a red flag for the log creation rate. Same principle applies with the database in scenarios where the rapid log growth is associated to new content creation.

In other cases, the database size or transaction log file quantity may increase, but signal other indicators of things going on with the server. For example, if backups have been failing for a few days and the log files are not getting purged, the log file disk will start to fill up and appear to have more logs than usual. In this example, the cause wouldn’t necessarily be rapid log growth, but an indicator that the backups which are responsible for purging the logs are failing and must be resolved. Another example is with the database, where retention settings have been modified or online maintenance has not been completing, therefore, the database will begin to grow on disk and eat up free space. These scenarios and a few others are also discussed in the “Proactive monitoring and mitigation efforts” section of the previously published blog.

It should be noted that in some cases, you may run into a scenario where the database size is expanding rapidly, but you do not experience log growth at a rapid rate. (As with new content creation in rapid log growth, we would expect the database to grow at a rapid rate with the transaction logs.) This is often referred to as database “bloat” or database “space leak”. The steps to troubleshoot this specific issue can be a little more invasive as you can see in some analysis steps listed here (taking databases offline, various kinds of dumps, etc.), and it may be better to utilize support for assistance if a reason for the growth cannot be found.

Once you have established that the rate of growth for the database and transaction log files is abnormal, we would begin troubleshooting the issue by doing the following steps. Note that in some cases the steps can be done out of order, but the below provides general suggested guidance based on our experiences in support.

Step 1

Use Exchange User Monitor (Exmon) server side to determine if a specific user is causing the log growth problems.

Sort on CPU (%) and look at the top 5 users that are consuming the most amount of CPU inside the Store process. Check the Log Bytes column to verify for this log growth for a potential user.
If that does not show a possible user, sort on the Log Bytes column to look for any possible users that could be attributing to the log growth

If it appears that the user in Exmon is a ?, then this is representative of a HUB/Transport related problem generating the logs. Query the message tracking logs using the Message Tracking Log tool in the Exchange Management Consoles Toolbox to check for any large messages that might be running through the system. See #15 for a PowerShell script to accomplish the same task.

Step 2

With Exchange 2007 Service Pack 2 Rollup Update 2 and higher, you can use KB972705 to troubleshoot abnormal database or log growth by adding the described registry values. The registry values will monitor RPC activity and log an event if the thresholds are exceeded, with details about the event and the user that caused it. (These registry values are not currently available in Exchange Server 2010)

Check for any excessive ExCDO warning events related to appointments in the application log on the server. (Examples are 8230 or 8264 events). If recurrence meeting events are found, then try to regenerate calendar data server side via a process called POOF. See http://blogs.msdn.com/stephen_griffin/archive/2007/02/21/poof-your-calender-really.aspx for more information on what this is.

Event Type: Warning
Event Source: EXCDO
Event Category: General
Event ID: 8230
Description: An inconsistency was detected in username@domain.com: /Calendar/<calendar item> .EML. The calendar is being repaired. If other errors occur with this calendar, please view the calendar using Microsoft Outlook Web Access. If a problem persists, please recreate the calendar or the containing mailbox.

Event Type: Warning
Event ID : 8264
Category : General
Source : EXCDO
Type : Warning
Message : The recurring appointment expansion in mailbox <someone's address> has taken too long. The free/busy information for this calendar may be inaccurate. This may be the result of many very old recurring appointments. To correct this, please remove them or change their start date to a more recent date.

Important: If 8230 events are consistently seen on an Exchange server, have the user delete/recreate that appointment to remove any corruption

Step 3

Collect and parse the IIS log files from the CAS servers used by the affected Mailbox Server. You can use Log Parser Studio to easily parse IIS log files. In here, you can look for repeated user account sync attempts and suspicious activity. For example, a user with an abnormally high number of sync attempts and errors would be a red flag. If a user is found and suspected to be a cause for the growth, you can follow the suggestions given in steps 5 and 6.

Once Log Parser Studio is launched, you will see convenient tabs to search per protocol:

Some example queries for this issue would be:

Step 4

If a suspected user is found via Exmon, the event logs, KB972705, or parsing the IIS log files, then do one of the following:

Disable MAPI access to the users mailbox using the following steps (Recommended):
- Run
  Set-Casmailbox –Identity <Username> –MapiEnabled $False
- Move the mailbox to another Mailbox Store. Note: This is necessary to disconnect the user from the store due to the Store Mailbox and DSAccess caches. Otherwise you could potentially be waiting for over 2 hours and 15 minutes for this setting to take effect. Moving the mailbox effectively kills the users MAPI session to the server and after the move, the users access to the store via a MAPI enabled client will be disabled.
Disable the users AD account temporarily
Kill their TCP connection with TCPView
Call the client to have them close Outlook or turn of their mobile device in the condition state for immediate relief.

Step 5

If closing the client/devices, or killing their sessions seems to stop the log growth issue, then we need to do the following to see if this is OST or Outlook profile related:

Have the user launch Outlook while holding down the control key which will prompt if you would like to run Outlook in safe mode. If launching Outlook in safe mode resolves the log growth issue, then concentrate on what add-ins could be attributing to this problem.

For a mobile device, consider a full resync or a new sync profile. Also check for any messages in the drafts folder or outbox on the device. A corrupted meeting or calendar entry is commonly found to be causing the issue with the device as well.

If you can gain access to the users machine, then do one of the following:

1. Launch Outlook to confirm the log file growth issue on the server.

2. If log growth is confirmed, do one of the following:

Check users Outbox for any messages.
- If user is running in Cached mode, set the Outlook client to Work Offline. Doing this will help stop the message being sent in the outbox and sometimes causes the message to NDR.
- If user is running in Online Mode, then try moving the message to another folder to prevent Outlook or the HUB server from processing the message.
- After each one of the steps above, check the Exchange server to see if log growth has ceased
Call Microsoft Product Support to enable debug logging of the Outlook client to determine possible root cause.

3. Follow the Running Process Explorer instructions in the below article to dump out dlls that are running within the Outlook Process. Name the file username.txt. This helps check for any 3rd party Outlook Add-ins that may be causing the excessive log growth.
970920 Using Process Explorer to List dlls Running Under the Outlook.exe Process
http://support.microsoft.com/kb/970920

4. Check the Sync Issues folder for any errors that might be occurring

Let’s attempt to narrow this down further to see if the problem is truly in the OST or something possibly Outlook Profile related:

Run ScanPST against the users OST file to check for possible corruption.
With the Outlook client shut down, rename the users OST file to something else and then launch Outlook to recreate a new OST file. If the problem does not occur, we know the problem is within the OST itself.

If renaming the OST causes the problem to recur again, then recreate the users profile to see if this might be profile related.

Step 6

Ask Questions:

Is the user using any type of devices besides a mobile device?
Question the end user if at all possible to understand what they might have been doing at the time the problem started occurring. It’s possible that a user imported a lot of data from a PST file which could cause log growth server side or there was some other erratic behavior that they were seeing based on a user action.

Step 7

Check to ensure File Level Antivirus exclusions are set correctly for both files and processes per http://technet.microsoft.com/en-us/library/bb332342(v=exchg.141).aspx

Step 8

If Exmon and the above methods do not provide the data that is necessary to get root cause, then collect a portion of Store transaction log files (100 would be a good start) during the problem period and parse them following the directions in http://blogs.msdn.com/scottos/archive/2007/11/07/remix-using-powershell-to-parse-ese-transaction-logs.aspx to look for possible patterns such as high pattern counts for IPM.Appointment. This will give you a high level overview if something is looping or a high rate of messages being sent. Note: This tool may or may not provide any benefit depending on the data that is stored in the log files, but sometimes will show data that is MIME encoded that will help with your investigation

Step 9

If nothing is found by parsing the transaction log files, we can check for a rogue, corrupted, and large message in transit:

1. Check current queues against all HUB Transport Servers for stuck or queued messages:

get-exchangeserver | where {$_.IsHubTransportServer -eq "true"} | Get-Queue | where {$_.Deliverytype –eq “MapiDelivery”} | Select-Object Identity, NextHopDomain, Status, MessageCount | export-csv HubQueues.csv

Review queues for any that are in retry or have a lot of messages queued:

Export out message sizes in MB in all Hub Transport queues to see if any large messages are being sent through the queues:

get-exchangeserver | where {$_.ishubtransportserver -eq "true"} | get-message –resultsize unlimited | Select-Object Identity,Subject,status,LastError,RetryCount,queue,@{Name="Message Size MB";expression={$_.size.toMB()}} | sort-object -property size –descending | export-csv HubMessages.csv

Export out message sizes in Bytes in all Hub Transport queues:

get-exchangeserver | where {$_.ishubtransportserver -eq "true"} | get-message –resultsize unlimited | Select-Object Identity,Subject,status,LastError,RetryCount,queue,size | sort-object -property size –descending | export-csv HubMessages.csv

2. Check Users Outbox for any large, looping, or stranded messages that might be affecting overall Log Growth.

get-mailbox -ResultSize Unlimited| Get-MailboxFolderStatistics -folderscope Outbox | Sort-Object Foldersize -Descending | select-object identity,name,foldertype,itemsinfolder,@{Name="FolderSize MB";expression={$_.folderSize.toMB()}} | export-csv OutboxItems.csv

Note: This does not get information for users that are running in cached mode.

Step 10

Utilize the MSExchangeIS Client\Jet Log Record Bytes/sec and MSExchangeIS Client\RPC Operations/sec Perfmon counters to see if there is a particular client protocol that may be generating excessive logs. If a particular protocol mechanism if found to be higher than other protocols for a sustained period of time, then possibly shut down the service hosting the protocol. For example, if Exchange Outlook Web Access is the protocol generating potential log growth, then stopping the World Wide Web Service (W3SVC) to confirm that log growth stops. If log growth stops, then collecting IIS logs from the CAS/MBX Exchange servers involved will help provide insight in to what action the user was performing that was causing this occur.

Step 11

Run the following command from the Management shell to export out current user operation rates:

To export to CSV File:

get-logonstatistics |select-object username,Windows2000account,identity,messagingoperationcount,otheroperationcount,
progressoperationcount,streamoperationcount,tableoperationcount,totaloperationcount | where {$_.totaloperationcount -gt 1000} | sort-object totaloperationcount -descending| export-csv LogonStats.csv

To view realtime data:

Key things to look for:

In the below example, the Administrator account was storming the testuser account with email.
You will notice that there are 2 users that are active here, one is the Administrator submitting all of the messages and then you will notice that the Windows2000Account references a HUB server referencing an Identity of testuser. The HUB server also has *no* UserName either, so that is a giveaway right there. This can give you a better understanding of what parties are involved in these high rates of operations

UserName : Administrator
Windows2000Account : DOMAIN\Administrator
Identity : /o=First Organization/ou=First Administrative Group/cn=Recipients/cn=Administrator
MessagingOperationCount : 1724
OtherOperationCount : 384
ProgressOperationCount : 0
StreamOperationCount : 0
TableOperationCount : 576
TotalOperationCount : 2684

UserName :
Windows2000Account : DOMAIN\E12-HUB$
Identity : /o= First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=testuser
MessagingOperationCount : 630
OtherOperationCount : 361
ProgressOperationCount : 0
StreamOperationCount : 0
TableOperationCount : 0
TotalOperationCount : 1091

Step 12

Enable Perfmon/Perfwiz logging on the server. Collect data through the problem times and then review for any irregular activities. You can reference Perfwiz for Exchange 2007/2010 data collection here http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange-2007-2010-performance-data-collection-script.aspx

Step 13

Run ExTRA (Exchange Troubleshooting Assistant) via the Toolbox in the Exchange Management Console to look for any possible Functions (via FCL Logging) that may be consuming Excessive times within the store process. This needs to be launched during the problem period. http://blogs.technet.com/mikelag/archive/2008/08/21/using-extra-to-find-long-running-transactions-inside-store.aspx shows how to use FCL logging only, but it would be best to include Perfmon, Exmon, and FCL logging via this tool to capture the most amount of data. The steps shown are valid for Exchange 2007 & Exchange 2010.

Step 14

Export out Message tracking log data from affected MBX server.

Method 1

Download the ExLogGrowthCollector script and place it on the MBX server that experienced the issue. Run ExLogGrowthCollector.ps1 from the Exchange Management Shell. Enter in the MBX server name that you would like to trace, the Start and End times and click on the Collect Logs button.

Note: What this script does is to export out all mail traffic to/from the specified mailbox server across all HUB servers between the times specified. This helps provide insight in to any large or looping messages that might have been sent that could have caused the log growth issue.

Method 2

Copy/Paste the following data in to notepad, save as msgtrackexport.ps1 and then run this on the affected Mailbox Server. Open in Excel for review. This is similar to the GUI version, but requires manual editing to get it to work.

#Export Tracking Log data from affected server specifying Start/End Times
Write-host "Script to export out Mailbox Tracking Log Information"
Write-Host "#####################################################"
Write-Host
$server = Read-Host "Enter Mailbox server Name"
$start = Read-host "Enter start date and time in the format of MM/DD/YYYY hh:mmAM"
$end = Read-host "Enter send date and time in the format of MM/DD/YYYY hh:mmPM"
$fqdn = $(get-exchangeserver $server).fqdn
Write-Host "Writing data out to csv file..... "
Get-ExchangeServer | where {$_.IsHubTransportServer -eq "True" -or $_.name -eq "$server"} | Get-MessageTrackingLog -ResultSize Unlimited -Start $start -End $end | where {$_.ServerHostname -eq $server -or $_.clienthostname -eq $server -or $_.clienthostname -eq $fqdn} | sort-object totalbytes -Descending | export-csv MsgTrack.csv -NoType
Write-Host "Completed!! You can now open the MsgTrack.csv file in Excel for review"

Method 3

You can also use the Process Tracking Log Tool at http://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspx to provide some very useful reports.

Step 15

Save off a copy of the application/system logs from the affected server and review them for any events that could attribute to this problem.

Step 16

Enable IIS extended logging for CAS and MB server roles to add the sc-bytes and cs-bytes fields to track large messages being sent via IIS protocols and to also track usage patterns (Additional Details).

Step 17

Get a process dump the store process during the time of the log growth. (Use this as a last measure once all prior activities have been exhausted and prior to calling Microsoft for assistance. These issues are sometimes intermittent, and the quicker you can obtain any data from the server, the better as this will help provide Microsoft with information on what the underlying cause might be.)

Download the latest version of Procdump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx and extract it to a directory on the Exchange server
Open the command prompt and change in to the directory which procdump was extracted in the previous step.
Type
procdump -mp -s 120 -n 2 store.exe d:\DebugData
This will dump the data to D:\DebugData. Change this to whatever directory has enough space to dump the entire store.exe process twice. Check Task Manager for the store.exe process and how much memory it is currently consuming for a rough estimate of the amount of space that is needed to dump the entire store dump process.
Important: If procdump is being run against a store that is on a clustered server, then you need to make sure that you set the Exchange Information Store resource to not affect the group. If the entire store dump cannot be written out in 300 seconds, the cluster service will kill the store service ruining any chances of collecting the appropriate data on the server.

Open a case with Microsoft Product Support Services to get this data looked at.

Most current related KB articles

2814847 - Rapid growth in transaction logs, CPU use, and memory consumption in Exchange Server 2010 when a user syncs a mailbox by using an iOS 6.1 or 6.1.1-based device

2621266 - An Exchange Server 2010 database store grows unexpectedly large

996191 - Troubleshooting Fast Growing Transaction Logs on Microsoft Exchange 2000 Server and Exchange Server 2003

Kevin Carker
(based on a blog post written by Mike Lagase)

Preserving Activation Blocks After Performing DAG Member Maintenance

The Exchange Team — Wed, 10 Apr 2013 18:20:00 GMT

In Exchange 2010, when a database availability group (DAG) member needs service, it can be placed into maintenance mode. Exchange 2010 includes scripts the StartDagServerMaintenance.ps1 and StopDagServerMaintenace.ps1 scripts to place/remove a DAG member from maintenance mode. For a summary of what these scripts do, see this post.

Within a DAG, it is not uncommon to have one or more databases or servers blocked from automatic activation by the system. Some customers configure entire servers to be blocked from activation, some block individual copies, and some do a combination of both, based on their business requirements. Administrators using the maintenance mode scripts will find their configured activation blocks reset to the unblocked. This behavior is not a problem with the scripts; in fact, the scripts are working as designed.

Here is an example of a database copy that has had activation suspended:

[PS] C:\>Get-MailboxDatabaseCopyStatus DAG-DB0\MBX-2 | fl name,activationSuspended

Name : DAG-DB0\MBX-2
ActivationSuspended : True

Here is an example of a server that has activation blocked:

[PS] C:\>Get-MailboxServer MBX-2 | fl name,databasecopyautoactivationpolicy

Name : MBX-2
DatabaseCopyAutoActivationPolicy : Blocked

When the administrator executes the stopDagServerMaintenance.ps1 script, these states are reset back to their defaults. Here is an example of the states post StopDagServerMaintenance.ps1:

[PS] C:\Program Files\Microsoft\Exchange Server\V14\Scripts>Get-MailboxDatabaseCopyStatus DAG-DB0\MBX-2 | fl name,activationSuspended

Name : DAG-DB0\MBX-2
ActivationSuspended : False

[PS] C:\Program Files\Microsoft\Exchange Server\V14\Scripts>Get-MailboxServer MBX-2 | fl name,databasecopyautoactivationpolicy

Name : MBX-2
DatabaseCopyAutoActivationPolicy : Unrestricted

Although the maintenance scripts behavior is by design, it can lead to undesirable scenarios, such as lagged database copies being activated. Of course, to eliminate these issues, an administrator can record database and server settings before and after maintenance and reconfigure any reset settings as needed.

To help with this, here is a sample script I created that records database and server activation settings into a CSV file which can then be used with the maintenance scripts to adjust the states automatically.

Download: MaintenanceWrapper.ps1

What the script does

When you run the script, it creates two CSV files (on the server you run it on) along with a transcript that contains the results of the command executed. The first CSV file contains all database copies assigned to the server and their activation suspension status. Here's an example:

#TYPE Selected.Microsoft.Exchange.Management.SystemConfigurationTasks.DatabaseCopyStatusEntry
"Name","ActivationSuspended"
"DAG-DB0\DAG-3","True"
"DAG-DB1\DAG-3","True"

The second CSV file contains the database copy auto activation policy of the server. For example:

#TYPE Selected.Microsoft.Exchange.Data.Directory.Management.MailboxServer
"Name","DatabaseCopyAutoActivationPolicy"
"DAG-3","Blocked"

Using maintenanceWrapper.ps1 to start and stop maintenance

Because this scipt is unsigned, you'll need to relax the execution policy on the server to allow for unsigned scripts.

IMPORTANT: Allowing unsigned PowerShell scripts to execute is a security risk. For details, see Running Windows PowerShell Scripts. If this option does not meet your organization's security policy, you can sign the script (requires a code-signing certificate).

This command sets the execution policy on a server to allow unsigned PowerShell scripts to execute:

Set-ExecutionPolicy UNSIGNED

You can use the maintenanceWrapper.ps1 script to start and stop maintenance procedure on a DAG member.

Use this command to start the maintenance procedure:

maintenanceWrapper.ps1 –server <SERVERNAME> –action START

The command creates the CSV files containing the original database states and then invokes the StartDagServerMaintenance.ps1 script to place the DAG member in maintenance mode.
After maintenance is completed, you can stop the maintenance procedure using this command:

maintenanceWrapper.ps1 –server <SERVERNAME> –action STOP

The command calls the StopDagServerMaintenance.ps1 script to remove the DAG member from maintenance mode and then resets the database and server activation states from the states recorded in the CSV file.

Give the script a try and see if it makes maintenance mode for activation-blocked servers and databases easier for you. I hope you find this useful, and I welcome any and all feedback.

*Special thanks to Scott Schnoll and Abram Jackson for reviewing this content and David Spooner for validating the script.

Tim McMichael

Sending common or canned responses from a shared or repository mailbox

The Exchange Team — Fri, 05 Apr 2013 22:12:16 GMT

As a Microsoft Premier Field Engineer (PFE), I assist companies with Lotus Notes and GroupWise migrations to Exchange/Outlook environments and have found that different applications act differently. One of the common questions is: Can Outlook/Exchange send canned responses from a common mailbox? The answer is yes, but it is done a little differently than other products.

While everyone here is familiar with the different tools and process in this article, I usually find that the ‘leap of faith’ to put them together in this combination is not always made. So here we go, easy steps put together to create an end result worthy of solving a situation that may apply to your environment.

The first question is, in the Exchange/Outlook realm, is it possible to have a common response from a single mailbox (sometimes referred to as a ‘repository’) from multiple people? Yes it is possible in Outlook, but several steps have to be setup for this to occur correctly and consistently.

Let’s say you have an ‘IT Helpdesk’ mailbox, that acts like a repository and you have several people accessing this resource throughout the day. The first step is to remind ourselves where ‘signatures’ are stored in Outlook. Most peoples’ first thought is within the Outlook profile. That’s not correct. The signatures are actually stored as .htm files on a user’s local machine:

By default on Vista\Windows 7\8:

C:\Users\%UserName%\AppData\Roaming\Microsoft\Signatures

By default on Windows XP:

C:\Documents and settings\%UserName%\Application Data\Microsoft\Signatures

Why do I mention %UserName%? That is the wild card entry for the Alias or AD User Account name of whoever is logged onto the computer. You’ll see later that we reference this again.

Another question is, ‘What can you do with an HTML file?’ And of course the better question is, ‘What CAN’T you do with an HTML file?’ The beauty of having these files on the computer is that we can act upon them at any time and Outlook will use the current file. In fact, even with Outlook open, you could add a new file in this location and it will show up in the available signature list, which is one of the more dynamic options available in Outlook. Within this HTML file, you could put all kinds of interesting mechanisms: hyperlinks, formatted text, images, marching ant text, blinking text, all of the cool and possibly annoying functions that a web page allows us to accomplish.

The next step we have to do is to be able to get common files to end users’ desktops. This is accomplished by a simple GPO (Group Policy Object) via AD (Active Directory). Hence, if a file needs to be copied to a desktop, a simple call to a UNC path that acts upon a logon process can get the file to the proper location. Something like:

xCopy \\ServerName\Share\*.* C:\Users\%UserName%\AppData\Roaming\Microsoft\Signatures

This will copy all files in this share location to whoever has the GPO applied to and get it to their desktop every time they log onto any machine they have access to. This also ensures consistency of implementing updated files and ease of deploying new machines. For those of you that thought the signature files were in the Outlook profile, you can now see that you can have signatures setup for an end user before they ever launch Outlook on a newly provisioned client machine. Pretty cool stuff.

You can also set permissions on the network folder share to only allow say a manager to edit the files inside the folder. This would allow only the proper people to have access that impacts many other users. Example, around December the files could mention holiday specific information like: “Thank you for the contact, our staff is cycling through holiday time off and your request may take a little longer than usual to get to.” Or at the beginning of the year: “Welcome to the New Year, we are excited about deploying our new products this year.” Whatever the message is, it can easily be updated and controlled from a central location and be deployed to the proper end users that send out the common responses.

Figure 1: Setting share permissions on a folder share.

The next problem we run into is when someone ‘reads’ a message in Outlook, it is marked as ‘read’. Now other mail products allow you to read a message and have it not be marked as read. We don’t do that in Outlook/Exchange. Once a message is read by anyone, you have to go back and mark it as ‘unread’. A training of end users is the only answer here. There are no Outlook settings available to set to change this behavior.

Figure 2: Right click a message and select ‘Mark as Unread’ or on the Ribbon, select Unread/Read option.

We’re almost done. The last few steps are back on the client machines. After you’ve trained your end users to simply apply the appropriate signature file and select the correct mailbox account to send from, you have one more training step. You have to inform them of which ‘sent items’ folder to send from. Yep, there’s another issue here, but one that can also be easily resolved.

Figure 3: Selecting specific signature and which mailbox account to send from.

By default, when sending a message from within Outlook, the message is recorded from the ‘sent item’ folder of the user sending the item. This is not the ideal behavior when we have a common resource mailbox that others have to view sent items from people who are also acting as a single voice from the identical repository location. This is accomplished by a registry edit on the client side. So once again we go back to our friend AD and edit a GPO for consistency. Remember that all registry settings can be edited via GPO’s. Depending on the version of AD you’re on, you may have to use a logon file that calls to a .reg file, or for 2008 and above DC’s, use GPO Preferences to achieve the same process.

There is no universal fix; each user who can ‘Send As’ another mailbox must perform one of the following on their computer, or deploy using a patch management solution, like System Center Configuration Manager (SCCM) and/or GPO’s.

Outlook 2003 (KB317865 and KB953804)

Install the office2003-KB953803-GLB.exe hotfix (http://support.microsoft.com/kb/953803). There is no reboot required; but, Outlook has to be closed.

1. Click Start, click Run, type regedit, and then click OK.

2. Locate and then click the following registry subkey:

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Preferences

3. On the Edit menu, point to New, and then click DWORD Value.

4. Type DelegateSentItemsStyle, and then press ENTER.

5. Right-click DelegateSentItemsStyle, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Exit Registry Editor.

Outlook 2007 (KB972148 and KB970944)

1. Click Start, click Run, type regedit, and then click OK.

2. Locate and then click the following registry subkey:

HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\Preferences

3. On the Edit menu, point to New, and then click DWORD Value.

4. Type DelegateSentItemsStyle, and then press ENTER.

5. Right-click DelegateSentItemsStyle, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Exit Registry Editor.

Outlook 2010 (KB2181579)

1. Click Start, click Run, type regedit, and then click OK.

2. Locate and then click the following registry subkey:

HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\Outlook\Preferences

3. On the Edit menu, point to New, and then click DWORD Value.

4. Type DelegateSentItemsStyle, and then press ENTER.

5. Right-click DelegateSentItemsStyle, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Exit Registry Editor.

Outlook 2013

1. Click Start, click Run, type regedit, and then click OK.

2. Locate and then click the following registry subkey:

HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Preferences

3. On the Edit menu, point to New, and then click DWORD Value.

4. Type DelegateSentItemsStyle, and then press ENTER.

5. Right-click DelegateSentItemsStyle, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Exit Registry Editor.

The other alternative is to have the sender manually move the sent message to the Sent Items folder of the user named as the sender in the email.

Important: After you set the DelegateSentItemsStyle registry value to 1, the functionality is only available when the Microsoft Exchange account is set to Use Cached Exchange Mode. The DelegateSentItemsStyle registry value will not work consistently on an Exchange account that is configured in Online mode.

Figure 4: Creating a GPO, using ‘preferences’, to set which ‘sent items’ folder is allowed to be selected.

So there you have it. Sending a consistent response, from a commonly shared mailbox, using signature files, Outlook client regedit, GPO’s, and a UNC share. Now go out and improve your commonly used shared resource mailboxes and present a stronger corporate image at the same time. Thank you and happy improvements.

Mike O'Neill

Mysterious mail loop on Edge Transport server: Check your size limits!

The Exchange Team — Fri, 29 Mar 2013 01:20:00 GMT

I'm a support enginer in CSS. I was working with a customer who reported a mail loop error for a specific domain like contoso.com. This error was only observed in large emails. Yeah that’s really mysterious until you figure out that the mail loop is due to size restriction on the Send connector. I thought this was curious enough to share.

Understanding the configuration and root cause of the issue:

I initially thought that this might have been the outcome of the Edge server being configured to use an external DNS server (a DNS server that resolves external hosts). Usually, when the Edge Tranport server is configured to use an external DNS, it resolves the domain name to the public IP addresses (generally pointing to itself, the exernal firewall, or the service provider) instead of a Hub Transport server in the Active Diectory site, causing a mail loop.

On reproducing the issue, I found out that the Edge Transport server was not configured to use an external DNS server. The environment I set up to reproduce the issue looked like the diagra below:

Here's what happens in this scenario: When the Edge Transport server receives a 20 MB email from an Internet sender, it accepts it. The Edge Transport server has two connectors that match the address space - one for the address space contoso.com to the Active Directory site and one for the address space *. When making the routing decision based on all available connectors, the one from the Edge to Hub is not considered because of the size restriction (it has 10 MB size limit). The best match is the * connector from Edge to the Internet (Please go over the connector selection algorithm documented in Understanding Message Routing) which has a message size limit of 30 MB.

End result: The message is routed back to the Internet causing the message loop between the Internet and the Edge Server.

Based on whether the Send connector to the Internet is configured to use DNS or a Smart Host to deliver oubound mail, we will get one of the following NDRs:

If using DNS:

#554 5.4.4 SMTPSEND.DNS.MxLoopback; DNS records for this domain are configured in a loop ##

If using a Smart Host:

5.4.6 smtp;554 5.4.6 Hop count exceeded - possible mail loop> #SMTP#

The Solution

This behavior is by design and can be easily rectified by modifying the message size limit on the connector. Based on your requirement, you can choose either of the following options:

Set the MaxMessageSize parameter on the Receive Connector (which receives inbound mail from the Internet) to 10MB, so messages from the Interent are restricted to 10 MB.
Set the MaxMessageSize on the Send connector from Edge to HUB to 30MB, which will allow you to receive 30 MB messages from external senders.

Mystery solved! Thanks to Arindam Thokder and Scott Landry, who helped me with getting this ready for the blog!

Suresh Kumar (XCON)

Announcing Microsoft Connectivity Analyzer (MCA) 1.0 and Microsoft Remote Connectivity Analyzer (RCA) 2.1

The Exchange Team — Mon, 11 Mar 2013 16:56:12 GMT

Back in November 2012, we announced our MCA Beta client. We have been very busy working to improve the testing options that are available from the MCA client. Here’s what we’ve built for the 1.0 release:

Microsoft Connectivity Analyzer Tool 1.0

We are excited to announce the 1.0 release of the Microsoft Connectivity Analyzer. This tool is a companion to the Microsoft Remote Connectivity Analyzer web site. The MCA tool provides administrators and end users with the ability to run connectivity diagnostics for five common connectivity symptoms directly from their local computer. Users can test their own connectivity, and save results in an HTML format that administrators will recognize from viewing results on the RCA website.

Install the MCA 1.0 tool here: https://testconnectivity.microsoft.com/?tabid=client

Watch the Introduction Video:

The MCA tool offers five test symptoms:

“I can’t log on with Office Outlook” – This test is equivalent to the Exchange RCA test for “Outlook Anywhere (RPC over HTTP)”. There is an option to run the SSO test provided on the parameters page.
“I can’t send or receive email on my mobile device”. – This test is equivalent to the Exchange RCA test for Exchange ActiveSync.
***New MCA Test*** “I can’t log on to Lync on my mobile device or the Lync Windows Store App” – This test checks for the Domain Name Server (DNS) records for your on-premise domain to ensure they are configured correctly for supporting Mobile Lync clients. Also it connects to the Autodiscover web service and makes sure that the authentication, certificate, web service for Mobility is correctly set up
***New MCA Test*** “I can’t send or receive email from Outlook (Office 365 only)” – This test checks Inbound/Outbound SMTP mail flow and also includes Domain Name Server validation checks for O365 customers.
***New MCA Test*** “I can’t view free/busy information of another user” – This test verifies that an Office 365 mailbox can access the free/busy information of an on-premises mailbox, and vice versa (one direction per test run).

Microsoft Lync Connectivity Analyzer Tool: You will also notice the Lync Connectivity Analyzer Tool on the client page. We are working on combining MCA with MLCA in the near future but wanted to make both these great tools available to customers now to improve our client diagnostics options. To learn more about MLCA – go HERE

Feedback: Send all feedback to the MCA Feedback alias. Please let us know what you think of the tool and whether this will be helpful in troubleshooting connectivity scenarios. Also feel free to provide feedback on additional tests you would like to see added in the future.

Microsoft Remote Connectivity Analyzer 2.1

We are excited to announce the 2.1 release of the Microsoft Remote Connectivity Analyzer web site. The tool provides administrators and end users with the ability to run connectivity diagnostics for our servers to test common issues with Exchange, Lync and Office 365. We have added new Office 365 Domain Name Server tests, enhanced existing tests, and improved the overall site experience.

Check out the updates to the website here: https://testconnectivity.microsoft.com

Here are the highlights of the 2.1 RCA release:

Version 2.1 (March 2013)

Added support for localized language support for 60 languages
Updated version of the downloadable Microsoft Connectivity Analyzer v1.0 Tool for troubleshooting connectivity from the local machine
Added Microsoft Lync Connectivity Analyzer downloadable tool for troubleshooting Lync issues from the local machine
Added Office 365 General Tests section
Added Office 365 Exchange Domain Name Server (DNS) Connectivity Test

Enjoy!

Thanks.

Brian Feck on behalf of the entire MCA/RCA team.
Follow the team on Twitter - @ExRCA

Updated: Exchange Server 2010 Deployment Assistant for Exchange 2010 SP3 and Office 365

The Exchange Team — Wed, 06 Mar 2013 23:55:19 GMT

We're happy to announce that the Exchange Server 2010 Deployment Assistant (ExDeploy) now includes updates for supporting hybrid deployments with Exchange 2010 Service Pack 3 (SP3) organizations and the latest release of Office 365.

Important information to know about the SP3 update for the Exchange 2010 hybrid deployment scenarios:

Updated information is available only in English at this time. Support in additional languages will follow shortly.
We’ve removed the qualifying question about existing Forefront Online Protection for Exchange for on-premises organizations. Forefront Online Protection for Exchange (FOPE), now known as Exchange Online Protection (EOP), is no longer a limiting factor in hybrid transport routing options. Organizations no longer need to request that their existing EOP service be merged with the EOP service provided with the Microsoft Office 365 tenant. The EOP services are merged automatically and do not require any additional configuration.
We’ve also removed the limitations on configuring centralized mail transport in the Manage Hybrid Configuration wizard. Centralized mail transport in hybrid deployments can be configured regardless of existing EOP service configuration.
If you have previously configured a hybrid deployment using ExDeploy and Exchange 2010 SP2, you’ll need to take a few basic administrative actions as part of updating your hybrid deployment for the latest release of Office 365. See Understanding Upgrading Office 365 Tenants for Exchange 2010-based Hybrid Deployments for more details.

We’d also like to remind everyone that we’ve released the Exchange Server 2013 Deployment Assistant for those organizations that want to take advantage of the features of a new Exchange 2013 installation and for Exchange 2010 and Exchange 2007 organizations that are interested in the benefits of an Exchange 2013-based hybrid deployment. See Now Available: Exchange Server 2013 Deployment Assistant.

Exchange 2010-based hybrid deployments offer Exchange 2010, Exchange 2007 and Exchange 2003 organizations the ability to extend the feature-rich experience and administrative control they have with their existing on-premises Microsoft Exchange organization to the cloud. It provides the seamless look and feel of a single Exchange organization between an on-premises organization and an Exchange Online organization in Office 365. In addition, hybrid deployments can serve as an intermediate step to moving completely to a cloud-based Exchange Online organization. This approach is different than the simple Exchange migration (“cutover migration”) and staged Exchange migration options currently offered by Office 365 outlined in E-Mail Migration Overview.

About the Exchange Server Deployment Assistant

The Exchange Server Deployment Assistant (ExDeploy) is a web-based tool that helps you upgrade to Exchange 2010 on-premises, configure a hybrid deployment between an on-premises and Exchange Online organization or migrate to Exchange Online.

Figure 1:The Exchange Deployment Assistant generates customized instructions to help you upgrade to Exchange 2010 on-premises or in the cloud

It asks you a small set of simple questions, and then based on your answers, it provides a checklist with instructions to deploy or configure Exchange 2010 that are customized to your environment. These environments include:

Stand-alone on-premises Exchange installations and upgrades
Hybrid deployment configurations and
Cloud-only Exchange deployment scenarios.

Besides getting the checklist online, you can also print instructions for individual tasks and download a PDF file of your complete configuration checklist.

Your feedback is very important for the continued improvement of this tool. We would love your feedback on this new scenario and any other area of the Deployment Assistant. Feel free to either post comments on this blog post, provide feedback in the Office 365 community Exchange Online migration and hybrid deployment forum, or send an email to edafdbk@microsoft.com via the Feedback link located in the header of every page of the Deployment Assistant.

Exchange Deployment Assistant Team