Is this thing on?

Recommended Hotfixes for Windows Server 2012 Failover Clusters

2013-06-17T18:10:05Z

Today, Microsoft released Knowledge Base article 2784261, which describes the hotfixes we recommend being installed on all Windows Server 2012 failover clusters. This includes Exchange database availability groups that are running on Windows Server 2012, as well.

Although these hotfixes are generally recommended for all customers, we recommend that you evaluate each fix to determine whether it applies to your environment. If you determine a server in your organization is affected by the problem(s) that a fix addresses, install the fix on each cluster node by using the procedures that are described in Knowledge Base article 174799: Patching Windows Server Failover Clusters.

The recommended hotfixes include two important updates:

2848344 – This is an updated roll-up hotfix that replaces 2838669. This fix is most important for any Hyper-V deployment, and critical for any deployment using DPM for backup.
2838043 – This is a fix for NetName, that addresses a password synchronization issue. This has been a top CSS call generator, where Repair was used in the past.

The hotfixes we recommend for Windows Server 2012 failover clusters are as follows:

Date added	Title and Link	Component	Why we recommend this hotfix
6/14/2013	Update improves cluster resiliency in Windows Server 2012	Failover Cluster Service, CSV Filter and NTFS.sys	This hotfix prevents a CSV failover by fixing an underlying issue in NTFS and by increasing the overall resilience of the Cluster service and CSV during expected pause states. Available for individual download.
6/14/2013	Can't access a resource that is hosted on a Windows 8 or Windows Server 2012-based failover cluster	Failover Cluster Resource DLL	This hotfix prevents an error when accessing resources hosted on a Windows 8-based or Windows Server 2012-based failover cluster from a Windows XP-based or Windows Server 2003-based client computer. It also resolves an Event ID 1196 with error The handle is invalid when the cluster network name resource fails to come online and register in DNS. Available for individual download.
1/23/2013	Failover Cluster Management snap-in crashes after you install update 2750149 on a Windows Server 2012-based failover cluster	Failover Cluster Management snap-in	Resolves a crash in the Failover Cluster Management snap-in after update 2750149 is installed on a Windows Server 2012-based failover cluster. Available from Windows Update or Microsoft Download Center.
11/13/2012	Windows 8 and Windows Server 2012 cumulative update: November 2012	Multiple	Improves clustered server performance and reliability in Hyper-V and Scale-Out File Server scenarios. Improves SMB service and client reliability under certain stress conditions. Install update 2770917 by using Windows Update in order to receive the cumulative update as described in KB 2770917.
11/13/2012	Error code when the kpasswd protocol fails after you perform an authoritative restore: "KDC_ERROR_S_PRINCIPAL_UNKNOWN”	KDCSVC	Install on every domain controller running Windows Server 2008 Service Pack 2 or Windows Server 2008 R2 in order to add a Windows Server 2012 failover cluster. Otherwise Create Cluster may fail when attempting to set the password for the cluster computer object with error message: CreateClusterNameCOIfNotExists (6783): Unable to set password on < ClusterName$> This hotfix is included in Windows Server 2008 R2 Service Pack 1.

For more information, see http://support.microsoft.com/kb/2784261.

High Availability Changes in Exchange Server 2013 Cumulative Update 1

2013-04-02T17:12:16Z

Exchange Server 2013 Cumulative Update 1 (CU1) has been released and is now available for download! CU1 is the first release to use the servicing model introduced with Exchange 2013. CU1 includes new features, new functionality and bug fixes, including in the area of high availability. The announcement post on the Exchange Team Blog already has some great information on what’s new in CU1, but I wanted to augment that announcement with some additional details. Below is a list of some of the high availability-related changes in CU1. This is by no means an exhaustive list; just a list of some of the changes that we have made.

Witness Server Warning Message When Using Certain Database Availability Group Tasks

I first wrote about this issue back in June 2011. This is where the system displays an incorrect warning message when you are using a non-Exchange server as your witness server, even when you have configured things correctly. This issue was eventually fixed in Exchange 2010 Service Pack 2 RU5, but it didn’t make it’s way into Exchange 2013 RTM. Fortunately, the fix did make its way into CU1.

Self-recovery Behaviors

Exchange 2013 continues the innovation introduced in Exchange 2010 by including functionality that allows the system to self-recover from failures that affect resiliency or redundancy. In addition to the Exchange 2010 self-recovery behaviors, Exchange 2013 RTM includes additional behaviors for long I/O times, excessive memory consumption by the Microsoft Exchange Replication service (MSExchangeRepl.exe), and severe cases where threads can't be scheduled. For example, every 30 seconds, the Exchange Replication service heartbeats the crimson channel, as it is a required component for normal operations. If this heartbeat fails, an indication that the crimson channel is inaccessible for some reason, the Exchange Replication service self-recovers the server by forcibly rebooting the server, thereby triggering a server failover.

In addition to the behaviors in Exchange 2013 RTM, CU1 includes new behaviors:

Bus Resets – Event 129 is logged in the System event log when a bus reset occurs. Bus resets, particularly the rare but possible bus reset storm, can often result in storage issues, such as hung IO. When these events occur, it typically requires administrator intervention to resolve the issue. To obviate the need for administrator intervention, CU1 includes new functionality that triggers a forcible reboot of the server when event 129 is detected in the System event log.
Replication service endpoints not responding – Exchange periodically verifies that the TCPListener component in the Exchange Replication service is responding to connection requests by periodically heart beating the local instance of the Exchange Replication service. If the TCPListener does not respond, the system will automatically self-recover by forcibly rebooting the server.

Autoreseed

Automatic reseed, or AutoReseed, is a feature that's the replacement for what is normally administrator-driven action in response to a disk failure, database corruption event, or other issue that necessitates a reseed of a database copy. When properly configured, AutoReseed is designed to automatically restore database redundancy after a disk failure by using spare disks that have been provisioned on the system.

CU1 includes numerous fixes to AutoReseed, including fixes for issues around AutoReseed not detecting spare disks correctly and AutoReseed not using detected spare disks. In addition, the following enhancements have been made to AutoReseed:

GetCopyStatus now has a new field 'ExchangeVolumeMountPoint', which shows the mount point of the database volume under C:\ExchangeVolumes (or a custom folder if you are not using the default setting of C:\ExchangeVolumes). This is useful information to know because in a configuration that uses multiple disks per volume, the LogicalDisk performance counters show up as the first mount point (which would be the one under C:\ExchangeVolumes) instead of as the database path like they used to in Exchange 2010 with a single disk per volume.
We now have better internal tracking around mount paths and the ExchangeVolume path.
The limits for AutoReseed have been increased from 4 databases per volume in Exchange 2013 RTM to 8 databases per volume in CU1.
AutoReseed properties have been added to Active Directory that allow you to enable and disable automatic reseeding and the DiskReclaimer function (which formats Exchange volumes). The two new properties are:
- AutoDagAutoReseedEnabled - Setting AutoDagAutoReseedEnabled to false turns off AutoReseed (including automatic resume, sparing, and in-place reseeds).
- AutoDagDiskReclaimerEnabled - Setting AutoDagDiskReclaimerEnabled to false turns off the DiskReclaimer, which formats exchange volumes. The default setting is true, and it only tries to format volumes mounted under C:\ExchangeVolumes.
The unused AutoDagFailedVolumesRootFolderPath property was also removed from the DAG object.

As a result of these and other changes, the workflow for AutoReseed in CU1 has changed. The primary input condition for the AutoReseed workflow is still a database copy that is in an Failed and Suspended (F&S) state for 15 consecutive minutes. When that condition is detected, the following AutoReseed workflow is initiated:

The system will first try to resume the database copy up to 3 times, with 5 minute sleeps in between each try. Sometimes, after an F&S database copy is resumed, the copy remains in a Failed state. This can happen for a variety of reasons, so this first step is designed to handle all such cases; AutoReseed will automatically suspend a database copy that has been Failed for 10 consecutive minutes to keep the workflow running. If the suspend and resume actions don’t result in a healthy database copy, the workflow continues.
Next, AutoReseed will perform a variety of pre-requisite checks. For example, it will verify that a spare disk is available, that the database and its log files are configured on the same volume, and in the appropriate locations that match the required naming conventions. In a configuration that uses multiple databases per volume, AutoReseed will also verify that all database copies on the volume are in an F&S state.
Next, AutoReseed will attempt to assign a spare volume up to 5 times, with 1 hour sleeps in between each try.
Once a spare has been assigned, AutoReseed will perform an InPlaceSeed operation using the SafeDeleteExistingFiles seeding switch. If one or more database files exists, AutoReseed will wait for 2 days before in-place reseeding (based on the LastWriteTime of the database file). This provides an administrator with an opportunity to preserve data, if needed. AutoReseed will attempt a seeding operation up to 5 times, with 1 hour sleeps in between each try.

Once all retries are exhausted, the workflow stops. If, after 3 days, the database copy is still F&S, the workflow state is reset and it starts again from Step 1. This reset/resume behavior is useful (and intentional) since it can take a few days to replace a failed disk, controller, etc..

Update-MailboxDatabaseCopy

The Update-MailboxDatabaseCopy cmdlet includes some new parameters in CU1 that are designed to aid with automation of seeding operations. These parameters include:

BeginSeed – This is useful for scripting reseeds, because with this parameter, the task asynchronously starts the seeding operation and then exits the cmdlet.
MaximumSeedsInParallel – This is used with the Server parameter to specify the maximum number of parallel seeding operations that should occur across the specified server during a full server reseed operation. The default value is 10.
SafeDeleteExistingFiles – This is used to perform a seeding operation with a single copy redundancy pre-check prior to the seed. Because this parameter includes the redundancy safety check, it requires a lower level of permissions than the DeleteExistingFiles parameter, enabling a limited permission administrator to perform the seeding operation.
Server – This is used as part of a full server reseed operation to reseed all database copies in a Failed and Suspended state. It can be used with the MaximumSeedsInParallel parameter to start reseeds of database copies in parallel across the specified server in batches of up to the value of the MaximumSeedsInParallel parameter copies at a time.

Set-DatabaseAvailabilityGroup

The Set-DatabaseAvailabilityGroup cmdlet includes a new parameter named SkipDagValidation. It is used to bypass the validation of the DAG's quorum model and the health check on the DAG's witness during certain DAG configuration operations. While this parameter has some usefulness for us in Exchange Online (and that is why it was introduced), and while it is enabled for on-premises use, it won’t be of much use to on-premises environments. I’m only pointing it out because, as I said, it is enabled for on-premises use.

Managed Availability: Get-ServerHealth and Get-HealthReport

The Get-ServerHealth and Get-HealthReport cmdlets are used to get and process raw health set data from Managed Availability, the new monitoring and recovery framework used by the various components within Exchange. Get-ServerHealth can be used to view the various health sets and their current status. In Exchange 2013 RTM, the Get-HealthReport cmdlet consumed results from Get-ServerHealth to produce a summary rollup of health. But the way in which it was implemented made it very slow and inefficient.

With CU1, instead of piping Get-ServerHealth to Get-HealthReport, Get-HealthReport is now capable of reporting the consolidated results on its own, and it now takes an Identity parameter that enables you to specify a server instead of InputObject/InputEntries. Get-HealthReport also includes a new HealthSet parameter, which is used to return the health state for a group of monitors. However, to use a rollup group, a list of names must be pipelined to Get-HealthReport. Unfortunately, Get-HealthReport -Identity does not support an array of names, so our recommended way to do this is to simply get the list of DAG members and pipe that to Get-HealthReport. For example to display a rollup summary of transport health on members of a DAG, you would run:

(Get-DatabaseAvailabilityGroup DAG1).Servers | Get-HealthReport -RollupGroup -HealthSet HubTransport

There are a couple of changes for Get-ServerHealth in CU1; namely, two parameters have also been added:

HaImpactingOnly – This is used to display only monitors that have HaImpacting set.
HealthSet – This is used to return the health state of a group of monitors.

Best Copy and Server Selection Changes

Best Copy and Server Selection (BCSS) is the algorithm used by Active Manager in Exchange 2013 to select the best database copy to activate in response to a failover or a target-less switchover. In CU1, a change was made so that the Primary Active Manager (PAM) now keeps track of the number of active databases per server, so that during BCSS it can honor the value of MaximumActiveDatabases, if configured. The server holding the PAM role now keeps an in-memory state that tracks the number of active databases per server. When the PAM role moves or when the Exchange Replication service is restarted on the PAM, this information is rebuilt from the cluster database.

This change allows Active Manager to exclude servers that are already hosting the maximum amount of active databases when determining potential candidates for activation. Prior to this change, Active Manager would not evaluate whether a potential server candidate for activation was already at its configured active database limit. Thus, if such a server were selected for activation, the activation process would fail during the mount attempt, and a new server would have to be selected (if available). This scenario is now avoided as a result of this change.

Other CU1 Changes

Of course there are other changes in CU1 besides the above, so be sure to read the Release Notes and other appropriate documentation when everything is released.

Storage, High Availability and Site Resilience in Exchange Server 2013, Part 3

2012-11-01T18:41:50Z

Microsoft Exchange Server 2013 continues to innovate in the areas of storage, high availability, and site resilience. In this three-part blog series, I’ll describe the significant improvements in Exchange 2013 related to these three areas. Part 1 focuses on the storage improvements that we’ve made. Part 2 focuses on high availability. And Part 3 will focus on site resilience.

Although Exchange 2013 continues to use DAGs and Windows Failover Clustering for Mailbox server role high availability and site resilience, site resilience is not the same in Exchange 2013. Site resilience is much better in Exchange 2013 because it has been operationally simplified. The underlying architectural changes that were made in Exchange 2013 have significant impact on site resilience configurations and their recovery aspects.

Challenges with Exchange 2010 Site Resilience

Exchange 2010 undoubtedly made achieving site resilience for the messaging service and data easier than any previous version of Exchange. By combining the native site resilience features in Exchange 2010 with proper planning, you were able to activate a second datacenter to serve a failed datacenter's clients. The process you perform to do this is referred to as a datacenter switchover. This is a well-documented and generally well-understood process, although it takes time to perform, and it requires human intervention in order to begin the process.

Anyone who has performed a datacenter switchover in Exchange 2010 will tell you that they are operationally complex. This is in part because in Exchange 2010, recovery of mailbox data (DAG) and client access (namespace) are tied together. This leads to other challenges for Exchange 2010 in certain scenarios:

If you lose all or a significant portion of your Client Access servers, or the VIP for the array, or the top of rack (TOR), or if you lose a significant portion of your DAG, you were in a situation where you needed to do a datacenter switchover.
You could deploy a DAG across two datacenters and host the witness in a third datacenter and enable failover for the Mailbox role for either datacenter. But you didn’t get failover for the messaging service, because the namespace still needed to be switched over for the non-Mailbox roles.

But all of that aside, the biggest challenge with Exchange 2010 is that the namespace is a single point of failure. In Exchange 2010, the most significant single point of failure in the messaging system is the FQDN that you give to users because it tells the user where to go. Changing the IP address for that FQDN is not that easy because you have to change DNS and deal with DNS latency, which in some parts of the world is very bad. And you have name caches in browsers which are typically around 30 minutes or more that also have to be dealt with.

Exchange 2013 Addresses the Challenges

Significant changes have been made in Exchange 2013 that address the challenges with Exchange 2010 site resilience head on. With the namespace simplification, consolidation of server roles, removal of AD-sited-ness, separation of CAS array and DAG recovery, and load balancing changes, Exchange 2013 provides new site resilience options, such as the ability to use a single global namespace. In addition, for customers with more than two locations in which to deploy messaging service components, Exchange 2013 also provides the ability to configure the messaging service for automatic failover in response to failures that required manual intervention in Exchange 2010.

Specifically, site resilience has been operationally simplified in Exchange 2013. In addition, in Exchange 2013, the namespace does not need to move with the DAG. Exchange leverages fault tolerance built into the namespace through multiple IP addresses, load balancing (and if need be, the ability to take servers in and out of service). One of the most significant changes we made in Exchange 2013 was to leverage the clients’ ability to get more than one place to go. Assuming the client has the ability to use more than one place to go (which almost all HTTP clients do, and since almost all of the client access protocols in Exchange 2013 are HTTP based (Outlook, Outlook Anywhere, EAS, EWS, OWA, EAC, RPS, etc.), all supported HTTP clients have the ability to use multiple IP addresses), thereby providing failover on the client side. You can configure DNS to hand multiple IP addresses to a client during name resolution. The client asks for mail.contoso.com and gets back two IP addresses, or four IP addresses, for example. However many IP addresses the client gets back will be used reliably by the client. This makes the client a lot better off because if one of the IP addresses fails, the client has one or more others to try to connect to. If a client tries one and it fails, it waits around 20 seconds and then tries the next one in the list. Thus, if you lose the VIP for the CAS array, and you have a second VIP for a second CAS array, recovery for the clients happens automatically, and in about 21 seconds.

Modern HTTP clients (operating systems and Web browsers that are ten years old or less) simply work with this redundancy automatically. The HTTP stack can accept multiple IP addresses for an FQDN, and if the first IP it tries fails hard (e.g., cannot connect), it will try the next IP in the list. In a soft failure (connect lost after session established, perhaps due to an intermittent failure in the service where, for example, a device is dropping packets and needs to be taken out of service), the user might need to refresh their browser.

Operationally Simplified Site Resilience

So what does it mean that site resilience has been operationally simplified in Exchange 2013? Going back to the failure scenarios discussed above for Exchange 2010, if you encounter those scenarios in Exchange 2013, depending on your site resilience configuration, you might not need to perform a datacenter switchover. With the proper configuration, failover will happen at the client level and clients will be automatically redirected to a second datacenter that has operating Client Access servers, and those operating Client Access servers will proxy the communication back to the user’s Mailbox server, which remains unaffected by the outage (because you don’t do a switchover). Instead of working to recover service, the service recovers itself and you can focus on fixing the core issue (e.g., replacing the failed load balancer). Any administrator will tell you that the stress involved with replacing a failed piece of equipment that isn’t blocking service is much lower than the stress involved in restoring service and data access via a datacenter switchover.

So by comparison, in Exchange 2010, if you lose the load balancer in your primary datacenter and you don’t have another one in that site, you had to do a datacenter switchover. In Exchange 2013, if you lose the load balancer in your primary site, you simply turn it off (or maybe turn off the VIP) and repair/replace it.

Clients that aren’t already using the VIP in the secondary datacenter will automatically failover to the secondary VIP without any change of namespace, and without any change in DNS. Not only does that mean you no longer have to perform a switchover, but it also means that all of the time normally associated with a datacenter switchover recovery is not spent either. In Exchange 2010, you had to deal with DNS latency (hence, the recommendation to set the TTL to 5 min, and the introduction of the Failback URL). In Exchange 2013, you don’t need to do that because you get fast failover (~21 seconds) of the namespace between VIPs (datacenters).

Since you can failover the namespace between datacenters now, all that is needed to achieve a datacenter failover is a mechanism for failover of the Mailbox role across datacenters. To get automatic failover for the DAG, you simply architect a solution where the DAG is evenly split between two datacenters, and then place the witness server in a third location so that it can be arbitrated by DAG members in either datacenter, regardless of the state of the network between the datacenters that contain the DAG members. The key is that third location is isolated from network failures that affect the first and/or second location (the locations containing the DAG members).

In this scenario, the administrator’s efforts are geared toward simply fixing the problem, and not spent restoring service. You simply fix the thing that failed; all the while service has been running and data integrity has been maintained. The urgency and stress level you feel when fixing a broken device is nothing like the urgency and stress you feel when you’re working to restore service. It’s better for the end user, and less stressful for the admin.

You can allow failover to occur without having to perform switchbacks (sometimes mistakenly referred to as failbacks). If you lose CAS in your primary datacenter and that results in a 20 second interruption for clients, you might not even care about failing back. At this point, your primary concern would be fixing the core issue (e.g., replacing the failed load balancer). Once that is back online and functioning, some clients will start using it, while others might remain operational through the second datacenter.

Exchange 2013 also provides functionality that enables administrators to deal with intermittent failures. An intermittent failure is where, for example, the initial TCP connection can be made, but nothing happens afterwards. An intermittent failure requires some sort of extra administrative action to be taken because it might be the result of a replacement device being put into service. While this repair process is happening, the device might be powered on and accepting some requests, but not really ready to service clients until the necessary configuration steps are performed. In this scenario, the administrator can perform a namespace switchover by simply removing the VIP for the device being replaced from DNS. Then during that service period, no clients will be trying to connect to it. Once the replacement process has completed, the administrator can add the VIP back to DNS and clients will eventually start using it.

Conclusion

Microsoft Exchange Server 2013 continues to innovate in the areas of storage, high availability, and site resilience, with its plethora of new, innovative features, such as:

Multiple databases per volume
Autoreseed
Automatic recovery from storage failures
Lagged copy enhancements
Managed Availability
Best Copy and Server Selection enhancements
Maintenance Mode
Automatic DAG network configuration
Operationally simplified site resilience
Separation of Mailbox and Client Access recovery
Leveraging client-side DNS behaviors, such as IP failover for the namespace

It’s important to understand that major architectural changes had to take place in Exchange 2013 in order to enable these features. That means that while the Exchange 2010 design guidelines can apply to an Exchange 2013 organization, the additional enhanced Exchange 2013 design guidelines cannot be applied to Exchange 2010. All of the goodness above around new behaviors and design options applies to Exchange 2013 only.

Storage, High Availability and Site Resilience in Exchange Server 2013, Part 2

2012-10-01T16:20:43Z

Exchange 2013 continues to use DAGs and mailbox database copies, along with other features such as Single Item Recovery, Retention Policies, lagged database copies, etc., to provide Exchange Native Data Protection. However, the high availability platform, the Exchange Information Store, the best copy selection process, the Extensible Storage Engine (ESE), and internal monitoring have been significantly modified and enhanced to provide even greater availability, easier management, and to reduce costs.

With respect to high availability, some of these enhancements include:

Managed Availability
A New Best Copy Selection process
Maintenance Mode
DAG auto-network configuration

Managed Availability

Managed availability is the integration of built-in monitoring and recovery actions with Exchange’s built-in high availability platform. It is designed to detect and recover from problems as soon as they occur and are discovered by the system. Unlike previous external monitoring solutions for Exchange, managed availability does not try to identify or communicate the root cause of an issue. It is instead focused on recovery aspects that address three key areas of the user experience:

Availability – can users access the service?
Latency – how is the experience for users?
Errors – are users able to accomplish what they want to?

The new architecture in Exchange 2013 makes each Exchange server an “island” where services on that island only service the active databases located on that server. The architectural changes in Exchange 2013 require a new approach to availability model used by Exchange. The Mailbox and Client Access server architecture imply that any Mailbox server with an active database is in production for all services, including all protocol services. As a result, this fundamentally changes the model used to manage the protocol services.

Managed availability was conceived of to address this change and to provide a native health monitoring and recovery solution. The integration of the building block architecture into a unified framework provides a powerful capability to detect failures and recover from them. Managed availability moves away from monitoring individual separate slices of the system to monitoring the end-to-end user experience, and protecting the end user’s experience through recovery-oriented computing.

Managed availability is an internal process that runs on every Exchange 2013 server. It is implemented in the form of two processes:

Exchange Health Manager Service (MSExchangeHMHost.exe) - A controller process that is used to manage worker processes. It is used to build, execute, and start and stop the worker process, as needed. It is also used to recover the worker process in case that process crashes, to prevent the worker process from being a single point of failure.
Exchange Health Manager Worker process (MSExchangeHMWorker.exe) - A worker process that is responsible for performing the runtime tasks.

Managed availability uses persistent storage to perform its functions:

The Front-End Transport service uses XML configuration files to initialize the work item definitions during startup of its worker processes.
- Other components have their configuration information stored in code.
The Windows registry is used to store runtime data, such as bookmarks.
The Windows Crimson Channel event log infrastructure is used to store the work item results.

As illustrated in the following drawing, managed availability includes three main asynchronous components that are constantly doing work: the probe engine, the monitor, and the responder engine.

Figure 1 - Managed Availability in Exchange Server 2013

The probe engine measures and collects data from the servers and passes it to the monitor. The monitor contains the business logic used by the system to determine if the server is healthy, based on the pre-defined definitions of health and the collected data. Basically, the monitor is looking for patterns in the collected measurements, and then it makes a decision on whether or not something is healthy. If a server is considered unhealthy, the responder engine handles recovery actions. Generally, when a particular component becomes unhealthy, the first action is to try to recover just that component. If the recovery actions are unsuccessful, the system escalates the issue by notifying an administrator via event log entries.

The probe engine contains probes, checks and notification logic. Probes are synthetic transactions performed by the system to test the end-to-end user experience. Checks are the infrastructure that perform the collection of performance data, including user traffic, and measure the collected data against thresholds that are set to determine spikes in user failures. This enables the checks infrastructure to become aware of when users are experiencing issues. Finally, the notification logic enables the system to take action immediately based on a critical event, without having to wait for the results of the data collected by a probe. These are typically exceptions or conditions that can be detected and recognized without a large sample set.

Monitors query the data collected by probes to determine if action needs to be taken based on a pre-defined rule set. Depending on the rule, or the nature of the issue, a monitor can either initiate a responder or escalate the issue to a human via an event log entry. In addition, monitors define how much time after a failure that a responder is executed, as well as the workflow of the recovery action. Monitors have various states. From a system state perspective, monitors have two states:

Healthy - The monitor is operating properly and all collected metrics are within normal operating parameters
Unhealthy - The monitor is not healthy and has either initiated recovery through a responder or notified an administrator through escalation.

From an administrative perspective, monitors have additional states that will appear in PowerShell:

Degraded - When a monitor is in an unhealthy state for X seconds, it is considered Degraded. If a monitor is unhealthy for more than Y seconds, it is considered Unhealthy. Each component defines its time intervals.
Disabled - The monitor has been explicitly disabled by an administrator.
Unavailable - The Microsoft Exchange Health service periodically queries each monitor for its state. If it does not get a response to the query, the monitor state becomes Unavailable.
Repairing - Set by an administrator to indicate to the system that corrective action is in process by a human, which allows the system and humans to differentiate between other failures that may occur at the same time correction action is being taken (such as a database copy reseed operation).

Responders are responsible for executing a response to an alert generated by a monitor because a monitor has become Unhealthy. There are many different types of responders that are designed to address the particular nature of the alert by performing recovery actions in a specific sequence. For example, there are responders that terminate and restart services or cycle IIS application pools (the two that will likely be the most commonly triggered in most environments), responders that perform a server failover or forcibly reboots the server, responders that take servers in and out of service, and responders that escalate issues to administrators by generating additional alerts.

Probes, responders and monitors have default thresholds and parameters that can be overridden and configured by an administrator. This allows you to fine tune Managed Availability for your environment and it can be used to manage system behavior during unplanned outages or emergency situations. These overrides can be applied to the entire environment, or scoped to specific servers, or specific versions of the server. In addition, they can be configured to apply for a specific duration (e.g., while replacing failed equipment). Overrides that are applied to specific servers are stored in the registry of the servers to which they are applied. Overrides that apply to the entire environment are stored in Active Directory.

The probes, monitors and responders used by Managed Availability are grouped together into health sets. Health sets are used to determine whether or not a given system component is healthy. The overall status of a health set is an aggregate of the health of the monitors in the health set, with the least healthy status listed in the evaluation. Health sets are then grouped into health groups that are exposed in System Center Operations Manager (SCOM) for reporting purposes. Instead of SCOM or an external engine performing recovery actions, Managed Availability performs the recovery actions. SCOM is simply used as a portal to see health information related to the environment, and to receive alerts when an escalation responder is invoked.

For additional information on Managed Availability, see Ross Smith IV’s blog post.

Managed Availability Failovers

Failovers are the primary recovery mechanism for the Mailbox server and mailbox databases. This has been the case for the past few releases and past several years. And that continues to be the case today with Exchange 2013. However, the list of failures that can trigger failovers has been expanded as a result of the introduction of Managed Availability. Managed Availability-driven failovers are a new form of recovery from some detected failures. These are failures detected by Managed Availability probes and monitors via a synthetic operation or as a result of live data that is being monitored.

While many failures can be resolved through other means (for example, restarting a service or an IIS application pool), some require some form of failover to provide to provide corrective action. Managed Availability-driven failovers come in two forms: server and database. Protocol failures (for example, a failure affecting OWA) can trigger a server-wide response and initiate a server failover. Store-detected per-database failures can trigger a database failover.

When failovers due occur, internal throttling mechanisms across time and across the DAG are used to prevent database activation storms during certain types of failures (for example, when a protocol is repeatedly crashing).

In addition to these triggers, Active Manager also uses the information gathered by Managed Availability to determine where active copies should be moved to, based on a variety of health criteria regarding the passive copies and the servers hosting them. This has resulted in significant changes to the best copy selection algorithm in Exchange 2013, as described below.

Best Copy and Server Selection

Best copy selection (BCS) is an internal Active Manager algorithm used to select the “best” copy of a specific database to activate based on a list of inputs that include potential copies and their status. In Exchange 2010, four criteria are used to evaluate potential copies and determine which one to try to activate first:

Copy queue length
Replay queue length
Database copy status
Content index status

One of the architectural tenets of Exchange 2013 is that every server is an “island.” Among other things, this means that access to a database is provided through the protocols running on the server hosting the active copy. As a result of this architectural change, using only the above four criteria is not good enough for Exchange 2013 because protocol health is not considered. The new architecture in Exchange 2013 requires a much smarter selection of the copy being activated, and it begins with the premise that no failover operation should ever make the health worse.

Significant changes have been made to the BCS process to include Managed Availability health set information and to further tune the process. Exchange 2013 uses the database copy and content index health status information used by Exchange 2010, but Exchange 2013 also evaluates the health of the entire protocol stack using Managed Availability health sets. It evaluates the health of the triggering functionality on the potential target systems (e.g., if OWA failed on the current active copy, how healthy is OWA on the servers hosting passive copies), the health of the servers hosting passive copies, and the priority of the protocol health set. All health sets have a priority of either low, medium, high or critical.

As a result of these significant changes and new behaviors, we’re calling the process something slightly different in Exchange 2013: Best Copy and Server Selection (BCSS).

There are four new additional checks performed during BCSS (listed in the order in which they are performed):

All Healthy - checks for a server hosting a copy of the affected database that has all monitoring components in a healthy state.
Up to Normal Healthy - checks for a server hosting a copy of the affected database that has all monitoring components with Normal priority in a healthy state.
All Better than Source - checks for a server hosting a copy of the affected database that has monitoring components in a state that is better than the current server hosting the affected copy.
Same as Source - checks for a server hosting a copy of the affected database that has monitoring components in a state that is the same as the current server hosting the affected copy.

If BCSS is invoked as a result of a failover that is triggered by a monitoring component (e.g., via a failover responder), an additional mandatory constraint is enforced where the target server's component health must be better than the server on which the failover occurred. For example, if a failure of Outlook Web App (OWA) triggers a failover via a Failover responder, then BCSS must select a server hosting a copy of the affected database on which OWA is healthy.

TIP: To see what Managed Availability is reporting for the health and health sets on your servers, you can use the Get-ServerHealth and Get-HealthReport cmdlets. The way these cmdlets are designed, the obviously way to get a health summary is to simply run get-serverhealth | get-healthreport. However, these cmdlets are capable of returning so much data, that this simplest way can often time out or error out. So when reviewing status of health sets, we recommend that you filter Get-ServerHealth to return a single health set (e.g., get-serverhealth | ? {$_.HealthSet -eq “AD”} | get-healthreport).

Maintenance Mode

Maintenance mode is new functionality that enables you to designate a server as in-service or out-of-service by using the Set-ServerComponentState cmdlet. This is significantly different from StartDagServerMaintenance.ps1, the script we introduced in Exchange 2010, for preparing a server for maintenance.

Maintenance mode is used to determine whether or not the server should be available to service clients. In addition to marking a Mailbox server or a Client Access server as out-of-service, it also records information to prevent client workloads from occurring on the system. For example, if you want to perform maintenance on a Mailbox server you would perform a server switchover, and then use the Set-ServerComponentState cmdlet to put the Mailbox server into maintenance mode, which prevents the active copies from being activated on the server in maintenance mode.

When you put a Mailbox server into maintenance mode, it means that there are no active database copies on the server and that Transport services are offline. When you take a CAS out of service, CAS will stop acknowledging load balancer heartbeats and all proxy services will be offline.

DAG Network Autoconfig

As you may have gathered from previous Exchange releases, we have been pushing to make it easier and easier to have a highly available messaging system; to lower the bar for adoption, if you will. For example, in previous versions of Exchange, the system automatically installs and configures Windows Failover Clustering when needed. In Exchange 2013, we’ve taken that behavior even further by providing default behavior where DAG networks are managed for you automatically.

As with Exchange 2010, there are some tasks that the administrator needs to perform ahead of time; namely, the installation and configuration of network interface cards (NICs). It is still important that each NIC be configured based on its intended use as either a MAPI or Replication network (for example, the MAPI network is registered in DNS, and the Replication network is not; the MAPI network has a default gateway in a multi-subnet environment, and the Replication network does not). As long as the configuration settings are correct, then DAG networks become much more simplified in terms of management.

In addition, in a multi-subnet DAG, it is no longer necessary to collapse DAG networks in Exchange 2013. Exchange now does that for you automatically, provided that the above-mentioned configuration settings are correct.

By default, all DAG networks are automatically managed by the system. To view DAG network settings in auto-config mode, you’ll need to use the Exchange Management Shell. You can, however, use the Exchange Administration Center (EAC) to configure manual control over DAG networks (see the following figure), which enables you to view, edit, create and remove DAG networks using EAC or the Shell.

You can also enable manual control by using Set-DatabaseAvailabilityGroup <DAGName> –ManualDagNetworkConfiguration:$True. Once you enable manual control, you can perform some common tasks, such as:

Add or remove DAG networks, as needed
Configuring the DAG to ignore iSCSI or dedicated backup networks, if any
Disable replication on the MAPI network (optionally)

Storage, High Availability and Site Resilience in Exchange Server 2013, Part 1

2012-09-19T17:24:00Z

Microsoft Exchange Server 2013 continues to innovate in the areas of storage, high availability, and site resilience. In this three-part blog series, I’ll describe the improvements in Exchange 2013 related to these three areas. Part 1 focuses on the storage improvements that we’ve made. Part 2 will focus on high availability. And part 3 will focus on site resilience.

Exchange 2013 continues to use DAGs and mailbox database copies, along with other features such as Single Item Recovery, Retention Policies, lagged database copies, etc., to provide Exchange Native Data Protection. The high availability platform, the Exchange Information Store and the Extensible Storage Engine (ESE), have all been enhanced to provide greater availability, easier management, and to reduce costs.

With respect to storage, these enhancements include:

Reduction in IOPS over Exchange 2010 – enables you to leverage larger disks in terms of capacity and IOPS as efficiently as possible.
Multiple databases per volume – enables you to host multiple databases (mixtures of active and passive copies) on the same volume, thereby leveraging larger disks in terms of capacity and IOPS as efficiently as possible.
AutoReseed – enables you to quickly restore database redundancy after disk failure. If a disk fails, the database copy stored on that disk are copied from the active database copy to a spare disk on the same server. If multiple database copies were stored on the failed disk, they can all be automatically re-seeded on a spare disk. This enables faster reseeds, as the active databases are likely to be on multiple servers and the data is copied in parallel.
Automatic recovery from storage failures – allows the system to recover from failures that affect resiliency or redundancy. Exchange 2013 includes recovery behaviors for long IO times, excessive memory consumption by the Microsoft Exchange Replication service (MSExchangeRepl.exe), and severe cases where the system is in such a bad state, threads cannot be scheduled.
Lagged copy enhancements – Lagged copies can now care for themselves to a certain extent using automatic log play down. In addition, lagged copies can leverage Safety Net, making recovery or activation much easier.

Of course, we also introduced the Managed Store. The Managed Store is the name of the newly rewritten Information Store processes (Microsoft.Exchange.Store.Service.exe and Microsoft.Exchange.Store.Worker.exe) in Exchange 2013. The new Managed Store is written in C# and tightly integrated with MSExchangeRepl.exe to provide higher availability through improved resiliency. In addition, the Managed Store has been architected to enable more granular management of resource consumption and faster root cause analysis through improved diagnostics. The Managed Store works with the Microsoft Exchange Replication service to manage mailbox databases, which continues to leverage Extensible Storage Engine (ESE) as the database engine. Exchange 2013 includes significant changes to the mailbox database schema that provide many optimizations over previous versions of Exchange. In addition to these changes, the Microsoft Exchange Replication service is responsible for all service availability related to back-end servers. The architectural changes enable faster database failover and better physical disk failure handling. Managed Store will be the subject of a future blog post.

Improved Resilience for JBOD Environments

Let’s dig a little deeper into each of these features by first providing some context. While most of these features were designed primarily for configurations that use just a bunch of disks (JBOD), all of these features can work with any supported Exchange storage. JBOD environments bring unique challenges to Exchange:

The trend of capacity increase continues, with 8TB drives expected to be available soon. When using 8TB drives in conjunction with Exchange’s database size best practices guidelines (2 TB), you would waste 5+ TB of disk space. One solution would be to simply grow the databases larger, but that inhibits manageability because it introduces very long reseed times, including, in some cases, operationally unmanageable reseed times. Not to mention the reliability of copying that amount of data over the network.
In the Exchange 2010 model, the disk storing a passive copy is under-utilized in terms of IOPS. And in the case of a lagged passive copy, not only is the disk under-utilized in terms of IOPS, but its also asymmetric in terms of its size, relative to the disks used to store the active and non-lagged passive copies.
In Exchange 2010, if you are in a scenario where you are running out of disk space in a JBOD configuration, you have only a few options. You could for example, throw away the catalogs while you move users off. It does impair the search experience, but it gives you breathing room when you need it. Still, you are limited in terms of available options.

Reduction in IOPS over Exchange 2010

In Exchange 2010, passive database copies have a very low checkpoint depth which is required for fast failover. In addition, in Exchange 2010, the passive copy performs aggressive pre-reading of data in order to keep up with a 5 MB checkpoint depth. As a result of using a low checkpoint depth, and performing these aggressive pre-read operations, IOPS for a passive database copy was equal to IOPS for an active copy in Exchange 2010.

In Exchange 2013, we’re able to get fast failover without a low checkpoint depth on the passive copy, and in fact, we now have fast failover with a high checkpoint depth on the passive copy (100 MB). Since we now have 100 MB checkpoint depth on the passive, the passive copy has been de-tuned to no longer be so aggressive. As a result of increasing the checkpoint depth and de-tuning the aggressive pre-reads, IOPS for a passive copy is about 50% of the active copy IOPS in Exchange 2013.

Of course, we had to deal with having a higher checkpoint depth on the passive copy. On failover in Exchange 2010, we flush the cache as we are converting the copy from a passive copy to an active copy, and with a high checkpoint depth we would not have fast failover.

In Exchange 2013, the ESE team re-wrote logging in ESE to persist the cache as the transition from passive to active is made. Because ESE doesn’t need to flush the cache, we get fast failover. In fact, in Exchange 2013, failover times are generally down by 50% over Exchange 2010. Internally, in our Exchange 2013 dogfood environment, we see failovers around 10 seconds. In our production Exchange 2010 environment, they are around 20 seconds.

One other change we made was around background database maintenance. Based on our bad block experience in Office 365, we decided that BDM did not need to be as aggressive. So, BDM is now throttled back from 5MB per sec/copy to 1MB per sec/copy. At that speed, BDM doesn’t cause any IO or latency issues.

Multiple Databases Per Volume

This feature is about Exchange optimizing for large disks. These optimizations result in a much more efficient use of large disks in terms of capacity, IOPS and re-seed times, and they are meant to address the challenges I describe above that are associated with running in a JBOD storage configuration:

Database sizes must be manageable
Reseed operations must be fast and reliable
Storage capacity is increasing, but IOPS are not
Disks hosting passive database copies are underutilized in terms of IOPS
Lagged copies have asymmetric storage requirements
You have limited agility to recover from low disk space conditions

Continuing a long-standing practice for Exchange, we have optimized Exchange 2013 so that it can use very large disks (e.g., 8 TB) in a JBOD configuration more efficiently. In Exchange 2013, with multiple databases per disk, you can have the same sized disks storing multiple database copies, including lagged copies. The goal is to drive the distribution of users across the number of volumes that exist, providing you with a symmetric design where during normal operations each DAG member hosted a combination of active, passive and optional lagged copies on the same volumes.

An example of a configuration that uses multiple databases per volume is illustrated below:

The above configuration provides a nice, symmetrical design. All four servers have the same four databases all hosted on a single disk per server. The key here is that the number of copies of each database that you have should be equal to the number of database copies per disk. In the above example, there are four copies of each database: one active copy, two passive copies, and one lagged copy. Because there are four copies of each database, the proper configuration is one that has 4 copies per volume. In addition, Activation Preference (AP) is configured so that it is balanced across the DAG and across each server. For example, the active copy will have an AP value of 1, the first passive copy will have an AP value of 2, the second passive copy will have an AP value of 3, and the lagged copy will have an AP value of 4.

In addition to having a better distribution of users across the existing volumes, another benefit of using multiple databases per disk is that it reduces the amount of time to restore data protection in the event of a failure that necessitates a reseed (e.g., disk failure).

As a database gets bigger, reseeding the database takes longer and longer. For example, a 2TB database could take 23 hours to reseed, whereas an 8TB database could take as long as 93 hours (almost 4 days). Both seeds would occur at around 20MB/sec. This generally means that a very large database cannot be seeded within an operationally reasonable amount of time.

In the case of a single-database-copy-per-disk scenario, the seeding operation is effectively source-bound, because it is always seeding the disk from a single source. By dividing the volume into multiple database copies, and by having the active copy of the passive databases on a given volume stored on separate DAG members, the system is no longer source-bound in the context of reseeding the disk. When a failed disk is replaced, it can be reseeded from multiple sources. This allows the system to reseed and restore data protection for these databases in a much shorter amount of time.

Another benefit is a 25% increase in aggregate disk utilization, and a balanced configuration of four databases with four copies across four disks and four servers. In the event you lose partial service (for example, you lose two DAG members and you have a datacenter *over), you are still at only 50% IOPS utilization.

There are specific architecture requirements when using multiple databases per volume:

You must use a single logical disk partition per physical disk. Do not create multiple partitions on the disk.
Each database copy and its companion files (transaction logs, content index, etc.) must be hosted in a unique directory on the single partition.

We also recommend adhering to the following best practices:

Database copies should have the same neighbors (e.g., they should all share the same disk on each server).
Balance activation preference across the DAG, such that each database copy on a given disk has a unique Activation Preference value.

Automatic Reseed

Automatic reseed, or AutoReseed for short, is a feature that is the replacement for what is normally administrator-driven action in response to a disk failure, database corruption event, or other issue that necessitates a reseed of a database copy. AutoReseed is designed to automatically restore database redundancy after a disk failure by using spare disks that have been provisioned on the system.

In an AutoReseed configuration, a standardized storage presentation structure is used, and the administrator picks the starting point. AutoReseed is about restoring redundancy as soon as possible after a drive fails. This involves pre-mapping a set of volumes (including spare volumes) and databases using mount points. In the event of a disk failure where the disk is no longer available to the operating system, or is no longer writable, a spare volume is allocated by the system, and the affected database copies are re-seeded automatically. AutoReseed uses the following process:

The Microsoft Exchange Replication service periodically scans for copies that have a status of FailedAndSuspended.
When it finds a copy with that status, it performs some pre-requisite checks, such as checking to see if this is a single copy situation, checking to see if spares are available, and making sure there is nothing preventing the system from performing an automatic reseed.
If the pre-requisite checks pass successfully, the Replication service allocates and remaps a spare.
Next, the seeding operation is performed.
Once the seed has been completed, the Replication service verifies that the newly seeded copy is healthy.

At this point, if the failure was a disk failure, it would require manual intervention by an operator or administrator to remove and replace the failed disk, format it, initialize it and re-configure it as a spare.

AutoReseed is configured using three properties of the DAG. Two of the properties refer to the two mount points that are in use. Exchange 2013 Preview leverages the fact that Windows Server allows multiple mount points per volume. The AutoDagVolumesRootFolderPath property refers to the mount point that contains all of the volumes that are available. This includes volumes that host databases and spare volumes. The AutoDagDatabasesRootFolderPath property refers to the mount point that contains the databases. A third DAG property, AutoDagDatabaseCopiesPerVolume, is used to configure the number of database copies per volume.

An example AutoReseed configuration is illustrated below:

In this example, there are three volumes, two of which will contain databases (VOL1 and VOL2), and one of which is a blank, formatted spare (VOL3).

To configure AutoReseed:

All three volumes are mounted under a single mount point. In this example, the administrator has configured a mount point of C:\ExchVols is used. This represents the directory used to get storage for Exchange databases.
Then, the root directory of the mailbox databases is mounted as another mount point. In this example, the administrator has configured a mount point of C:\ExchDbs is used. Next, create a directory structure so that each database has two directories: one directory for the database file and one for the log files.
Then create databases. The above example illustrates a simple design using a single database per volume. Thus, on VOL1, there are two directories: one for DB1’s database file, and one for its logs. On VOL2, there are two directories: one for DB2’s database file, and one for its logs.

In this configuration, if DB1 or DB2 were to experience a failure, VOL3 will be automatically re-purposed by the system, and a copy of the failed database will be automatically reseeded to VOL3.

Automatic Recovery from Storage Failures

This feature continues the innovation introduced in Exchange 2010 to allow the system to recover from failures that affect resiliency or redundancy. In addition to the Exchange 2010 bugcheck behaviors, Exchange 2013 Preview includes additional recovery behaviors for long IO times, excessive memory consumption by the Microsoft Exchange Replication service (MSExchangeRepl.exe), and severe cases where the system is in such a bad state, threads cannot be scheduled.

Even in JBOD environments, storage array controllers can have issues, such as crashing or hanging. Exchange 2010 included hung IO detection and recovery features that provided enhanced resilience. Exchange 2013 enhances server and storage resilience by including new behaviors for other serious conditions. These conditions and behaviors are described in the following table.

Name	Check	Action	Threshold
System Bad State	No threads, including non-managed threads, can be scheduled	Restart the server	302 seconds
Long IO times	IO operation latency measurements	Restart the server	41 seconds
Replication service memory use	Measure the working set of MSExchangeRepl.exe	Log event 4395 in the crimson channel with a service termination request Initiate termination of MSExchangeRepl.exe If service termination fails, restart the server	4 GB

Lagged Copy Enhancements

Lagged copy enhancements include integration with Safety Net and automatic play-down of log files in certain scenarios. Safety Net is a feature of transport that replaces the Exchange 2010 feature known as transport dumpster. Safety Net is similar to transport dumpster, in that it is a delivery queue that's associated with the Transport service on a Mailbox server. This queue stores copies of messages that were successfully delivered to the active mailbox database on the Mailbox server. Each active mailbox database on the Mailbox server has its own queue that stores copies of the delivered messages. You can specify how long Safety Net stores copies of the successfully delivered messages before they expire and are automatically deleted.

Safety Net takes also over some responsibility from shadow redundancy in DAG environments. In DAG environments, shadow redundancy doesn't need to keep another copy of the delivered message in a shadow queue while it waits for the delivered message to replicate to the passive copies of mailbox database on the other Mailbox servers in the DAG. The copy of the delivered message is already stored in Safety Net, so shadow redundancy can re-deliver the message from Safety Net if necessary.

With the introduction of Safety Net, activating a lagged database copy becomes significantly easier. Say for example, you have a lagged copy that has a 2 day replay lag. In that case you would also configure Safety Net for a period of 2 days. If you encounter a situation in which you need to use your lagged copy, you can suspend replication to it, and copy it twice (to preserve the lagged nature of the database and to create an extra copy in case you need it). Then, take a copy and throw away all of the log files, except for those in the required range. Mount the copy, and that will trigger an automatic request to Safety Net to redeliver the last two days of mail. No more hunting for where the point of corruption was introduced. You would get back the last two days’ worth of mail, minus of course, the data that is ordinarily lost on a lossy failover.

Lagged copies can now care for themselves by invoking automatic log replay to play down the log files in certain scenarios:

When a low disk space threshold is reached
When the lagged copy has physical corruption and needs to be page patched
When there are fewer than 3 available healthy copies (active or passive) for more than 24 hours

Lagged copy playdown behavior is disabled by default, and can be enabled by running the following command:

Set-DatabaseAvailabilityGroup <DAGName> –ReplayLagManagerEnabled $True

This enables playdown when there are fewer than X copies. The default value is 3, but number is configurable by modifying the following registry value:

HKLM\Software\Microsoft\ExchangeServer\v15\Replay\Parameters\ReplayLagManagerNumAvailableCopies

To enable playdown for low disk space thresholds, you must configure another registry entry:

HKLM\Software\Microsoft\ExchangeServer\v15\Replay\Parameters\ReplayLagPlayDownPercentDiskFreeSpace

Understanding Internet Database Availability Groups

2012-04-01T16:40:00Z

The high availability and site resilience platform built into Exchange 2010 provides automatic recovery from storage, network and server and other failures that affect active mailbox database copies. In addition, it provides manual recovery from failures affecting an entire site or datacenter.

Customers have asked for, and many have tried desperately to create, an Exchange 2010 architecture or design that provides automatic failover from datacenter-level events. At the time we released the RTM version of Exchange 2010, we strongly felt that manual recovery from events that affect an entire datacenter (for example, loss of power, fire, natural disaster, etc.) was preferable to automatic recovery by the system. After all, the system has no awareness of the nature of the failure and it can’t really make intelligent decisions as to whether or not users should have service and data access moved to an alternate datacenter.

But after more than five years of running Exchange in a cloud-based service environment in one form or another (Business Productivity Online Suite, Live@EDU, Office 365, etc.) we have realized that automatic recovery could be accomplished by the system in an intelligent way, provided the system had enough replication endpoints, and a sufficient number of points of Internet ingress and egress. So, we also started side development on a feature we internally called the “super DAG”. Obviously, using that name as the actual feature name does not work well for localization and other reasons, so we asked Marketing to come up with a more professional name.

The Internet Database Availability Group

Today, we are announcing the Internet database availability group (iDAG). The iDAG is an extension to the database availability group (DAG) component built into Exchange 2010. An iDAG is a group of an unlimited number of Mailbox servers that hosts an unlimited number of databases and provides automatic recovery from datacenter-level failure events.

Once a DAG has been extended with an iDAG, the iDAG becomes the new boundary for mailbox database replication, database and server switchovers, failovers, and a new internal component called Internet Active Manager (iAM). iAM, which runs on every server in a DAG, manages switchovers and failovers for iDAGs. Will iAM enable better management and recovery scenarios for your iDAG? Absolutely!

The iDAG overcomes several limitations of the traditional DAG:

It can contain more than 16 Mailbox servers. In fact, there is no limit to the number of Mailbox servers that can be added to an iDAG.
It can contain more than 1600 databases. In fact, there is no limit to the number of mailbox databases that can be hosted on a Mailbox server in an iDAG.
It uses HTTPS instead of TCP for log shipping, making it much more firewall-friendly.
It provides automatic recovery from all failures.

Any server in an iDAG can host a copy of a mailbox database from any other server in the same iDAG.

Internet Database Availability Group Lifecycle

As with DAGs, iDAGs leverage the Exchange 2010 feature known as incremental deployment, which is the ability to deploy redundant Mailbox servers and databases after Exchange is installed. After you deploy Exchange 2010, you create an iDAG, add Mailbox servers to the iDAG, and then replicate mailbox databases between the iDAG members.

Note: It is supported to create an iDAG that contains a combination of physical Mailbox servers and virtualized Mailbox servers, provided that the servers and solution comply with the Exchange 2010 System Requirements. As with all high availability configurations, you must ensure that all Mailbox servers in the iDAG are sized appropriately to handle the necessary workload during scheduled or unscheduled outages.

An iDAG is created by using the New-InternetDatabaseAvailabilityGroup cmdlet. An iDAG is initially created as an empty Active Directory object. This directory object is used to store relevant information about the iDAG, such as server membership information, HTTP URL and location. When you add the first server to an iDAG, an Internet-based failover cluster is automatically created. This failover cluster is used exclusively by the iDAG, however the cluster does not need to be dedicated to the DAG. You can use the cluster for any other purpose you like.

In addition to a failover cluster being created, the infrastructure that monitors the servers for network or server failures is initiated. The failover cluster heartbeat mechanism and cluster database are then used to track and manage information about the iDAG that can change quickly, such as database mount status, replication status, and last mounted location.

During creation, an iDAG is given a unique name (that must start with the letter “i” in lower case, and assigned three static IPv6 addresses (for redundancy). Specify a comma-separated list of IP addresses by using the InternetDatabaseAvailabilityGroupIPAddresses parameter.

Consider an iDAG that will have three servers. Two servers (EX1 and EX2) are on the same subnet, and the third server (EX3) is on a different subnet.

New-InternetDatabaseAvailabilityGroup -Name iDAG1
-InternetDatabaseAvailabilityGroupIPAddresses 2001:0:4137:1f9a:1037:1d90:b3e4:3e79, 2001:4898:80a8:f019:d8d6:250e:baf0:9393
Add-InternetDatabaseAvailabilityGroupServer -Identity iDAG1 –Server EX1
Add-InternetDatabaseAvailabilityGroupServer -Identity iDAG1 –Server EX2
Add-InternetDatabaseAvailabilityGroupServer -Identity iDAG1 –Server EX3

The cluster for iDAG1 is created when EX1 is added to the iDAG. During cluster creation, the Add-InternetDatabaseAvailabilityGroupServer cmdlet retrieves the IP addresses configured for the iDAG and ignores the ones that don't match any of the subnets found on EX1. In this example, the cluster for iDAG1 is created with an IP address of 2001:0:4137:1f9a:1037:1d90:b3e4:3e79, and 2001:4898:80a8:f019:d8d6:250e:baf0:9393 is ignored.

Then, EX2 is added, and the Add-InternetDatabaseAvailabilityGroupServer cmdlet again retrieves the IP addresses configured for the iDAG. There are no changes to the cluster's IP addresses because EX2 is on the same subnet as EX1.

Then, EX3 is added, and the Add-InternetDatabaseAvailabilityGroupServer cmdlet again retrieves the IP addresses configured for the DAG. Because a subnet matching 2001:4898:80a8:f019:d8d6:250e:baf0:9393 is present on EX3, the 2001:4898:80a8:f019:d8d6:250e:baf0:9393 address is added as an IP address resource in the cluster group. In addition, an OR dependency for the Network Name resource for each IP address resource is automatically configured. The 2001:4898:80a8:f019:d8d6:250e:baf0:9393 address will be used by the cluster when the cluster group moves to EX3.

This process is then repeated an unlimited number of times to extend the iDAG further and further into cyberspace.

Windows failover clustering registers the IP addresses for the cluster in the Domain Name System (DNS) when the Network Name resource is brought online. In addition, a cluster network object (CNO) is created in Active Directory. The name, IP addresses and CNO for the cluster are used only internally by the system to secure the iDAG and for internal communication purposes. Administrators and end users don't need to interface with or connect to the iDAG name or IP address for any reason.

In addition to a name and one or more IP addresses, the iDAG is also configured to use a witness server, witness directory, alternate witness server, and alternate witness directory. The witness server and witness directory are either automatically specified by the system, or they can be manually specified by the administrator. The alternate witness server and directory must be specified manually by the administrator. In addition, multiple alternate witness servers can be configured for increased redundancy. In the event of a datacenter failover, the iDAG will automatically reconfigure the witness server and alternate witness server for you.

By default, an iDAG is designed to use the built-in continuous replication feature to replicate mailbox databases among servers in the iDAG. If you're using third-party data replication that supports the Third Party Replication API in Exchange 2010, you must create the iDAG for use with third-party replication mode by using the New-InternetDatabaseAvailabilityGroup cmdlet with the ThirdPartyReplication parameter. After this mode is enabled, it can't be disabled.

After the iDAG is created, Mailbox servers can be added to the iDAG. When the first server is added to the iDAG, a cluster is formed for use by the iDAG. iDAGs make limited use of Windows failover clustering technology, such as the cluster heartbeat, cluster networks, and the cluster database (for storing data that changes, such as database state changes from active to passive or vice versa, or from mounted to dismounted and vice versa). As each subsequent server is added to the iDAG, it's joined to the underlying cluster, the cluster's quorum model is automatically adjusted by the system, and the server is added to the iDAG Active Directory object in.

After Mailbox servers are added to an iDAG, you can configure a variety of iDAG properties, such as whether to use network encryption or network compression for database replication within the iDAG. You can also configure iDAG networks and create additional iDAG networks.

After you add members to an iDAG and configure the iDAG, the active mailbox databases on each server can be replicated to the other iDAG members. After you create mailbox database copies, you can monitor the health and status of the copies using a variety of built-in monitoring tools. In addition, you can perform database and server switchovers.

Internet Database Availability Group Quorum Models

Underneath every iDAG is a Windows failover cluster. Failover clusters use the concept of quorum, which uses a consensus of voters to ensure that only one subset of the cluster members (which could mean all members or a majority of members) is functioning at one time. Quorum isn't a new concept for Exchange 2010. Highly available Mailbox servers in previous versions of Exchange also use failover clustering and its concept of quorum.

But what is new is the iDAGs use of a new quorum model called the Internet Node Majority and File Share Witness Model (iNMFSW). The iNMFSW is similar to traditional quorum models that require a minimum number of votes to maintain quorum. However, the minimum number of votes needed for quorum in an iDAG is one. This is because, unlike traditional DAGs (which are limited to using voting members and a single witness server), an iDAG uses all Internet-capable devices to maintain quorum. So unless the Internet is down or otherwise unavailable, an iDAG can never lose quorum. If quorum is lost, however, administrator intervention will be required to correct the quorum problem and restore iDAG operations.

Using an Internet Database Availability Group for High Availability

To illustrate how an iDAG can provide high availability and site resilience for your mailbox databases, consider an iDAG with five hundred members.

In this example, the database copies aren't mirrored across each server, but rather spread across hundreds of servers scattered across the Internet. This ensures that no two servers in the iDAG have the same set of database copies, providing the iDAG with greater resilience to failures, including failures that occur while other components are unavailable as a result of regular maintenance.

Consider the following scenario, using the preceding example iDAG, which illustrates resilience to multiple database and server failures.

Initially, all databases and servers are healthy. You need to install some operating system updates on EX212. You perform a server switchover, which activates the database copies on another Mailbox server somewhere on the Internet. A server switchover moves all active mailbox database copies from their current server to one or more other Mailbox servers in the iDAG in preparation for a scheduled outage for the current server. You can perform a server switchover quickly by running the following command in the Exchange Management Shell.

Move-ActiveMailboxDatabase -Server EX212

In this example, all of the active mailbox databases on EX212 are moved. By omitting the ActivateOnServer parameter in the preceding command, you chose to have the system select the best possible new active copy.

While you perform maintenance on EX212, EX306 experiences a catastrophic hardware failure and goes offline. Prior to going offline, EX306 active database copies. To recover from the failure, the system automatically activates the copies on alternate servers in the iDAG within 30 seconds.

After the scheduled maintenance is completed for EX212, you bring the server online. As soon as EX212 is available, the other members of the iDAG are notified, and the database copies hosted on EX212 are automatically synchronized with the active copy of each database.

After the failed hardware component in EX306 is replaced with a new component, EX306 is brought online. After EX306 is available, the other members of the iDAG are notified, and the databases hosted on EX306 are automatically synchronized with the active copy of each database.

Using an Internet Database Availability Group for Site Resilience

In addition to providing high availability within a datacenter, an iDAG can also be extended an unlimited number of datacenters in a configuration that provides site resilience all datacenters. Incremental deployment can be used to extend any iDAG to any datacenter by deploying Mailbox servers and the necessary supporting resources.

Using Multiple Internet Database Availability Groups for Site Resilience

Prior to iDAGs, in order to achieve site resilience for multiple locations, you had to use multiple DAGs. With an iDAG, this is no longer necessary. A single iDAG can be extended across an unlimited number of datacenters, providing site resilience for all of your locations and databases. When using a single iDAG to provide site resilience in an environment where each datacenter to which you extend the DAG has an active user population, the Internet eliminates all single points of failure because quorum no longer requires a majority of the voters to be active and able to communicate with each other. It simply requires connectivity to another iDAG member.

Client Experience When Using Internet Database Availability Groups

As mentioned above, iDAGs can be used to provide both high availability and site resilience. The client experience when using an iDAG no longer depends on the type and version of the client and the protocol used by the client to access mailbox data. For example, if a datacenter failover occurs, the behavior and reconnection logic used by an Exchange ActiveSync, POP3, or IMAP4 client is the same as the behavior and reconnection logic used by Microsoft Outlook clients.

TechEd 2012 Registration Time!

2012-01-24T20:58:32Z

Registration is now open for both TechEd North America 2012 and TechEd Europe 2012!

Celebrate 20 years of Microsoft TechEd North America! TechEd is Microsoft's premier technology conference for IT professionals and developers, offering the most comprehensive technical education across Microsoft's current and soon-to-be-released suite of products, solutions, tools, and services. Our four day event in Orlando, Florida offers hands-on learning, deep product exploration and countless opportunities to connect with a community of industry and Microsoft experts.

There are many reasons why you can't miss TechEd 2012. If you are developing, deploying, managing, securing and mobilizing Microsoft solutions, TechEd is your best opportunity to focus on the key technology and business issues that will help you solve today's real-world IT challenges and prepare you for tomorrow's innovations.

Year after year, technology professionals gather at Microsoft TechEd North America to immerse themselves in more than 1,000 learning opportunities that cover an unparalleled breadth and depth of Microsoft technologies. TechEd is the best place as an industry professional to focus on technology education and professional development opportunities that are relevant to you and your business without all the distractions of the office. Meeting face-to-face with Microsoft and industry experts can start relationships that will help your work for years to come.

Join myTechEd to network with the brightest people in the industry, and then register for the 20th Annual TechEd!

Come to TechEd Europe, unplug from your day job, & dive into the thousands of learning & networking opportunities. In addition to over 400 sessions with Microsoft and industry speakers, TechEd offers you face-to-face connections with thousands of your peers who share your challenges. TechEd is the forum to gain the expertise and insights that will help you get the most from your IT investments.
TechEd offers 4 days to:

LEARN in-depth about the latest technology trends and how you can leverage these effectively in your business
DISCOVER the future of Microsoft’s products, technologies, solutions and services directly from Microsoft’s leaders with news, announcements, and demos
NETWORK with Microsoft and industry thought leaders, and fellow delegates that share your technology interests and business challenges
PLAN the features and architecture to support your business goals and product roadmap

Need Even More? Attend a Pre-Conference Seminar

By arriving a day early on Monday, 25 June and registering for the special Pre-Conference Seminars you will get in-depth training and insights on the Microsoft technologies and products that power your business. Select from 10 different topics taught by John Craddock, Kate Gregory, Steve Fox, Mikael Nystrom, Kent Agerlund, Alberto Ferrari, Richard Hundhausen, and more!

Register NOW for TechEd Europe (limited number of academic tickets available for students and educators)

Free Microsoft Private Cloud Training

2012-01-19T19:23:51Z

On February 21 & 22 we are running a 2-day virtual training event to help the world learn about the upcoming enhancements with Creating & Managing a Private Cloud with System Center 2012 Jump Start. It is 100% free so register now!

Event Overview

Adopting this exciting new computing paradigm provides a whole new landscape of technology and career direction for IT professionals. Microsoft Learning and the Microsoft System Center 2012 team have partnered to bring you an exciting opportunity to learn what you need to know to deploy, manage and maintain Microsoft’s private cloud solution. Leveraging the popular Jump Start virtual classroom approach, the industry’s most gifted cloud experts will show attendees why this new private cloud solution, based on System Center 2012 and Windows Server, has garnered so much attention. Presenters include Symon Perriman, Sean Christensen, Adam Hall, Kenon Owens, Prabu Rambadran & Chris Van Wesep and there will be a live Q&A during the event.

Event Agenda

Day 1: Deployment & Configuration (Feb. 21)

Part 1: Understanding the Microsoft Private Cloud
Part 2: Deploying the Infrastructure Components
Part 3: Deploying the Private Cloud Infrastructure
Part 4: Deploying the Service Layer
Part 5: Deploying the Applications & VMs

Day 2: Management & Operations (Feb. 22)

Part 6: Managing the Infrastructure Components
Part 7: Managing the Private Cloud Infrastructure
Part 8: Managing the Service Layer
Part 9: Managing the Applications & VMs

Recommended Windows Hotfix for Database Availability Groups running Windows Server 2008 R2

2011-11-21T16:12:35Z

I just posted an entry on the Exchange Team Blog about a Windows hotfix that is strongly recommended for database availability groups that are deployed on Windows Server 2008 R2. Go check it out and install the hotfix, if applicable.

TechNet Print-a-Book Beta Available

2011-07-21T19:23:46Z

This week, a feature known as Print/Export a Collection (a.k.a. Print-a-Book) went live in our TechNet Library Lightweight as Beta service. This new feature offers improved printing features for TechNet articles and the ability to export one or more topics to PDF format.

To check out this feature, go to the TechNet Lightweight Library and click on the printer icon in the upper right-hand corner.

Enjoy!