In both Exchange 2010 and Exchange 2013, customers can deploy one or more replication networks in a database availability group (DAG). There can be many reasons for using a replication network, but in most cases they are used to provide a dedicated log shipping channel between members of the same DAG.
As documented on TechNet, when a replication network fails, replication should automatically failover to the DAG’s MAPI network:
· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.150).aspx (Exchange 2013)
· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.141).aspx (Exchange 2010)
“In the event of a failure affecting the Replication network, if the MAPI network is unaffected by the failure, log shipping and seeding operations will revert to use the MAPI network, even if the MAPI network has it's ReplicationEnabled property set to False.”
Log shipping operations occur by connecting to the Microsoft Exchange Replication service on the server that hosts the active database copy, on TCP port 64327 using a random ephemeral port on the passive server. Log files are then pushed from the active server to the passive server via this channel. There can exist several issues that result in this log shipping channel being interrupted. For example, a firewall may block port 64327, static routes may be missing on multi-subnet replication networks, or an intermediary network device may not route traffic correctly.
In the following example, we have a 4-member DAG running Exchange 2013. There is a single active database that is replicated to three other servers.
The DAG has two networks that each has two subnets. These networks represent the MAPI and Replication networks for the DAG. Automatic network detection in Exchange 2013 was disabled for this example.
RunspaceId : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name : MapiDagNetwork
Description :
Subnets : {{192.168.0.0/24,Up}, {192.168.1.0/24,Up}}
Interfaces : {{MBX-1,Up,192.168.0.11}, {MBX-2,Up,192.168.0.12}, {MBX-3,Up,192.168.1.11},
{MBX-4,Up,192.168.1.12}}
MapiAccessEnabled : True
ReplicationEnabled : True
IgnoreNetwork : False
Identity : DAG\MapiDagNetwork
IsValid : True
ObjectState : New
RunspaceId : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name : ReplicationDagNetwork01
Description :
Subnets : {{10.0.1.0/24,Up}, {10.0.0.0/24,Up}}
Interfaces : {{MBX-1,Up,10.0.0.1}, {MBX-2,Up,10.0.0.2}, {MBX-3,Up,10.0.1.1}, {MBX-4,Up,10.0.1.2}}
MapiAccessEnabled : False
ReplicationEnabled : True
IgnoreNetwork : False
Identity : DAG\ReplicationDagNetwork01
IsValid : True
ObjectState : New
Using Get-MailboxDatabaseCopyStatus with the –ConnectionStatus switch, we can verify that the Replication network is currently in use:
[PS] C:\>Get-MailboxDatabaseCopyStatus * -connectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections
Name : DAG-DB0\MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : DAG-DB0\MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-4
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
The router servicing the network link between the two subnets of the Replication network is shutdown. This will block replication from succeeding over the Replication network. What happens to the database copies?
[PS] C:\>Get-MailboxDatabaseCopyStatus * | fl name,status
Name : DAG-DB0\MBX-1
Status : Mounted
Name : DAG-DB0\MBX-2
Status : Healthy
Name : DAG-DB0\MBX-3
Status : DisconnectedAndHealthy
Name : DAG-DB0\MBX-4
Status : DisconnectedAndHealthy
In this example, the database copies enter a DisconnectedAndHealthy state. Reviewing the connection status, we note that the connection has timed out between the servers; this is expected since the route is down.
[PS] C:\>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections
Name : DAG-DB0\MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections : {{MBX-2,ReplicationDagNetwork01}}
Name : DAG-DB0\MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
could not be completed within 15 seconds."}
OutgoingConnections :
Name : DAG-DB0\MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
could not be completed within 15 seconds."}
OutgoingConnections :
Why didn’t the Replication network failover to the MAPI network in this case? For one thing, the ability to establish a log shipping channel is not one of the criteria that the Replication service uses to determine the health of a given network. The Replication service relies on feedback from the Cluster service regarding individual network interface status in order to determine the health of a log shipping channel.
For each subnet that exists on a DAG member, an associated cluster network is created. In this example, there are 4 subnets and therefore there are 4 cluster networks.
[PS] C:\>Get-ClusterNetwork | fl
Name : Cluster Network 1
State : Up
Name : Cluster Network 2
State : Up
Name : Cluster Network 3
State : Up
Name : Cluster Network 4
State : Up
The interface associated with each of these subnets is included in the appropriate cluster network. Each of these interfaces has a status based on status reporting in Windows Failover Clustering.
[PS] C:\>Get-ClusterNetworkInterface | fl
Name : MBX-1 - LAN-A
Node : MBX-1
Network : Cluster Network 1
State : Up
Name : MBX-2 - LAN-A
Node : MBX-2
Network : Cluster Network 1
State : Up
Name : MBX-1 - REPL-A
Node : MBX-1
Network : Cluster Network 2
State : Up
Name : MBX-2 - REPL-A
Node : MBX-2
Network : Cluster Network 2
State : Up
Name : MBX-3 - REPL-B
Node : MBX-3
Network : Cluster Network 3
State : Up
Name : MBX-4 - REPL-B
Node : MBX-4
Network : Cluster Network 3
State : Up
Name : MBX-3 - LAN-B
Node : MBX-3
Network : Cluster Network 4
State : Up
Name : MBX-4 - LAN-B
Node : MBX-4
Network : Cluster Network 4
State : Up
In this scenario, the Cluster service considers all of the interfaces as “UP.” Since the interfaces are UP, the Replication service does not failover to the MAPI network even though replication cannot occur over the replication network.
If the Cluster service reports a network as “FAILED” the behavior is different. On a server hosting a passive database copy, the network cable was removed from the Replication network interface, causing the cluster to mark that interfaces as failed:
[PS] C:\>Get-ClusterNetworkInterface | fl
Name : MBX-1 - LAN-A
Node : MBX-1
Network : Cluster Network 1
State : Up
Name : MBX-2 - LAN-A
Node : MBX-2
Network : Cluster Network 1
State : Up
Name : MBX-1 - REPL-A
Node : MBX-1
Network : Cluster Network 2
State : Up
Name : MBX-2 - REPL-A
Node : MBX-2
Network : Cluster Network 2
State : Up
Name : MBX-3 - REPL-B
Node : MBX-3
Network : Cluster Network 3
State : Up
Name : MBX-4 - REPL-B
Node : MBX-4
Network : Cluster Network 3
State : Failed
Name : MBX-3 - LAN-B
Node : MBX-3
Network : Cluster Network 4
State : Up
Name : MBX-4 - LAN-B
Node : MBX-4
Network : Cluster Network 4
State : Up
The copy status for the databases hosted on MBX-4 is healthy.
[PS] C:\>Get-MailboxDatabaseCopyStatus *
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
DAG-DB0\MBX-1 Mounted 0 0 Healthy
DAG-DB0\MBX-2 Healthy 0 0 4/29/2014 2:11:47 PM Healthy
DAG-DB0\MBX-3 Healthy 0 0 4/29/2014 2:11:47 PM Healthy
DAG-DB0\MBX-4 Healthy 0 0 4/29/2014 2:11:47 PM Healthy
When reviewing the connection status for the databases on MBX-4, we see that the Replication service is using the MAPI network for log shipping. Servers that have no issues with the Replication interface continue to use that interface.
[PS] C:\>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections
Name : DAG-DB0\MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : DAG-DB0\MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-4
IncomingLogCopyingNetwork : {MBX-1,MapiDagNetwork}
OutgoingConnections :
When the interface was marked as failed, the Replication service successfully failed over to the MAPI network.
But what happens if the interface is not FAILED, but the interface configuration is invalid? In this event, the Cluster service cannot pass cluster traffic between these two interfaces and the associated network is marked as partitioned.
[PS] C:\>Get-ClusterNetwork | fl
Name : Cluster Network 1
State : Up
Name : Cluster Network 2
State : Up
Name : Cluster Network 3
State : Partitioned
Name : Cluster Network 4
State : Up
The interfaces within the partitioned network are marked as unreachable.
[PS] C:\>Get-ClusterNetworkInterface | fl
Name : MBX-1 - LAN-A
Node : MBX-1
Network : Cluster Network 1
State : Up
Name : MBX-2 - LAN-A
Node : MBX-2
Network : Cluster Network 1
State : Up
Name : MBX-1 - REPL-A
Node : MBX-1
Network : Cluster Network 2
State : Up
Name : MBX-2 - REPL-A
Node : MBX-2
Network : Cluster Network 2
State : Up
Name : MBX-3 - REPL-B
Node : MBX-3
Network : Cluster Network 3
State : Unreachable
Name : MBX-4 - REPL-B
Node : MBX-4
Network : Cluster Network 3
State : Unreachable
Name : MBX-3 - LAN-B
Node : MBX-3
Network : Cluster Network 4
State : Up
Name : MBX-4 - LAN-B
Node : MBX-4
Network : Cluster Network 4
State : Up
What happens to the copies that are hosted on the servers with an unreachable interface status?
[PS] C:\>Get-MailboxDatabaseCopyStatus * | fl name,status
Name : DAG-DB0\MBX-1
Status : Mounted
Name : DAG-DB0\MBX-2
Status : Healthy
Name : DAG-DB0\MBX-3
Status : DisconnectedAndHealthy
Name : DAG-DB0\MBX-4
Status : DisconnectedAndHealthy
An unreachable interface is not the same as a failed interface. This resulted in the databases entering a disconnected state and replication not failing over to the MAPI network. The connection status confirms this failure.
[PS] C:\>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections
Name : DAG-DB0\MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : DAG-DB0\MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections :
Name : DAG-DB0\MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
could not be completed within 15 seconds."}
OutgoingConnections :
Name : DAG-DB0\MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
could not be completed within 15 seconds."}
OutgoingConnections :
The network failure detection mechanism is the same in both Exchange 2010 and Exchange 2013, as well as the same in Windows 2008 R2, Windows 2012, and Windows 2012 R2 failover clusters. In order for the Replication service to detect a failure of a Replication network, the operating system and Cluster service must report the underlying interface as failed. If the Cluster service reports any other status for the interface, the Replication service will consider the network to be valid and replication will not failover to another network.
action on the toolbar.
to expose the Insert HyperLink button. 
