In order for any database availability group (DAG) that has Datacenter Activation Coordination (DAC) mode to be able to mount any databases, the DAG itself much reach automount consensus. In the following blog posts, I discuss and outline situations where automount consensus may not be achieved, resulting in dismounted databases. A DAG in DAC mode which cannot achieve automount consensus will not automatically mount databases, and administrators are blocked from manually mounting them, as well.
Let’s look at an example to illustrate this scenario. In this example, I have a 4-member DAG comprised of servers DAG-1, DAG-2, DAG-3, and DAG-4
DAG-4 experiences a hardware failure, goes down, and is scheduled for replacement. Although the server is offline, it is still a member of the DAG and it’s underlying cluster.
Next, a power failure occurs resulting in DAG-1, DAG-2, and DAG-3 going offline.
After power is restored, DAG-1, DAG-2, and DAG-3 are booted up.
The cluster service is successfully initialized and a cluster is formed. You can verify the nodes are operational by running
PS C:\> cluster node Listing status for all available nodes:
Node Node ID Status -------------- ------- --------------------- DAG-4 1 Down DAG-3 2 Up DAG-2 3 Up DAG-1 4 Up
Windows 2008 R2:
PS C:\> Import-Module FailoverClusters
PS C:\> Get-ClusterNode | fl name,state
Name : DAG-1 State : Up
Name : DAG-2 State : Up
Name : DAG-3 State : Up
Name : DAG-4 State : Down
Using the Get-MailboxDatabase cmdlet, we can see that, even though the servers are up and the cluster is running, no databases are mounted:
[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted
Name : Mailbox Database 1252068500 Mounted : False
Name : Mailbox Database 1757981393 Mounted : False
Name : Mailbox Database 1370762657 Mounted : False
Name : Mailbox Database 1511135053 Mounted : False
Attempting to mount a database (with or without the –Force parameter) results in the following error:
[PS] C:\>Mount-Database "Mailbox Database 1252068500" Couldn't mount the database that you specified. Specified database: Mailbox Database 1252068500; Error code: An Active Manager operation failed. Error An Active Manager operation encountered an error. To perform this operation, the server must be a member of a database availability group, and the database availability group must have quorum. Error: Automount consensus not reached.. [Server: DAG-1.exchange.msft]. + CategoryInfo : InvalidOperation: (Mailbox Database 1252068500:ADObjectId) [Mount-Database], InvalidOperationException + FullyQualifiedErrorId : FD5FDFE8,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase
So what needs to be done to achieve automount consensus and mount the databases? It’s simple: the administrator can run Start-DatabaseAvailabilityGroup. The cmdlet can be run using the –MailboxServer parameter to specify one of the mailbox servers that is functional. After the cmdlet is run, the specified server will have its DACP bit set to 1. The other servers in the DAG will see that the DACP bit on the specified server is set to 1 and this will cause them to set their DACP bits to 1. And once all of the DAG members have DACP set to 1, automount consensus is reached, and the databases can be mounted. Here is an example:
[PS] C:\>Start-DatabaseAvailabilityGroup -Identity DAG -MailboxServer MBX-1
Name : Mailbox Database 1252068500 Mounted : True
Name : Mailbox Database 1757981393 Mounted : True
Name : Mailbox Database 1370762657 Mounted : True
Name : Mailbox Database 1511135053 Mounted : True
Start-DatabaseAvailabilityGroup can also be used in scenarios where the remaining survivor of a DAG failure is a single member that is using the witness server boot time to achieve automount consensus. In this case the cmdlet forces the member’s DACP bit to 1 and updates the boot times in the registry for future restarts.
As a side note of caution, administrators should be careful when using Start-DatabaseAvailabilityGroup. It is possible that a split brain condition could be created by using this cmdlet to start the wrong DAG members, or it by using it at the wrong stage of a datacenter switchover. To ensure your datacenter switchovers go smoothly, I recommend using the recently released Datacenter Switchover Tool.
Datacenter Activation Coordination Series:
Part 1: My databases do not mount automatically after I enabled Datacenter Activation Coordination (http://aka.ms/F6k65e) Part 2: Datacenter Activation Coordination and the File Share Witness (http://aka.ms/Wsesft) Part 3: Datacenter Activation Coordination and the Single Node Cluster (http://aka.ms/N3ktdy) Part 4: Datacenter Activation Coordination and the Prevention of Split Brain (http://aka.ms/C13ptq) Part 5: Datacenter Activation Coordination: How do I Force Automount Concensus? (http://aka.ms/T5sgqa) Part 6: Datacenter Activation Coordination: Who has a say? (http://aka.ms/W51h6n) Part 7: Datacenter Activation Coordination: When to run start-databaseavailabilitygroup to bring members back into the DAG after a datacenter switchover. (http://aka.ms/Oieqqp) Part 8: Datacenter Activation Coordination: Stop! In the Name of DAG... (http://aka.ms/Uzogbq) Part 9: Datacenter Activation Coordination: An error cause a change in the current set of domain controllers (http://aka.ms/Qlt035)
Thanks a ton. u cant imagine how did your article saved me today. i was lost and all the DBs were dismounted.
following ur artcile i was able to bring my DAG back online.
Thanks & great article
I'm glad you enjoyed it.
Thanks Buddy. That was awesome.