Tim McMichael

Navigating the world of high availability...and occasionally sticking my head in the cloud...

Exchange 2010: Remove-databaseavailabilitygroupserver–configurationOnly does not evict the member from the cluster.

Exchange 2010: Remove-databaseavailabilitygroupserver–configurationOnly does not evict the member from the cluster.

  • Comments 6
  • Likes

Administrators may encounter conditions where DAG members cannot be gracefully removed from a database availability group.  For example, a member server may have encountered an unrecoverable failure or the server may need to be removed from the DAG in order to perform a server recovery.


In order to account for these and similar conditions the remove-databaseavailabilitygroupserver –configurationOnly command exists.  This command, when utilized, simply removes the member from the Database Availability Groups Active Directory object.


Here is an example…


Using get-databaseavailabilitygroup –status | fl name,servers,operationalservers the membership of the DAG can be verified.  In this example the only operational server is MBX-1 since that is the only server currently running in the DAG.


[PS] C:\>Get-DatabaseAvailabilityGroup DAG -status | fl name,Servers,OperationalServers

Name               : DAG
Servers            : {MBX-1, MBX-2}
OperationalServers : {MBX-1}


Using the remove-databaseavailabilitygroupserver –configurationOnly command a DAG member can be removed.


[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer MBX-2 -ConfigurationOnly

Are you sure you want to perform this action?
Removing Mailbox server "MBX-2" from database availability group "DAG".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a


The results of the command can be verified using get-databaseavailabilitygroup –status | fl name,servers,operationalservers:


[PS] C:\>Get-DatabaseAvailabilityGroup DAG -status | fl name,Servers,OperationalServers

Name : DAG
Servers : {MBX-1}
OperationalServers : {MBX-1}


When a server is removed from the DAG in this manner it is not evicted from the corresponding cluster.  You can verify cluster membership using the built in cluster commands.  Here is an example from this test:


( Windows 2008 / Windows 2008 R2 )


[PS] C:\>cluster.exe node
Listing status for all available nodes:

Node           Node ID Status
-------------- ------- ---------------------
MBX-1                1 Up
MBX-2                2 Down

( Windows 2008 R2 )

[PS] C:\>Import-Module FailoverClusters

[PS] C:\>Get-ClusterNode

Name                                State
----                                -----
mbx-1                                  Up
mbx-2                                Down


In general this issue surfaces when administrators complete a server rebuild operation and note that the rebuilt node cannot be added back to the cluster because it already exists in the cluster.  Here is an example:


[PS] C:\>Add-DatabaseAvailabilityGroupServer –identity DAG –mailboxServer MBX-2

WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Node mbx-2 is already joined to a cluster. [Server: MBX-1.domain.com]
    + CategoryInfo          : InvalidArgument: (:) [Add-DatabaseAvailabilityGroupServer], DagTaskOperationFailedException
    + FullyQualifiedErrorId : D05F37CD,Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer



When using the remove-databaseavailabilitygroupserver –configurationOnly administrators must remove the node from the cluster.  This can be accomplished through two methods:


( Windows 2008 / Windows 2008 R2 )


Administrators may utilize Failover Cluster Manager.  After connecting to the cluster servicing the Database Availability Group the nodes hive can be expanded.  The administrator can right click on the node that was removed –> select more actions –> evict




( Windows 2008 R2 )

[PS] C:\>Import-Module FailoverClusters

[PS] C:\>Remove-ClusterNode MBX-2

Are you sure you want to evict node mbx-2?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y
Remove-ClusterNode : The cluster node 'MBX-2' was evicted from the cluster, but was not fully cleaned up.  Please see the Failover Clustering application event log on node MBX-2 for more information.
    The RPC server is unavailable
At line:1 char:19
+ Remove-ClusterNode <<<<  MBX-2
    + CategoryInfo          : NotSpecified: (:) [Remove-ClusterNode], ClusterCmdletException
    + FullyQualifiedErrorId : Remove-ClusterNode,Microsoft.FailoverClusters.PowerShell.RemoveClusterNodeCommand


(Note:  The RPC error is expected as the command attempts to cleanup the local cluster configuration on the node but the node is not accessible)


After cleaning up the cluster configuration the administrator can run set-databaseavailabilitygroup –identity <DAGNAME> to ensure the appropriate cluster configuration is utilized.

  • Great article! Thanks for taking the time to put this together and for sharing it with the rest of the Exchange Community!

    Best Regards,


  • Excellent Article

    I have DAG with 4 nodes on the primary site and 3 nodes on the dr site
    I need to shutdown all 4 nodes on the primary site at the same time for maintenance
    I assume that there is a way of doing that without having to use classic DAG Disaster Recovery (and haing the whole DAG going down) ?

    If I use the procedure described in your article to evict the 4 servers of the primary site , will it be easy to add them back (reseed all databases... ) ?


  • Thanks for this, really saved my bacon!

  • go back to profill

  • Great article! Wondering if you could comment on a scenario. Take a server that's going to be down for a while, but isn't going to be otherwise rebuilt, maybe we need to wait some time for parts or something. The server is part of a 4 node DAG and in the primary data center, so for the long duration of the outage we don't have Quorum in the primary data center alone, a WAN outage could bring down production! I add another server to the DAG to get an extra vote, but that switches the quorum model to "Majority Node" effectively removing the witness vote. The issue is I still don't have Quorum in the primary data center for the duration of the outage, and would have to add yet another server to get it. There are 2 questions I garnish from this:

    1) Is there any way to force the Quorum model to "Majority Node & Witness" regardless of node count?
    2) If no to number 1, does adding the second server at least give me a quick way to re-establish quorum by using Stop-Database AvailabilityGroup -MailboxServer or is there another take like using Remove-DatabaseAvailabilityGroupServer -Configuration Only as you've mentioned above.

    Let me know what you think. Thanks

  • @Steven:

    This is always a tough situation. There is no way to force the quorum model to change as Exchange enforces it and it's not supported to do so outside of Exchange.

    Stop and remove will not help much here overall, as it would just do what adding or removing the node is.

    You may have to consider adding additional voter nodes.


Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment