Tim McMichael

Navigating the world of high availability...and occasionally sticking my head in the cloud...

Exchange 2010: Remove-databaseavailabilitygroupserver–configurationOnly does not evict the member from the cluster.

Exchange 2010: Remove-databaseavailabilitygroupserver–configurationOnly does not evict the member from the cluster.

  • Comments 2
  • Likes

Administrators may encounter conditions where DAG members cannot be gracefully removed from a database availability group.  For example, a member server may have encountered an unrecoverable failure or the server may need to be removed from the DAG in order to perform a server recovery.

 

In order to account for these and similar conditions the remove-databaseavailabilitygroupserver –configurationOnly command exists.  This command, when utilized, simply removes the member from the Database Availability Groups Active Directory object.

 

Here is an example…

 

Using get-databaseavailabilitygroup –status | fl name,servers,operationalservers the membership of the DAG can be verified.  In this example the only operational server is MBX-1 since that is the only server currently running in the DAG.

 

[PS] C:\>Get-DatabaseAvailabilityGroup DAG -status | fl name,Servers,OperationalServers


Name               : DAG
Servers            : {MBX-1, MBX-2}
OperationalServers : {MBX-1}

 

Using the remove-databaseavailabilitygroupserver –configurationOnly command a DAG member can be removed.

 

[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer MBX-2 -ConfigurationOnly

Confirm
Are you sure you want to perform this action?
Removing Mailbox server "MBX-2" from database availability group "DAG".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a

 

The results of the command can be verified using get-databaseavailabilitygroup –status | fl name,servers,operationalservers:

 

[PS] C:\>Get-DatabaseAvailabilityGroup DAG -status | fl name,Servers,OperationalServers


Name : DAG
Servers : {MBX-1}
OperationalServers : {MBX-1}

 

When a server is removed from the DAG in this manner it is not evicted from the corresponding cluster.  You can verify cluster membership using the built in cluster commands.  Here is an example from this test:

 

( Windows 2008 / Windows 2008 R2 )

 

[PS] C:\>cluster.exe node
Listing status for all available nodes:

Node           Node ID Status
-------------- ------- ---------------------
MBX-1                1 Up
MBX-2                2 Down

( Windows 2008 R2 )

[PS] C:\>Import-Module FailoverClusters

[PS] C:\>Get-ClusterNode

Name                                State
----                                -----
mbx-1                                  Up
mbx-2                                Down

 

In general this issue surfaces when administrators complete a server rebuild operation and note that the rebuilt node cannot be added back to the cluster because it already exists in the cluster.  Here is an example:

 

[PS] C:\>Add-DatabaseAvailabilityGroupServer –identity DAG –mailboxServer MBX-2


WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
"C:\ExchangeSetupLogs\DagTasks\dagtask_2012-06-24_14-51-47.841_add-databaseavailabiltygroupserver.log".
A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Node mbx-2 is already joined to a cluster. [Server: MBX-1.domain.com]
    + CategoryInfo          : InvalidArgument: (:) [Add-DatabaseAvailabilityGroupServer], DagTaskOperationFailedException
    + FullyQualifiedErrorId : D05F37CD,Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer

 

 

When using the remove-databaseavailabilitygroupserver –configurationOnly administrators must remove the node from the cluster.  This can be accomplished through two methods:

 

( Windows 2008 / Windows 2008 R2 )

 

Administrators may utilize Failover Cluster Manager.  After connecting to the cluster servicing the Database Availability Group the nodes hive can be expanded.  The administrator can right click on the node that was removed –> select more actions –> evict

 

image

 

( Windows 2008 R2 )


[PS] C:\>Import-Module FailoverClusters

[PS] C:\>Remove-ClusterNode MBX-2

Remove-ClusterNode
Are you sure you want to evict node mbx-2?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y
Remove-ClusterNode : The cluster node 'MBX-2' was evicted from the cluster, but was not fully cleaned up.  Please see the Failover Clustering application event log on node MBX-2 for more information.
    The RPC server is unavailable
At line:1 char:19
+ Remove-ClusterNode <<<<  MBX-2
    + CategoryInfo          : NotSpecified: (:) [Remove-ClusterNode], ClusterCmdletException
    + FullyQualifiedErrorId : Remove-ClusterNode,Microsoft.FailoverClusters.PowerShell.RemoveClusterNodeCommand

 

(Note:  The RPC error is expected as the command attempts to cleanup the local cluster configuration on the node but the node is not accessible)

 

After cleaning up the cluster configuration the administrator can run set-databaseavailabilitygroup –identity <DAGNAME> to ensure the appropriate cluster configuration is utilized.

Comments
  • Great article! Thanks for taking the time to put this together and for sharing it with the rest of the Exchange Community!

    Best Regards,

    FT

  • Excellent Article

    I have DAG with 4 nodes on the primary site and 3 nodes on the dr site
    I need to shutdown all 4 nodes on the primary site at the same time for maintenance
    I assume that there is a way of doing that without having to use classic DAG Disaster Recovery (and haing the whole DAG going down) ?

    If I use the procedure described in your article to evict the 4 servers of the primary site , will it be easy to add them back (reseed all databases... ) ?

    Thanks

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment