In recent weeks I’ve had several peers and customers present with concerns that when using Local Continuous Replication (LCR), Cluster Continuous Replication (CCR), Standby Continuous Replication (SCR), of the Database Availability Group (DAG) passive database copies appear in a dirty shutdown state.
In general a database has two states – Clean Shutdown and Dirty Shutdown. A clean shutdown state indicates that the database has had all outstanding transaction logs committed to it and the database can now stand on its own. A dirty shutdown database indicates that the database requires additional transaction logs to be committed in order for the database to stand on its own.
Here is a eseutil /mh dump of a database that has been dismounted and therefore should be in a clean shutdown state.
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 08.03
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode... Database: MBX-1-SG1-DB1.edb
File Type: Database Format ulMagic: 0x89abcdef Engine ulMagic: 0x89abcdef Format ulVersion: 0x620,12 Engine ulVersion: 0x620,12 Created ulVersion: 0x620,12 DB Signature: Create time:06/30/2010 09:18:04 Rand:205158 Computer: cbDbPage: 8192 dbtime: 49856 (0xc2c0) State: Clean Shutdown Log Required: 0-0 (0x0-0x0) Log Committed: 0-0 (0x0-0x0) Streaming File: No Shadowed: Yes Last Objid: 135 Scrub Dbtime: 0 (0x0) Scrub Date: 00/00/1900 00:00:00 Repair Count: 0 Repair Date: 00/00/1900 00:00:00 Old Repair Count: 0 Last Consistent: (0x30,102,19E) 12/05/2010 11:33:55 Last Attach: (0x2F,9,86) 12/05/2010 09:23:20 Last Detach: (0x30,102,19E) 12/05/2010 11:33:55 Dbid: 1 Log Signature: Create time:06/30/2010 09:18:03 Rand:202227 Computer: OS Version: (6.0.6002 SP 2 NLS 500100.50100)
Previous Full Backup: Log Gen: 15-32 (0xf-0x20) - OSSnapshot Mark: (0x21,8,16) Mark: 07/06/2010 21:08:21
Previous Incremental Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00
Previous Copy Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00
Previous Differential Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00
Current Full Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00
Current Shadow copy backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00
cpgUpgrade55Format: 0 cpgUpgradeFreePages: 0 cpgUpgradeSpaceMapPages: 0
ECC Fix Success Count: none Old ECC Fix Success Count: none ECC Fix Error Count: none Old ECC Fix Error Count: none Bad Checksum Error Count: none Old bad Checksum Error Count: none
Operation completed successfully in 0.15 seconds.
This particular database is enabled for LCR. With the source database mounted and with storage group copy suspended, here is an eseutil /mh dump of the database. (We should expect the same for a CCR passive, SCR passive, or DAG passive database)
File Type: Database Format ulMagic: 0x89abcdef Engine ulMagic: 0x89abcdef Format ulVersion: 0x620,12 Engine ulVersion: 0x620,12 Created ulVersion: 0x620,12 DB Signature: Create time:06/30/2010 09:18:04 Rand:205158 Computer: cbDbPage: 8192 dbtime: 49856 (0xc2c0) State: Dirty Shutdown Log Required: 50-50 (0x32-0x32) Log Committed: 0-50 (0x0-0x32) Streaming File: No Shadowed: Yes Last Objid: 135 Scrub Dbtime: 0 (0x0) Scrub Date: 00/00/1900 00:00:00 Repair Count: 0 Repair Date: 00/00/1900 00:00:00 Old Repair Count: 0 Last Consistent: (0x30,102,19E) 12/05/2010 11:33:56 Last Attach: (0x32,9,86) 12/05/2010 11:38:05 Last Detach: (0x0,0,0) 00/00/1900 00:00:00 Dbid: 1 Log Signature: Create time:06/30/2010 09:18:03 Rand:202227 Computer: OS Version: (6.0.6002 SP 2 NLS 500100.50100)
Operation completed successfully in 0.63 seconds.
In this case a state of dirty shutdown is expected. When a database is enabled for replication we utilize a special replay process called Replay Without Undo. Essentially we know that if the source database is mounted, and that there is a copy, we will encounter incomplete transactions in a log stream that will eventually be completed in logs that are either being generated or in the process of copying to the target. With this in mind we do not want to undo these logical transactions, but rather hold them open and wait for subsequent logs that will complete them.
With this in mind administrators should not be immediately concerned that their passive database copies show a state of dirty shutdown.
In Exchange 2010 administrators may attempt to move either log files or database files using the command move-DatabasePath. In some instances the command may fail containing an InvalidOperationException. Here are some examples of the errors (note that verbose output is utilized):
========================================================
Move-DatabasePath –identity <name> –logFolderPath <path> –verbose
[PS] C:\>Move-DatabasePath MBX-1-DB0 -LogFolderPath T:\NewPath -Verbose VERBOSE: [17:29:43.342 GMT] Move-DatabasePath : Active Directory session settings for 'Move-DatabasePath' are: View Entire Forest: 'False', Default Scope: 'home.domain.com', Configuration Domain Controller: 'DC-1.home.domain.com', Preferred Global Catalog: 'DC-2.home.domain.com', Preferred Domain Controllers: '{ DC-2.home.domain.com }' VERBOSE: [17:29:43.342 GMT] Move-DatabasePath : Runspace context: Executing user: home.domain.com/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Enabled. VERBOSE: [17:29:43.342 GMT] Move-DatabasePath : Beginning processing & VERBOSE: [17:29:43.342 GMT] Move-DatabasePath : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent". VERBOSE: [17:29:43.420 GMT] Move-DatabasePath : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} } VERBOSE: [17:29:44.124 GMT] Move-DatabasePath : Searching objects "MBX-1-DB0" of type "Database" under the root "$null". VERBOSE: [17:29:44.155 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [17:29:44.217 GMT] Move-DatabasePath : Processing object "MBX-1-DB0". VERBOSE: [17:29:45.186 GMT] Move-DatabasePath : Searching objects "MBX-1.home.domain.com" of type "Server" under the root "$null". VERBOSE: [17:29:45.280 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [17:29:46.873 GMT] Move-DatabasePath : Verifying that the path "T:\NewPath" on the server "MBX-1.home.domain.com" is located on a fixed or network drive. VERBOSE: [17:29:49.529 GMT] Move-DatabasePath : Verifying that the log location "T:\NewPath" on server "MBX-1.home.domain.com" has not been occupied by an existing file or directory. VERBOSE: [17:29:50.326 GMT] Move-DatabasePath : Checking the existence of log files under directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" on Server "MBX-1.home.domain.com". VERBOSE: [17:33:05.815 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:Validate. VERBOSE: [17:33:05.815 GMT] Move-DatabasePath : Admin Audit Log: Entered ClassFactory:InitializeConfig. VERBOSE: [17:33:05.924 GMT] Move-DatabasePath : Admin Audit Log: Exited ClassFactory:InitializeConfig. VERBOSE: [17:33:07.190 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:Validate.
Confirm Are you sure you want to perform this action?
Moving database path "MBX-1-DB0". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [17:33:52.375 GMT] Move-DatabasePath : Resolved current organization: . VERBOSE: [17:33:52.594 GMT] Move-DatabasePath : The Admin RPC connection is being established with server "MBX-1.home.domain.com".
Confirm To perform the move operation, database "MBX-1-DB0" must be temporarily dismounted, which will make it inaccessible to all users. Do you want to continue? [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [17:34:04.422 GMT] Move-DatabasePath : Reading new object "9d654028-b845-43fc-9b3b-a011c442c9e6" of type "Server". VERBOSE: [17:34:04.422 GMT] Move-DatabasePath : Dismounting the database, "MBX-1-DB0". VERBOSE: [17:34:05.297 GMT] Move-DatabasePath : Copying the database Log files under directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to the target directory "T:\NewPath" on Server "MBX-1". VERBOSE: [17:34:05.344 GMT] Move-DatabasePath : Checking the existence of directory "T:\NewPath" on Server "MBX-1.home.domain.com". VERBOSE: [17:34:05.359 GMT] Move-DatabasePath : Set access control for directory "T:\NewPath" on server "MBX-1.home.domain.com". VERBOSE: [17:34:05.359 GMT] Move-DatabasePath : Checking the existence of directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" on Server "MBX-1.home.domain.com". VERBOSE: [17:34:05.406 GMT] Move-DatabasePath : Copying the files from directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to target directory "T:\NewPath" on server "MBX-1.home.domain.com". VERBOSE: [17:35:54.090 GMT] Move-DatabasePath : Reading new object "9d654028-b845-43fc-9b3b-a011c442c9e6" of type "Server". VERBOSE: [17:35:54.105 GMT] Move-DatabasePath : The task is attempting to communicate with the Microsoft Exchange Replication service on server "MBX-1.home.domain.com" to obtain updated configuration changes for database "MBX-1-DB0". VERBOSE: [17:35:54.965 GMT] Move-DatabasePath : Mounting database "MBX-1-DB0". VERBOSE: [17:35:56.199 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:OnComplete. VERBOSE: [17:36:03.902 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:OnComplete. Failed to move database log files from "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to " T:\NewPath". + CategoryInfo : InvalidOperation: (MBX-1-DB0:ADObjectId) [Move-DatabasePath], InvalidOperationException + FullyQualifiedErrorId : 7280D023,Microsoft.Exchange.Management.SystemConfigurationTasks.MoveDatabasePath VERBOSE: [17:36:04.074 GMT] Move-DatabasePath : Ending processing & [PS] C:\>
Move-DatabasePath –identity <name> –logFolderPath <path> –edbFilePath <path\name.edb> –verbose
[PS] C:\>Move-DatabasePath -Identity MBX-1-DB0 -EdbFilePath t:\NewPath\MBX-1-DB0.edb
Moving database path "MBX-1-DB0". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a
Confirm To perform the move operation, database "MBX-1-DB0" must be temporarily dismounted, which will make it inaccessible to all users. Do you want to continue? [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a [PS] C:\>Move-DatabasePath -Identity MBX-1-DB0 -EdbFilePath "E:\MBX-1\MBX-1-DB0\MBX-1-DB0-Database\MBX-1-DB0.edb"
Confirm To perform the move operation, database "MBX-1-DB0" must be temporarily dismounted, which will make it inaccessible to all users. Do you want to continue? [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a [PS] C:\> Move-DatabasePath MBX-1-DB0 -LogFolderPath T:\NewPath -EdbFilePath t:\NewPath\MBX-1-DB0.edb -Verbose VERBOSE: [17:43:42.454 GMT] Move-DatabasePath : Active Directory session settings for 'Move-DatabasePath' are: View Entire Forest: 'False', Default Scope: 'home.domain.com', Configuration Domain Controller: 'DC-1.home.domain.com', Preferred Global Catalog: 'DC-2.home.domain.com', Preferred Domain Controllers: '{ DC-2.home.domain.com }' VERBOSE: [17:43:42.454 GMT] Move-DatabasePath : Runspace context: Executing user: home.domain.com/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Enabled. VERBOSE: [17:43:42.454 GMT] Move-DatabasePath : Beginning processing & VERBOSE: [17:43:42.454 GMT] Move-DatabasePath : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent". VERBOSE: [17:43:42.470 GMT] Move-DatabasePath : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} } VERBOSE: [17:43:42.470 GMT] Move-DatabasePath : Searching objects "MBX-1-DB0" of type "Database" under the root "$null". VERBOSE: [17:43:42.486 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [17:43:42.486 GMT] Move-DatabasePath : Processing object "MBX-1-DB0". VERBOSE: [17:43:42.501 GMT] Move-DatabasePath : Searching objects "MBX-1.home.domain.com" of type "Server" under the root "$null". VERBOSE: [17:43:42.533 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [17:43:42.595 GMT] Move-DatabasePath : Verifying that EDB file path "t:\NewPath\MBX-1-DB0.edb" is available for use. VERBOSE: [17:43:42.704 GMT] Move-DatabasePath : Checking the existence of file "E:\MBX-1\MBX-1-DB0\MBX-1-DB0-Database\MBX-1-DB0.edb" on server "MBX-1". VERBOSE: [17:43:42.783 GMT] Move-DatabasePath : Verifying that the path "t:\NewPath\MBX-1-DB0.edb" on the server "MBX-1" is located on a fixed or network drive. VERBOSE: [17:43:42.783 GMT] Move-DatabasePath : Checking the existence of file "t:\NewPath\MBX-1-DB0.edb" on server "MBX-1". VERBOSE: [17:43:42.814 GMT] Move-DatabasePath : Checking the existence of directory "t:\NewPath\MBX-1-DB0.edb" on Server "MBX-1". VERBOSE: [17:43:42.845 GMT] Move-DatabasePath : Checking the existence of directory "t:\NewPath" on Server "MBX-1.home.domain.com". VERBOSE: [17:43:42.845 GMT] Move-DatabasePath : Set access control for directory "t:\NewPath" on server "MBX-1.home.domain.com". VERBOSE: [17:43:42.845 GMT] Move-DatabasePath : Checking the existence of directory "e:\mbx-1\mbx-1-db0\mbx-1-db0-database\catalogdata-ed71935d-a14f-4937-b20d-50b3 7e136797-9d654028-b845-43fc-9b3b-a011c442c9e6" on Server "MBX-1". VERBOSE: [17:43:42.845 GMT] Move-DatabasePath : Verifying that the path "T:\NewPath" on the server "MBX-1.home.domain.com" is located on a fixed or network drive. VERBOSE: [17:43:42.861 GMT] Move-DatabasePath : Verifying that the log location "T:\NewPath" on server "MBX-1.home.domain.com" has not been occupied by an existing file or directory. VERBOSE: [17:43:42.923 GMT] Move-DatabasePath : Checking the existence of log files under directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" on Server "MBX-1.home.domain.com". VERBOSE: [17:46:51.018 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:Validate. VERBOSE: [17:46:51.034 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:Validate.
Moving database path "MBX-1-DB0". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [17:51:35.320 GMT] Move-DatabasePath : Resolved current organization: . VERBOSE: [17:51:35.351 GMT] Move-DatabasePath : The Admin RPC connection is being established with server "MBX-1.home.domain.com".
Confirm To perform the move operation, database "MBX-1-DB0" must be temporarily dismounted, which will make it inaccessible to all users. Do you want to continue? [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [17:51:37.586 GMT] Move-DatabasePath : Reading new object "9d654028-b845-43fc-9b3b-a011c442c9e6" of type "Server". VERBOSE: [17:51:37.586 GMT] Move-DatabasePath : Dismounting the database, "MBX-1-DB0". VERBOSE: [17:51:37.851 GMT] Move-DatabasePath : The file is being copied from "E:\MBX-1\MBX-1-DB0\MBX-1-DB0-Database\MBX-1-DB0.edb" to "t:\NewPath\MBX-1-DB0.edb" on server "MBX-1.home.domain.com". VERBOSE: [17:53:03.177 GMT] Move-DatabasePath : Copying the database Log files under directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to the target directory "T:\NewPath" on Server "MBX-1". VERBOSE: [17:53:03.209 GMT] Move-DatabasePath : Checking the existence of directory "T:\NewPath" on Server "MBX-1.home.domain.com". VERBOSE: [17:53:03.209 GMT] Move-DatabasePath : Set access control for directory "T:\NewPath" on server "MBX-1.home.domain.com". VERBOSE: [17:53:03.209 GMT] Move-DatabasePath : Checking the existence of directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" on Server "MBX-1.home.domain.com". VERBOSE: [17:53:03.271 GMT] Move-DatabasePath : Copying the files from directory "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to target directory "T:\NewPath" on server "MBX-1.home.domain.com". VERBOSE: [17:56:19.237 GMT] Move-DatabasePath : Deleting the file "t:\NewPath\MBX-1-DB0.edb" on server "MBX-1". VERBOSE: [17:56:19.378 GMT] Move-DatabasePath : Reading new object "9d654028-b845-43fc-9b3b-a011c442c9e6" of type "Server". VERBOSE: [17:56:19.378 GMT] Move-DatabasePath : The task is attempting to communicate with the Microsoft Exchange Replication service on server "MBX-1.home.domain.com" to obtain updated configuration changes for database "MBX-1-DB0". VERBOSE: [17:56:19.503 GMT] Move-DatabasePath : Mounting database "MBX-1-DB0". VERBOSE: [17:56:21.487 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:OnComplete. VERBOSE: [17:56:21.831 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:OnComplete. Failed to move database log files from "d:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" to " T:\NewPath". + CategoryInfo : InvalidOperation: (MBX-1-DB0:ADObjectId) [Move-DatabasePath], InvalidOperationException + FullyQualifiedErrorId : 7280D023,Microsoft.Exchange.Management.SystemConfigurationTasks.MoveDatabasePath VERBOSE: [17:56:22.003 GMT] Move-DatabasePath : Ending processing & [PS] C:\>
The issue that causes this error is due to the way free space is determined when a mount point is utilized. In this case we have the following disk layout:
Disk T:\ –> 8 megs free space.
Disk T:\NewPath –> NewPath is a mount point created from a folder residing on the T volume. NewPath has 127 gig of free space.
In this case the first check that is performed is a WMI query to determine if the destination has enough free space to accommodate the log files being moved. Unfortunately the WMI call utilized checks the free space of the mount point root disk, and not the mount point itself. This results in the destination being preceived as not having enough disk space and the move operation returning failed with an exception.
Note: This issue only happens when moving log files or moving log files with a database. If only the database path is adjusted this issue does not occur.
To correct this issue the log files can be moved by hand. The following steps will work around this condition:
1) Dismount the database that you want to move log files for.
Dismount-Database –identity <NAME> -verbose
[PS] C:\>Dismount-Database -Identity MBX-1-DB0 -Verbose VERBOSE: [18:12:18.536 GMT] Dismount-Database : Active Directory session settings for 'Dismount-Database' are: View Entire Forest: 'False', Default Scope: 'home.domain.com', Configuration Domain Controller: 'DC-1.home.domain.com', Preferred Global Catalog: 'DC-2.home.domain.com', Preferred Domain Controllers: '{ DC-2.home.domain.com }' VERBOSE: [18:12:18.536 GMT] Dismount-Database : Runspace context: Executing user: home.domain.com/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Enabled. VERBOSE: [18:12:18.536 GMT] Dismount-Database : Beginning processing & VERBOSE: [18:12:18.536 GMT] Dismount-Database : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent". VERBOSE: [18:12:18.583 GMT] Dismount-Database : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} } VERBOSE: [18:12:18.583 GMT] Dismount-Database : Searching objects "MBX-1-DB0" of type "Database" under the root "$null". VERBOSE: [18:12:18.598 GMT] Dismount-Database : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [18:12:18.598 GMT] Dismount-Database : Processing object "MBX-1-DB0". VERBOSE: [18:12:18.645 GMT] Dismount-Database : Admin Audit Log: Entered Handler:Validate. VERBOSE: [18:12:18.645 GMT] Dismount-Database : Admin Audit Log: Exited Handler:Validate.
Dismounting database "MBX-1-DB0". This may result in reduced availability for mailboxes in the database. [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [18:12:20.458 GMT] Dismount-Database : Resolved current organization: . VERBOSE: [18:12:20.489 GMT] Dismount-Database : The Admin RPC connection is being established with server "MBX-1.home.domain.com". VERBOSE: [18:12:20.489 GMT] Dismount-Database : Dismounting the database, "MBX-1-DB0". VERBOSE: [18:12:20.817 GMT] Dismount-Database : Admin Audit Log: Entered Handler:OnComplete. VERBOSE: [18:12:20.880 GMT] Dismount-Database : Admin Audit Log: Exited Handler:OnComplete. VERBOSE: [18:12:20.895 GMT] Dismount-Database : Ending processing & [PS] C:\>
2) Move the log file path using the configuration only switch which will simply write the new path to the active directory.
Move-DatabasePath –identity <NAME> –logFolderPath <path> –configurationOnly:$TRUE –verbose
[PS] C:\>Move-DatabasePath -Identity MBX-1-DB0 -LogFolderPath T:\NewPath -ConfigurationOnly:$TRUE -Verbose VERBOSE: [18:18:12.047 GMT] Move-DatabasePath : Active Directory session settings for 'Move-DatabasePath' are: View Entire Forest: 'False', Default Scope: 'home.domain.com', Configuration Domain Controller: 'DC-1.home.domain.com', Preferred Global Catalog: 'DC-2.home.domain.com', Preferred Domain Controllers: '{ DC-2.home.domain.com }' VERBOSE: [18:18:12.063 GMT] Move-DatabasePath : Runspace context: Executing user: home.domain.com/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Enabled. VERBOSE: [18:18:12.063 GMT] Move-DatabasePath : Beginning processing & VERBOSE: [18:18:12.063 GMT] Move-DatabasePath : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent". VERBOSE: [18:18:12.063 GMT] Move-DatabasePath : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} } VERBOSE: [18:18:12.079 GMT] Move-DatabasePath : Searching objects "MBX-1-DB0" of type "Database" under the root "$null". VERBOSE: [18:18:12.360 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [18:18:12.360 GMT] Move-DatabasePath : Processing object "MBX-1-DB0". VERBOSE: [18:18:12.391 GMT] Move-DatabasePath : Searching objects "MBX-1.home.domain.com" of type "Server" under the root "$null". VERBOSE: [18:18:12.391 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'.
Confirm This operation will skip the safety check and make the change to Active Directory directly. Do you want to continue? [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [18:18:14.798 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:Validate. VERBOSE: [18:18:14.798 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:Validate.
Moving database path "MBX-1-DB0". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a VERBOSE: [18:18:16.110 GMT] Move-DatabasePath : Resolved current organization: . VERBOSE: [18:18:16.126 GMT] Move-DatabasePath : The Admin RPC connection is being established with server "MBX-1.home.domain.com". VERBOSE: [18:18:16.126 GMT] Move-DatabasePath : Saving object "MBX-1-DB0" of type "Database" and state "Changed". VERBOSE: [18:18:16.126 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [18:18:16.126 GMT] Move-DatabasePath : Getting the name of the domain controller to be used to double write the configurable object's changes. Double write is the mechanism to make sure that the changes are persisted in the domain controller used by the store service on server "MBX-1.home.domain.com". VERBOSE: [18:18:16.141 GMT] Move-DatabasePath : Reading new object "MBX-1-DB0" of type "Database". VERBOSE: [18:18:16.141 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [18:18:16.141 GMT] Move-DatabasePath : Double writing changes of the configurable object, "MBX-1-DB0", on domain controller "DC-1.home.domain.com". VERBOSE: [18:18:16.141 GMT] Move-DatabasePath : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [18:18:16.141 GMT] Move-DatabasePath : The task is attempting to communicate with the Microsoft Exchange Replication service on server "MBX-1.home.domain.com" to obtain updated configuration changes for database "MBX-1-DB0". VERBOSE: [18:18:16.298 GMT] Move-DatabasePath : Admin Audit Log: Entered Handler:OnComplete. VERBOSE: [18:18:16.313 GMT] Move-DatabasePath : Admin Audit Log: Exited Handler:OnComplete. VERBOSE: [18:18:16.313 GMT] Move-DatabasePath : Ending processing & [PS] C:\>
3) Using ROBOCopy – which is included with Windows 2008 and Windows 2008 R2 – mirror the current log directory to the new location.
Note: This command will MOVE the files – that is copy them to the new directory and then delete the original file.
Note: This command assumes that only log files exist in the directory. If the database and catalog files exist in the directory they should be skipped.
ROBOCOPY <SOURCE-DIR> <TARGET-DIR> /MOVE /XD *Catalog* /XF *.edb
[PS] C:\>robocopy "D:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs" "T:\NewPath" /MOVE /XD *Catalog* /XF *.edb
------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Mon Sep 27 14:50:08 2010
Source : D:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs\ Dest : T:\NewPath\
Files : *.*
Exc Files : *.edb
Exc Dirs : *Catalog*
Options : *.* /COPY:DAT /MOVE /R:1000000 /W:30
------------------------------------------------------------------------------
100% New File 1.0 m E00res00005.jrs 100% New File 1.0 m E00res00006.jrs 100% New File 1.0 m E00res00007.jrs 100% New File 1.0 m E00res00008.jrs 100% New File 1.0 m E00res00009.jrs 100% New File 1.0 m E00res0000A.jrs 100% New File 0 E00tmp.log
Total Copied Skipped Mismatch FAILED Extras Dirs : 2 0 2 0 0 1 Files : 6430 6429 1 0 0 0 Bytes : 9.158 g 6.275 g 2.882 g 0 0 0 Times : 0:04:03 0:03:40 0:00:00 0:00:22
Speed : 30539839 Bytes/sec. Speed : 1747.503 MegaBytes/min.
Ended : Mon Sep 27 14:44:39 2010 [PS] C:\>
3) Optional: Move the database file to the new location.
Move-DatabasePath –identity <NAME> –edbFilePath <TargetPath\Name.edb> –verbose
4) Mount the database post all move operations (allow sufficient time for AD replication to occur)
Mount-Database –identity <NAME> –verbose
[PS] C:\>Mount-Database -Identity MBX-1-DB0 -Verbose VERBOSE: [19:06:06.471 GMT] Mount-Database : Active Directory session settings for 'Mount-Database' are: View Entire Forest: 'False', Default Scope: 'home.domain.com', Configuration Domain Controller: 'DC-1.home.domain.com', Preferred Global Catalog: 'DC-2.home.domain.com', Preferred Domain Controllers: '{ DC-2.home.domain.com }' VERBOSE: [19:06:06.486 GMT] Mount-Database : Runspace context: Executing user: home.domain.com/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Enabled. VERBOSE: [19:06:06.486 GMT] Mount-Database : Beginning processing & VERBOSE: [19:06:06.486 GMT] Mount-Database : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent". VERBOSE: [19:06:07.033 GMT] Mount-Database : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} } VERBOSE: [19:06:07.033 GMT] Mount-Database : Searching objects "MBX-1-DB0" of type "Database" under the root "$null". VERBOSE: [19:06:07.049 GMT] Mount-Database : Previous operation run on domain controller 'DC-1.home.domain.com'. VERBOSE: [19:06:07.065 GMT] Mount-Database : Processing object "MBX-1-DB0". VERBOSE: [19:06:07.111 GMT] Mount-Database : Admin Audit Log: Entered Handler:Validate. VERBOSE: [19:06:07.111 GMT] Mount-Database : Admin Audit Log: Exited Handler:Validate. VERBOSE: Mounting database "MBX-1-DB0". VERBOSE: [19:06:07.127 GMT] Mount-Database : Resolved current organization: . VERBOSE: [19:06:07.158 GMT] Mount-Database : Searching objects "home.domain.com/Microsoft Exchange System Objects/SystemMailbox{ed71935d-a14f-4937-b20d-50b37e136797}" of type "ADRecipient" under the root "$null". VERBOSE: [19:06:07.752 GMT] Mount-Database : Mounting database "MBX-1-DB0". VERBOSE: [19:06:08.971 GMT] Mount-Database : Admin Audit Log: Entered Handler:OnComplete. VERBOSE: [19:06:09.065 GMT] Mount-Database : Admin Audit Log: Exited Handler:OnComplete. VERBOSE: [19:06:09.065 GMT] Mount-Database : Ending processing & [PS] C:\>
At this time we expect this issue to be resolved in Exchange 2010 Service Pack 2.
Recently some customers have experienced an issue where database copies do not display when using the Exchange Management Console after upgrading to Exchange 2010 Service Pack 1. When attempting to view database copies using the Exchange Management Shell no issue is displayed.
Database copies can be viewed in two locations.
The first location is under Organization Configuration –> Mailbox –> Database Management. In this view when an administrator selects a database from the list, the Database Copies displayed in the bottom portion of the display are missing ALL or SOME copies for a given database. Here is an example of a 4 node DAG with a database replicated to all 4 members:
All Database Copies Missing:
Some Database Copies Missing:
Expected output showing all database copies:
The second location is under Server Configuration –> Mailbox. By selecting a server with the mailbox role you can view the individual database copies assigned to that server. Here is an example of database copies missing from the server:
Whether all database copies are missing <or> some database copies are missing the Exchange Management Shell command Get-MailboxDatabaseCopyStatus always returns accurate information. Here is a copy of Get-MailboxDatabaseCopyStatus *:
When any Exchange role is installed on the server (with exception of Edge transport) the hostname of the server is written into the Exchange configuration container within Active Directory. In this case the server name was established with a lower case. It was established in the container with a lower case because the hostname of the server, which was established during setup, was established with a lower case name (or a portion of the name lowercase).
In reference customers running HOSTNAME <or> reviewing the full computer name in server management showed that all or a portion of the name was lower case.
The Exchange Management Console code incorrectly compares case when comparing a database copy against a server name. This causes the console to not display all the valid database copies.
Let’s step through an example.
In this example there is a four node DAG. The server names are as follows:
Dag-1
dag-2
DAG-3
DAG-4
You can verify these server names by running get-databaseavailabilitygroup –identity <DAGName> | fl name,servers
[PS] D:\>Get-DatabaseAvailabilityGroup -Identity DAG | fl name,servers
Name : DAG Servers : {DAG-4, DAG-3, dag-2, Dag-1}
When looking in the management console, and selecting any database replicated to all four members, the only copies that will be displayed are on nodes DAG-4 and DAG-3.
When the Exchange Management Console draws the database copies pane, it compares the host server name of a database copy to the server name of a database copy status. This comparison is case sensitive. Let’s take a look at a database copy that fails to display when viewing the copies for DAG-1.
(Get-MailboxDatabase –identity <NAME>).databasecopies | fl hostservername
[PS] D:\>(Get-MailboxDatabase DAG-DB0).databasecopies | fl hostservername
HostServerName : Dag-1
HostServerName : DAG-3
HostServerName : DAG-4
HostServerName : dag-2
(Get-MailboxDatabaseCopyStatus <NAME>\<Server>).mailboxserver
[PS] D:\>(Get-MailboxDatabaseCopyStatus DAG-DB0\DAG-1).mailboxServer DAG-1
In this case DAG-1 != Dag-1 and therefore the copy does not display in the management console.
Let’s take a look at a copy that does display.
[PS] D:\>(Get-MailboxDatabaseCopyStatus DAG-DB0\DAG-4).mailboxServer DAG-4
In this case DAG-4 == DAG-4 and therefore the copy does display in the management console.
Unfortunately at this time we can only recommend that database copies for missing servers be managed using the Exchange Management Shell. It is not recommended to attempt to modify the active directory at this time to overcome this issue.
At this time we are filing the necessary bugs to get this permanently corrected without modifying server names. I will update this blog as this process progresses.
=======================================
Update 10/10/2010
This issue is currently scheduled to be corrected in Exchange 2010 SP1 Rollup Update 3.
Have a DAG? Thinking about having a DAG? I recommend you check out this awesome blog on designing a highly available database copy layout…
http://msexchangeteam.com/archive/2010/09/10/456211.aspx
This is a great blog post on Windows 2008 and Windows 2008 R2 Cluster Logs.
http://blogs.technet.com/b/askcore/archive/2010/04/13/understanding-the-cluster-debug-log-in-2008.aspx
TIMMCMIC
If an Exchange 2010 RTM server <or> an Exchange 2010 SP1 Beta has been upgraded to Exchange 2010 SP1 RTM administrators may experience an error when attempting to utilize the remove-mailboxdatabasecopy <or> add-mailboxdatabasecopy commandlets.
When running remove-mailboxdatabasecopy the following error is noted:
Remove-MailboxDatabaseCopy DAG-DB0\DAG-2 –Verbose
Registry key has subkeys and recursive removes are not supported by this method. + CategoryInfo : NotSpecified: (:) [Remove-MailboxDatabaseCopy], InvalidOperationException + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.SystemConfigurationTasks. RemoveMailboxDatabaseCopy
Although the error is reported, the remove was successful in updating the database object within the active directory to show the server no longer hosts a copy of the database. You can verify the copy was successfully removed by reviewing the Servers with the get-mailboxdatabase –identity <NAME> | fl name, servers commandlet.
Here is sample output (note DAG-2 is missing):
[PS] D:\>Get-MailboxDatabase DAG-DB0 | fl name,servers
Name : DAG-DB0 Servers : {DAG-1, DAG-3, DAG-4}
If an administrator attempts to add a database copy to a DAG member, the same error may also be returned.
Add-MailboxDatabaseCopy DAG-DB0 -MailboxServer DAG-2
WARNING: An unexpected error has occurred and a Watson dump is being generated: Registry key has subkeys and recursive removes are not supported by this method.
Registry key has subkeys and recursive removes are not supported by this method. + CategoryInfo : NotSpecified: (:) [Add-MailboxDatabaseCopy], InvalidOperationException + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.SystemConfigurationTasks. AddMailboxDatabaseCopy
Unlike the remove-mailboxdatabasecopy this command is not successful in adding the copy <or> updating the Active Directory to show the copy was added.
To work around this issue the administrator should:
1) Identify the GUID of the database that is being added.
2) On the server specified in the add command, using the database GUID identified, remove the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\Replay\State\{DB-GUID}\DumpsterInfo
To identify the mailbox database GUID, use the following command:
[PS] D:\>Get-MailboxDatabase DAG-DB0 | fl name,GUID
Name : DAG-DB0 Guid : 8d3a9778-851c-40a4-91af-65a2c487b4cc
The GUID identified in this case is 8d3a9778-851c-40a4-91af-65a2c487b4cc. With this information we can no export and delete the DUMPSTERINFO key on the server where you are attempting to add the mailbox database copy.
Once the registry key is removed the add-mailboxdatabasecopy command will complete successfully and the database copy will be added.
A question that has come up a few times recently is why does the date modified timestamp on my Exchange databases not change (even though the database is mounted and functioning). Specifically some administrators have been looking at this as an indicator of health on a passive database copy – which it is not.
The date modified timestamp will generally get updated on an Exchange database when one of two things happen:
1) The EDB file size is extended in order to accommodate data that does not fit into whitespace that currently exists in the database.
2) The database is dismounted and all open handles to the file are released.
Note that the modified time is not subject to change if the contents of the file are changed – for example if whitespace is utilized within the database for the storage of new messages etc the date modified will not change.
To show this I used my lab to generate some examples. Here is a screen shot of a database that was mounted last on 8/3/2010. The database screen shot was taken 8/8/2010 before 8:29 am edt.
Using the Exchange Management Console, I dismounted the database at 8:29 am edt on 8/8/2010.
You will note that the date modified changed to the time and date the dismount occurred. I then used the Exchange Management Console to re-mount the database.
After remounting the database I noted that the time remained the same as in the previous screen shot. I then took some test mailboxes with content, and moved them into the mailbox store. You will note in this screen shot that both the size and date modified changed – in this case the database file was extended on the partition so the change was expected.
It is normal for an Exchange database to not show an updated date modified and this field should be used to judge the health or utilization of an Exchange database.
In Exchange 2007 there are two clustered installation models. Some customers elect to utilize a clustered installation model based on shared storage – this is a single copy cluster installation. In order to achieve site resiliency or provide for disaster recovery, some customers will implement a SAN based data replication solution.
Recently I encountered a customer that was utilizing SAN based data replication and the single copy cluster installation model to provide their site resilient solution. The installation encompassed a source cluster with single copy configuration and a target cluster with single copy cluster configuration. Each clustered mailbox server was established utilizing a different name – for example Exchange-Main and Exchange-DR. The physical disk resources that were assigned to each CMS instance represented the LUNs that were replicated between SANs. When it was necessary to activate the solution databases would be marked as “Allow this database to be overwritten by a restore” and then mounted. Mailboxes would be moved utilizing the move-mailbox –configurationOnly to restore client access to the replicated databases
This presented an interesting challenge for this customer when it came to deploying service packs. When the same physical disk resources are utilized between clusters, only one set of the physical disk resources can be brought online. This is because one SAN has a Read / Write setting and the other SAN has a Read Only setting. Essentially an online attempt of the database instances of the CMS Exchange-DR would fail because their dependant physical disks could not be brought online (because they were read only).
When an /upgradeCMS is performed after upgrading the binaries on a clustered node, the resources are initially in an offline state. As a completion of the upgradeCMS the setup process initiates an online to the cluster mailbox server group. Should any resources fail to come online this is considered a failure of the upgrade. The administrator performing the upgrade is notified that a failure occurred and the upgrade setup watermark persists in the registry. Therefore it is necessary that the /upgradeCMS be allowed to complete. In this case database instances could not be brought online because their associated storage could not be brought online due to the storage being Read Only.
In order to complete the upgrade process the following steps were utilized (utilizing my sample clustered mailbox server names).
At this point both Exchange-Main and Exchange-DR are online. This means that the databases that were previously replicated to Exchange-DR are no longer equal to the databases that exist on Exchange-Main. As a post upgrade step we need to do the following:
In this installation it was necessary to temporarily break and re-establish replication in order to complete the /upgradeCMS process.
Exchange 2007 SP3 adds the support for utilizing Windows 2008 R2 servers.
In Exchange 2007 Cluster Continuous Replication (CCR) installations, all log shipping activity by default occurs over the “public” cluster interface. When administrators desire to have log shipping activities occur over a “private” network or desire to implement multiple replication paths between nodes, continuous replication hostnames can be utilized.
More information on Exchange 2007 CCR clusters and continuous replication hostnames can be found at http://technet.microsoft.com/en-us/library/bb124521(EXCHG.80).aspx.
Prior to implementing a continuous replication host name the get-clusteredservermailboxstatus commandlet can be utilized to see the current names services replication. Here is a sample output from a cluster not configured to utilize continuous replication hostnames.
Identity : MBX-3 ClusteredMailboxServerName : MBX-3.domain.com State : Online OperationalMachines : {NODE-1 <Active>, Node-2 <Quorum Owner>} FailedResources : {} OperationalReplicationHostNames : {node-1, node-2} FailedReplicationHostNames : {} InUseReplicationHostNames : {node-1, node-2} IsValid : True ObjectState : Unchanged
After establishing the pre-requisites necessary to utilize continuous replication hostnames, the hostnames creation is performed using the enable-continuousreplicationhostname shell command. (http://technet.microsoft.com/en-us/library/bb690985(EXCHG.80).aspx)
When attempting to enable a replication hostname on a Windows 2008 R2 cluster, the following error may be displayed in the management shell.
[PS] C:\>Enable-ContinuousReplicationHostName -TargetMachine Node-1 -HostName Node-1-Repl-A -IPv4Address 10.0.1.3
Enabling continuous replication host name "Node-1-Repl-A". [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"):a Enable-ContinuousReplicationHostName : Enable-ContinuousReplicationHostNameNetw ork configuration could not be completed. At line:1 char:37 + Enable-ContinuousReplicationHostName <<<< -TargetMachine Node-1 -HostName Node-1-Repl-A -IPv4Address 10.0.1.3 + CategoryInfo : InvalidOperation: (:) [Enable-ContinuousReplicat ionHostName], NetworkConfigException + FullyQualifiedErrorId : C3F1320,Microsoft.Exchange.Management.SystemConf igurationTasks.EnableContinuousReplicationHostName
When reviewing Failover Cluster Manager, the replication host name group containing the correct network name and ipv4 address appear to have been created successfully.
Although the continuous replication hostname group was created, reviewing get-clusteredservermailboxstatus indicates the name is not being utilized by the replication service on the cluster.
When the replication service first starts up <or> the configuration time expires the replication service enumerates all network names on the cluster to determine which are valid endpoints for log shipping. This is initially based on two cluster private properties stamped on each name, MSExchange_NetName and MSExchange_UseNetworkForLogCopying. Each of these should have a value of 1 on a network name utilized as a continuous replication host name.
Listing private properties for 'Network Name (Node-1-Repl-A)':
T Resource Name Value
-- -------------------- ------------------------------ -----------------------
BR Network Name (Node-1-Repl-A) ResourceData 01 00 00 00 ... (260 bytes)
DR Network Name (Node-1-Repl-A) StatusNetBIOS 0 (0x0)
DR Network Name (Node-1-Repl-A) StatusDNS 0 (0x0)
DR Network Name (Node-1-Repl-A) StatusKerberos 0 (0x0)
SR Network Name (Node-1-Repl-A) CreatingDC \\DC-1.domain.com
FTR Network Name (Node-1-Repl-A) LastDNSUpdateTime 7/11/2010 2:26:26 PM
SR Network Name (Node-1-Repl-A) ObjectGUID 5adc38b3281a004788f2a3e27ae7a0ce
S Network Name (Node-1-Repl-A) Name NODE-1-REPL-A
S Network Name (Node-1-Repl-A) DnsName Node-1-Repl-A
D Network Name (Node-1-Repl-A) RemapPipeNames 0 (0x0)
D Network Name (Node-1-Repl-A) HostRecordTTL 1200 (0x4b0)
D Network Name (Node-1-Repl-A) RegisterAllProvidersIP 0 (0x0)
D Network Name (Node-1-Repl-A) PublishPTRRecords 0 (0x0)
D Network Name (Node-1-Repl-A) TimerCallbackAdditionalThreshold 5 (0x5)
D Network Name (Node-1-Repl-A) MSExchange_NetName 1 (0x1)
D Network Name (Node-1-Repl-A) RequireDNS 1 (0x1)
D Network Name (Node-1-Repl-A) MSExchange_UseNetworkForLogCopying 1 (0x1)
On the surface it would appear that there is nothing preventing this name from operating correctly as a continuous replication host name. After performing some internal tracing it was determined that the replication service is also implementing another check on a network name resource to ensure that it can be satisfactorily utilized for replication – is Kerberos enabled for the network name. The replication service performs this check by reviewing a private property of a network name resource – requirekerberos and ensuring it has a value of 1.
In Windows 2003 network name resources could be enabled for Kerberos at the administrators discretion. In Windows 2008 and Windows 2008 R2 all network names must be Kerberos enabled. In Windows 2008 requireKerberos is a valid private property and can be programatically set. In Windows 2008 R2 the requireKerberos property has been deprecated and can be no longer be programmatically set. Without the requireKerberos property in Windows 2008 R2 the enable-continuousreplicationhostname commandlet fails with the previously documented error.
To work around this issue and allow the replication host names created with the enable-continuousreplicationhostname command to function the following steps can be performed:
At this time you can utilize either cluster.exe or powershell to verify that the requirekerboros key has been created with a value of 1.
Cluster.exe <clusterFQDN> res <Network Name> /priv --> Cluster.exe cluster.domain.com res “Network Name (Node-1-Repl-A)” /priv
D Network Name (Node-1-Repl-A) requirekerberos 1 (0x1)
Get-ClusterResource <NAME> | Get-ClusterParameter
Object Name Value Type ------ ---- ----- ---- Network Name (No... Name NODE-1-REPL-A String Network Name (No... DnsName Node-1-Repl-A String Network Name (No... RemapPipeNames 0 UInt32 Network Name (No... HostRecordTTL 1200 UInt32 Network Name (No... RegisterAllProvi... 0 UInt32 Network Name (No... PublishPTRRecords 0 UInt32 Network Name (No... TimerCallbackAdd... 5 UInt32 Network Name (No... MSExchange_NetName 1 UInt32 Network Name (No... RequireDNS 1 UInt32 Network Name (No... MSExchange_UseNe... 1 UInt32 Network Name (No... requirekerberos 1 UInt32 Network Name (No... ResourceData {1, 0, 0, 0, 118... ByteArray Network Name (No... StatusNetBIOS 0 UInt32 Network Name (No... StatusDNS 0 UInt32 Network Name (No... StatusKerberos 0 UInt32 Network Name (No... CreatingDC \\DC-1.domain...... String Network Name (No... LastDNSUpdateTime 7/11/2010 9:26:2... DateTime Network Name (No... ObjectGUID 5adc38b3281a0047... String
By restarting the replication service after setting this key the replication services configuration is immediately updated. At this time the replication service should detect and begin to utilize the replication hostnames created. This can be verified using the get-clusteredservermailboxstatus commandlet.
Identity : MBX-3 ClusteredMailboxServerName : MBX-3.exchange.msft State : Online OperationalMachines : {NODE-1 <Active>, Node-2 <Quorum Owner>} FailedResources : {} OperationalReplicationHostNames : {node-1-repl-a, node-1, node-2} FailedReplicationHostNames : {} InUseReplicationHostNames : {node-1-repl-a, node-2} IsValid : True ObjectState : Unchanged
At this time we are investigating a fix that does not require a workaround. As changes occur I will update this blog.
In Exchange 2010 when a Database Availability Group (DAG) it utilized, and there is an even number of DAG members, the underlying cluster is implemented utilizing the quorum type Node and File Share Majority. The settings utilized for the File Share Witness are defined on the DAG when the logical DAG object is created and are either set by the administrator or automatically defined.
To verify the quorum type you can use either cluster.exe or cluster powershell extensions (Preferred)
Cluster.exe <cluster> /quorum (Windows 2008 & Windows 2008 R2)
Cluster.exe cluster.domain.com /quorum
Witness Resource Name Path Type
--------------------- --------------------------------------------- --------
File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) Majority
Get-Cluster <cluster> | Get-ClusterQuorum | FL (Windows 2008 R2 Only)
Cluster : DAG QuorumResource : File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) QuorumType : NodeAndFileShareMajority
In Failover Cluster Manager, the resources can be viewed by looking at the Cluster Core Resources.
It may become necessary to change the server hosting the file share witness. In Exchange 2010 this is not done utilizing Failover Cluster Manager, but rather utilizing the set-databaseavailabilitygroup commandlet. It is after the witness server is successfully updated that the oddity occurs. Here’s an example:
Currently the DAG utilizes the witness server HT-1. Using the set-databaseavailabilitygroup command the witness server is changed to HT-2. (set-databaseavailabilitygroupserver –witnessServer HT-2) The command returns without error. When running the previous cluster commands the following output is noted:
Cluster.exe cluster.domain.com /quorum (Windows 2008 and Windows 2008 R2)
Also in Failover Cluster Manager the following is noted in the cluster core resources group.
After looking at this output the administrator could be lead to believe that the witness server did not successfully update. After all both cluster.exe and powershell both show the File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM). It is only in Failover Cluster Manager, if the windows is fully expanded, that you can see both (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) and (\\HT-2.DOMAIN.COM\DAG.DOMAIN.COM). This leads administrators to believe that two file share witness servers are currently in use.
Thankfully both of these perceived conditions are false. The command was both successful in changing the witness server and only one file share witness is in use.
Each cluster resource has a display name and a set of public and private properties. Unfortunately when using set-databaseavailabilitygroup to change the witness server, the File Share Witness resource private property for where the witness is stored is updated but the public property display name, which contains the previous witness server, is not. Let’s take a look at this further.
Using cluster.exe or powershell I can review the private properties of the File Share Witness resource. (Command output truncated to show relevant values only.)
Cluster.exe <cluster> res <resource> /priv <or> /prop (Windows 2008 & Windows 2008 R2)
Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /prop
Listing properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':
SR File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) Name File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)
Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /priv
Listing private properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':
S File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) SharePath \\HT-1.domain.com\DAG.domain.com
Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | fl (Windows 2008 R2 Only – Public Properties)
Name : File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) State : Online OwnerGroup : Cluster Group ResourceType : File Share Witness
Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | Get-ClusterParameter fl (Windows 2008 R2 Only – Private Properties)
Name : SharePath IsReadOnly : False ParameterType : String Value : \\HT-1.domain.com\DAG.domain.com
At this time a set-databaseavailability group is issued to change the witness server. After the command completes successfully, the previous commands are run. (Command output truncated to show relevant values only.)
S File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) SharePath \\HT-2.domain.com\DAG.domain.com
(Note: The SharePath in the previous output reflects the new witness server as expected)
Name : SharePath IsReadOnly : False ParameterType : String Value : \\HT-2.domain.com\DAG.domain.com
As you can see the set-databaseavailability group command did complete it’s task successfully by updating the SharePath attribute of the quorum resource to utilize the correct witness server.
The use of mount points for Exchange is becoming more common place in many installations. Some customers feel the best implementation of mount points consists of a small root disk with mount points created from folders on that disk.
For example, I may have a Drive L: that is 10 megs and I may create 4 folders on this drive (Database1 / Database2 / Database3 / Database4). I will then create mount points utilizing the folders created from the L drive.
There are certain process in Exchange that often check for free drive space prior to performing certain operations. Unfortunately these processes are not necessarily mount point aware – therefore they end up querying the free drive space of the lettered volume rather than the mount point. One of these process is MSSearch.
MSSearch by default creates a catalog data folder co-located with each EDB file. In our example above the catalog data folder and the edb file would be in L:\Database1 (where Database1 is the mount point). In this this case the L drive has 10 megs free space but the Database1 mount point has 1.5 terabytes of free space. When MSSearch attempts to initialize the initial catalog this operation fails as the drive space reported by the disk L is not sufficient (even though there is plenty of space where the actual catalog is stored).
Here is an example of some events you may see when this occurs.
Log Name: Application Source: MSExchange Search Indexer Date: 6/14/2010 12:11:20 PM Event ID: 104 Task Category: General Level: Error Keywords: Classic User: N/A Computer: server.company.com Description: Exchange Search Indexer failed to enable the Mailbox Database DATABASE(GUID = 58c0ed8a-dbfc-4d55-b265-8a80f1dc477b) after 1 tries. The last failure was: System.ComponentModel.Win32Exception: Unable to SetProperty FTE_PluginList on catalog ExSearch-58c0ed8a-dbfc-4d55-b265-8a80f1dc477b-26fc1c62-d3e8-4711-b3c9-3bb0b32aec0a. Error = -2147215320 at Microsoft.Exchange.Msfte.CFTEAdmin.SetProperty(CatalogState catalogInfo, PropertyScope propertyScope, String propertyName, Object propertyValue, Boolean throwOnFailure) at Microsoft.Exchange.Msfte.CFTEAdmin.CreateCatalog(CatalogState catalogInfo) at Microsoft.Exchange.Search.Globals.CreateCatalog(CatalogState state, String reason) at Microsoft.Exchange.Search.Globals.RecreateCatalogAndPropertyStore(CatalogState catalogInfo, String reason) at Microsoft.Exchange.Search.CatalogState.CreateNew(String reason) at Microsoft.Exchange.Search.CatalogState.Reset(String reason) at Microsoft.Exchange.Search.CatalogState.HandleMountCatalogException(Exception exception) at Microsoft.Exchange.Search.Globals.CheckAndInitializeCatalog(CatalogState catalogInfo) at Microsoft.Exchange.Search.Driver.ProcessNewCatalogInternal(CatalogState catalog, List`1 mdbsToCrawl, Int32& numberOfDisabledMDBs). It will retry after 10 minutes.
Log Name: Application Source: ExchangeStoreDB Date: 6/14/2010 12:12:51 PM Event ID: 222 Task Category: Database recovery Level: Error Keywords: Classic User: N/A Computer: server.company.com Description: At '6/14/2010 11:12:50 AM' the Microsoft Exchange Information Store Database ‘DATABASE' copy on this server experienced a corrupted search catalog. The error returned by failover was "There is only one copy of this mailbox database (DATABASE). Automatic recovery is not available.". Consult the event log on the server for other "ExchangeStoreDb" and "MSExchange Search Indexer" events for more specific information about the failures.
The important information is actually contained in the first event – the error code –2147215320. This error code translates to CI_E_CONFIG_DISK_FULL.
To resolve this issue you can:
Once this is done restarting the MSSearch services may be necessary so that initial catalog creation can occur.
When Exchange 2007 CCR is installed on Windows 2008 or Windows 2008 R2 the following error may be noted in the application log of the passive node:
Log Name: Application Source: MSExchangeRepl Event ID: 2104 Task Category: Service Level: Error Keywords: Classic User: N/A Computer: MACHINE Description: Log file action LogCopy failed for storage group EXCLUST01\SG2. Reason: CreateFile(\\Server\StorageGroupGUID$\LogFile.log) = 2
If the CCR cluster is not utilizing continuous replication host names the following event series may also be noted:
Event ID : 2147 Raw Event ID : 2147 Source : MSExchangeRepl Type : Error Machine : SERVER Message : There was a problem with 'ActiveNode', which is an alternate name for 'ActiveNode'. The list of aliases is now 'ActiveNode', and the alias 'was' removed from the list. The specific problem is 'CreateFile(\\ActiveNode\StorageGroupGuid$\LogFile.log) = 2'.
ID: 2127 Level: Information Provider: MSExchangeRepl Machine: SERVER Message: The system has detected a change in the available replication networks. The system is now using network 'ActiveNode' instead of network 'ActiveNode' for log copying from node ActiveNode.
In this situation if the solution is aggressively monitored you may not that replication is temporarily failed and then resumes automatically as healthy. This occurs due to a temporary pause in replication when the error condition is detected, while the replication service attempts to find other replication paths, and then automatically re-attempts the same copy operation.
If the CCR cluster is utilizing continuous replication host names the following event series may also be noted:
Event ID : 2147 Raw Event ID : 2147 Source : MSExchangeRepl Type : Error Machine : SERVER Message : There was a problem with ‘ReplicationHostName’, which is an alternate name for 'ActiveNode'. The list of aliases is now 'ActiveNode', and the alias 'was' removed from the list. The specific problem is 'CreateFile(\\ReplicationHostName\StorageGroupGUID$\LogFile.log) = 2'. ID: 2127 Level: Information Provider: MSExchangeRepl Machine: SERVER Message: The system has detected a change in the available replication networks. The system is now using network 'ActiveNode' instead of network ‘ReplicationHostName’ for log copying from node ActiveNode.
Error 2 is ERROR_FILE_NOT_FOUND
In this situation the error is detected on the replication host name. The replication service will temporarily pause replication while other network paths are enumerated. If other continuous replication host names are in use, the replication serivce will select an alternate replication host name and automatically resume log copying. If the only path valid is the “public” path, the replication service will begin copying log files over the “public” network. Eventually this error occurs on the public network, forcing network re-enumeration to occur and replication to automatically switch back to the replication network. If the solution is aggressively monitored, the replication status may be failed during this switch but will automatically resume healthy.
In almost all incidences these errors are considered benign to the operation of the Exchange Server.
The replication service is extremely aggressive in its attempts to copy log files. The replication service is always aware of the next log file in the series that requires copying to the passive node. As part of normal processes the replication service may query multiple times for the presence of this file and make copy attempts. These attempts may result in the replication service querying for a log file that is not fully available. Under Windows 2003 this was not necessarily an issue. Windows 2008 introduces a component into SMBv2 that may cause this to be a problem.
SMBv2 introduces status caching into the LanManWorkstation service. When an application requests information from a file share, the workstation service caches the response from the server hosting the share. Subsequent requests for the same information are returned from cache rather than re-contacting the server hosting the share. Eventually this cache will expire (in our case it expires by the time replication is failed / resumed <or> a switch between replication host names occur). The replication service has received feedback that the log file in question should not be available for copy, attempts to copy it, and receives an older return status that the file is not ready (even though the file does exist on the source at the time the attempt is made). In turn the replication service detects this as an error condition and takes action.
From a Windows 2008 / Windows 2008 R2 perspective this is by design.
To correct these errors on an Exchange 2007 / Windows 2008 <or> Exchange 2007 / Windows 2008 R2 implementation, the following registry keys should be set to a zero (0) value and the nodes rebooted:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters
FileInfoCacheLifetime [DWORD]
FileNotFoundCacheLifetime [DWORD]
DirectoryCacheLifetime [DWORD]
If the DWORDs are not present they may need to be created. The recommended value is HEX / DEC 0.
More information on these keys can be found here: http://technet.microsoft.com/en-us/library/ff686200(WS.10).aspx (Note that registry path in the article is missing the SERVICES hive – correct path in blog post).
(An Exchange 2007 version of this article can be found here: http://blogs.technet.com/b/timmcmic/archive/2009/04/27/network-port-design-and-exchange-2007-clusters.aspx)
A question that has come up is how many network ports should I have in my DAG members and how should I use them.
I generally see three different hardware configurations:
In some hardware there are now 4 port cards. The information contained here can be expanded to include additional hardware / port configurations as they become available.
You’ll note that there is no configuration with a single network port – I personally do not recommend having only a single network port even though this is now a supported implementation. (Note: VLANS to a single port are not two network interfaces).
Network Teaming
In the recommendations I’ll outline next you will see references to the use of network teaming. It’s important to note that Microsoft does not support network teaming as this is hardware vendor supported and designed technology. What it is though is a recognition that in absence of anyway to provide multiple client facing ports for Exchange network teaming does have a valid place in the overall high availability design.
When using network teaming, only the client facing network should be a teamed adapter and at all times the team created for NETWORK FAULT TOLERANCE. Do not, for an Exchange instance, use any type of load balancing between ports.
For non-client facing networks it is not necessary to implement at network team (these would typically be your “heartbeat” networks). Windows clustering has the ability to balance and use all interfaces on the cluster designated for cluster use without the need to establish teaming for cluster / heartbeat communications.
From a support perspective any customer that establishes a teamed interface for the client side network should recognize that they may be asked to dissolve the team to support troubleshooting efforts.
MAPI Networks
For Exchange 2010 DAG MAPI networks I recommend using a network fault tolerant team consisting of two ports. More ports maybe utilized if they are available.
Replication Networks
After a team has been utilized for the MAPI network the remaining network interfaces can be divided into replication networks. I do not recommend that any form of network teaming be utilized on replication networks. Utilization of teaming on replication networks – although supported – is redundant. Both the replication service and cluster service have the ability to switch between these additional networks as necessary. All additional networks must be on their own subnet, subnets between networks may not overlap on the host.
Cluster Networks
There is no reason to establish dedicated cluster heartbeat networks with Exchange 2010 DAG members as cluster can utilized all configured interfaces between hosts for heartbeat exchange.
==============================
Updated – 6/2/10 – It is supported to use teaming on non-client facing networks although in theory this is redundant as both the replication service and cluster service have the ability to utilize multiple secondary interfaces.
There are some installations that utilize a single node cluster hosting a clustered mailbox server for Exchange 2007. It may become necessary to perform an upgrade of the solution to a different service pack. This process can introduce some issues as the upgrade and upgrade processes assume that there is a second node in the cluster that can own both the cluster core resources and Exchange resources.
I wanted to outline a process that can be utilized to upgrade service packs on a single node cluster.
In order to being the upgrade process the clustered and Exchange services must be stopped. This can be performed on the Exchange server by:
1) Open an instance of Exchange Management Shell.
2) Issue a stop-clusteredmailboxserver –identity <CMSName> –stopreason “Upgrade”
3) Stop the cluster core resources group / cluster group. Issue a cluster.exe group “cluster group” /offline.
4) Exit the Exchange Management Shell
At this point the Exchange services will be stopped as well as the cluster core resources. We should be able to begin the upgrade of the Exchange binaries at this time. Use the following process to begin the Exchange upgrade.
The upgrade of the mailbox role binaries is completed by running this command from the service pack media:
setup.com /mode:upgrade
When the mailbox role upgrade has completed you can install any pending roll up updates.
The next step of the upgrade is to issue an upgrade to the clustered mailbox server. To upgrade the clustered mailbox server:
1) Start the cluster core resources by issuing the following command – cluster.exe group “cluster group” /online
2) At this point the cluster core resources are online and cluster services should be started with the Exchange resources in an offline state (the same state prior to stopping the cluster services).
3) Upgrade the clustered mailbox server by running the command - setup.com /uprgadeCMS. When the command has completed the Exchange resources should be online on the node where the upgrade was performed.
These instructions were tested using Exchange 2007 SP1 on Windows 2008 single node Cluster Continuous Replication (CCR) installation.
Every Exchange 2010 server has a process internal to the replication service known as Active Manager. The Active Manager is responsible for all database mount, dismount, and move operations that occur in Exchange 2010.
When a server is a standalone server, Active Manager is configured as a Standalone Active Manager.
When a server is a member of a Database Availability Group (DAG), Active Manager is either configured as:
The Active Manager status in a DAG is determined by the node that owns the cluster core resources. If a node owns the cluster core resources group, this node is then known as the Primary Active Manager (PAM). All other nodes successfully participating in the cluster and not owning the cluster core resources are Secondary Active Managers.
Let’s take a look at an example database availability group.
DAGName: DAG
DagMembers: DAG-1,DAG-2,DAG-3,DAG-4
Running get-databaseavailabilitygroup –identity DAG –status | fl name,primaryActiveManager you can determine which machine currently owns the cluster core resources and is acting as the PAM.
Get-DatabaseAvailabilityGroup -Identity DAG -Status | fl name,primaryactivemanager
Name : DAG PrimaryActiveManager : DAG-3
Using cluster.exe we can also confirm the owner of the cluster core resources group
cluster.exe DAG.domain.com group
Group Node Status -------------------- --------------- ------ cluster group DAG-3 Online
Using the cluster command line, the cluster core resources can be moved to another DAG member and the PAM will subsequently change.
cluster.exe DAG.domain.com group "cluster group" /moveto:DAG-4
Moving resource group 'cluster group'...
Group Node Status -------------------- --------------- ------ cluster group DAG-4 Online
Name : DAG PrimaryActiveManager : DAG-4
Remember that Active Manager runs inside the Microsoft Exchange Replication service which is installed on every Exchange 2010 Mailbox Role Server. This is important – if the replication service on a DAG member is not started, but that DAG member owns the cluster core resources, database mount / dismount / move functionality will not function.
Here is an example…
Currently the cluster core resources are owned on the node DAG-4 which is successfully participating in the cluster DAG. Using the services control panel the Microsoft Exchange Replication service on the server DAG-4 was stopped. We can confirm using the commands above that DAG-4 is still seen as the PAM.
cluster dag.domain.com group Listing status for all available resource groups:
Group Node Status -------------------- --------------- ------ Cluster Group DAG-4 Online Available Storage DAG-1 Offline
Using test-replicationHealth and test-serviceHealth we can see that the replication service on node DAG-4 is unavailable.
Server Check Result Error ------ ----- ------ -----
DAG-4 ClusterService Passed DAG-4 ReplayService *FAILED* The Microsoft Exchange Replication service is not running on s... DAG-4 DagMembersUp Passed
Role : Mailbox Server Role RequiredServicesRunning : False ServicesRunning : {IISAdmin, MSExchangeADTopology, MSExchangeIS, MSExchangeMailboxAssistants, MSExchangeMailSubmission, MSExchangeRPC, MSExchangeSA, MSExchangeSearch, MSExchangeServiceHost, MSExchangeThrottling, MSExchangeTransportLogSearch, W3Svc, WinRM} ServicesNotRunning : {MSExchangeRepl}
At this time a dismount operation on a database was issuing using the dismount-database command. An error is immediately returned:
Dismount-Database DAG-DB0
Confirm Are you sure you want to perform this action? Dismounting database "DAG-DB0". This may result in reduced availability for mailboxes in the database. [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): y
Couldn't dismount the database that you specified. Specified database: DAG-DB0; Error code: An Active Manager operation failed. Error: The Microsoft Exchange Replication service may not be running on server DAG-4.domain.com. Specific RPC error message: Error 0x6d9 (There are no more endpoints available from the endpoint mapper) from cli_MountDatabase. + CategoryInfo : InvalidOperation: (DAG-DB0:ADObjectId) [Dismount-Database], InvalidOperationException + FullyQualifiedErrorId : D64CA7E2,Microsoft.Exchange.Management.SystemConfigurationTasks.DismountDatabase
To fix this error:
It is important that the replication service be monitored on all DAG members to ensure it remains functional.
*Updated – 5/30/2010 – Corrected the commandlet for testing services –> test-serviceHealth instead of test-serverHealth.
*Updated – 6/22/2011 – Corrected table formatting of output.
In Exchange 2010 we achieve high availability of mailbox databases by utilizing a Database Availability Group (DAG). When a DAG is utilized, and mailbox database copies are created on DAG members, they are either MOUNTED (active) or have a copy status – for example HEALTHY (passive).
If an administrator or another process gracefully stops the Information Store Service on a DAG node, any database copies that were mounted on that server enter a DISMOUNTED state. These copies do not fail over to another node, as the shutdown was graceful and not the result of an error condition that would trigger a failover event to occur. Should the processed have crashed or otherwise became unavailable, this would be detected as an error and the database instances failed over.
Let’s take a look at this.
In this environment I have a four mailbox server DAG. Database DAG-DB0 is replicated to all members of the DAG. Currently DAG-DB0 is mounted on mailbox server DAG-1.
Get-mailboxdatabasecopystatus *\DAG-1
Name Status ---- ------ DAG-DB0\DAG-1 Mounted DAG-1-DB0\DAG-1 Mounted DAG-DB1\DAG-1 Healthy
On DAG-1 I issued a net stop msexchangeis. This gracefully stopped the Information Store Service.
Log Name: System Source: Service Control Manager Date: 3/21/2010 11:27:19 AM Event ID: 7036 Task Category: None Level: Information Keywords: Classic User: N/A Computer: DAG-1.domain.com Description: The Microsoft Exchange Information Store service entered the stopped state.
Reviewing the mailbox database copy status for DAG-1, the following is noted. (Get-mailboxdatabasecopystatus *\DAG-1)
Name Status ---- ------ DAG-DB0\DAG-1 Dismounted DAG-1-DB0\DAG-1 Dismounted DAG-DB1\DAG-1 Healthy
As a response to the IS gracefully shutting down the mailbox database copies that were mounted on that host are now dismounted.
Subsequently I issued the command net start msexchangeis to start the Information Store Service.
Log Name: System Source: Service Control Manager Date: 3/21/2010 11:34:18 AM Event ID: 7036 Task Category: None Level: Information Keywords: Classic User: N/A Computer: DAG-1.domain.com Description: The Microsoft Exchange Information Store service entered the running state.
After restarting the service the databases that were previous dismounted are now mounted. (Get-mailboxdatabasecopystatus *\DAG-1)
When using continuous replication in Exchange 2007, an operation that sometimes needs to be performed is a database seed. This operation is typically performed as part of enabling replication, and infrequently it is performed as part of the process for recovering from divergence.
Seeding is most often performed with the Update-StorageGroupCopy cmdlet. During seeding, an ESE streaming backup is performed on the source database. This API is used to copy the database from the source to the target. There are sometimes where this process fails or for various reasons cannot be utilized. This means an alternate way to seed the database is needed.
One method is to perform a manual offline seeding. In this operation, the source database is dismounted, verified to be in a clean shutdown state, and then manually copied offline to the target. This can obviously be inconvenient, since the source database has to be down while the copy procedure is being performed.
Another method is to use a VSS backup of the database to seed the database copy. You can use VSS to back up the database, and VSS to restore the database. (Sorry – if you are using the online streaming backup API for your databases you will not be able to use these instructions).
When using an Exchange-aware VSS application, there are typically four destinations for a restore (note, your backup software may not enable all the options):
To use the VSS backup and restore method, you would choose to restore to the file system.
The following steps outline a high level process on how to utilize a VSS backup and restore to file system to complete an online offline database seed operation.
====================================
The first step is to enable replication for the storage group. In CCR this is handled for you automatically every time you create a database. When using SCR or LCR, this is accomplished using the Enable-StorageGroupCopy cmdlet. It is important to ensure that neither circular logging nor backups truncate any of the log files necessary to complete this process.
(SCR) Enable-StorageGroupCopy –Identity <ServerName\StorageGroupName> –StandbyMachine <SCRTargetName> –SeedingPostponed
(LCR) Enable-DatabaseCopy –Identity <ServerName\DatabaseName> –CopyEdbFilePath “path\database.edb”
(LCR) Enable-StorageGroupCopy –Identity <ServerName\StorageGroupName> –CopyLogFolderPath <path> –CopySystemFolderPath <path> –SeedingPostponed
For more information on enabling SCR, please see my blog post at http://blogs.technet.com/timmcmic/archive/2009/01/22/inconsistent-results-when-enabling-standby-continuous-replication-scr-in-exchange-2007-sp1.aspx
If you have already enabled continuous replication for the storage group, proceed to the second step.
The second step is to ensure that the storage group copy is in a suspended state. Storage group copies can be suspended either in bulk or one at a time. The following are example commands:
(All Storage Groups) Get-StorageGroup –Server <SourceServerName> | Suspend-StorageGroupCopy –StandbyMachine <TargetMachineName>
(Single Storage Group) Suspend-StorageGroupCopy –identity <ServerName\StorageGroupName> –StandbyMachine <TargetMachineName>
It is important that in the SCR environment these commands are run on both the source and target servers. All servers should indicate a suspended status, reflecting that both Active Directory replication and the Microsoft Exchange Replication service configuration updates occurred successfully.
The third step is to note the important paths that are necessary to complete the rest of these steps. Specifically, we are interested in the storage group log file path, the system folder path and copy system folder path, and the log file prefix. For the mailbox database we are interested in the database file path and copy database file paths.
To get all paths for all storage groups on the source, use the following command:
Get-StorageGroup –Server <ServerName> | fl Name,LogFolderPath,SystemFolderPath,CopyLogFolderPath,CopySystemFolderPath,LogFilePrefix
This will give you a formatted list of storage group names, log paths, and system paths.
To get the paths for all mailbox databases, use the following command:
Get-MailboxDatabase –Server <ServerName> | fl Name,EdbFilePath,CopyEdbFilePath
This will give you a formatted list of mailbox database names and mailbox database paths.
Here is an example of the output you can expect to see (copy path attributes will only be populated if you are utilizing LCR):
Name : Mailbox Database LCR EdbFilePath : d:\SG1\DB1.edb CopyEdbFilePath : d:\SG1-LCR\DB1.edb
Name : Mailbox Database CCR or SCR EdbFilePath : d:\SG2\DB2.edb CopyEdbFilePath :
Name : Storage Group LCR LogFolderPath : d:\SG1 SystemFolderPath : d:\SG1 CopyLogFolderPath : d:\SG1-LCR CopySystemFolderPath : d:\SG1-LCR LogFilePrefix : E00
Name : Storage Group CCR or SCR LogFolderPath : d:\SG2 SystemFolderPath : d:\SG2 CopyLogFolderPath : CopySystemFolderPath : LogFilePrefix : E01
The fourth step is to verify that the source log file sequence is in order. If the source log file sequence has been manually manipulated, and if any log file gaps are present, this results in a failure of the seed operation. This step ensures that log files are in sequence on the source machine.
To ensure that the log sequence on the source machine is in the correct order, perform the following operations:
1. Open a command prompt and navigate to the log directory of the storage group. This path can be found from the output gathered in step 3 above.
2. Run the following eseutil command:
eseutil /ml <LogFilePrefix>
The log file prefix can be found from the output gathered in step 3.
When you run this command it will scan every log file found in the source directory. If any gaps or errors are identified, you cannot continue with these steps. If the command completes and errors on the last log file in the series this is expected, as the Exx.log is currently open for writing and cannot be scanned. The following is sample output that you should receive for a storage group that is online.
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.02 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating FILE DUMP mode...
Verifying log files... Base name: e00
Log file: d:\SG1\E0000001353.log - OK Log file: d:\SG1\E0000001354.log - OK Log file: d:\SG1\E0000001355.log - OK Log file: d:\SG1\E0000001356.log - OK Log file: d:\SG1\E0000001357.log - OK Log file: d:\SG1\E0000001358.log - OK Log file: d:\SG1\E0000001359.log - OK Log file: d:\SG1\E000000135A.log - OK Log file: d:\SG1\E000000135B.log - OK Log file: d:\SG1\E000000135C.log - OK Log file: d:\SG1\E000000135D.log - OK Log file: d:\SG1\E000000135E.log - OK Log file: d:\SG1\E000000135F.log - OK Log file: d:\SG1\E0000001360.log - OK Log file: d:\SG1\E0000001361.log - OK Log file: d:\SG1\E0000001362.log - OK Log file: d:\SG1\E0000001363.log - OK Log file: d:\SG1\E0000001364.log - OK Log file: d:\SG1\E0000001365.log - OK Log file: d:\SG1\E0000001366.log - OK Log file: d:\SG1\E0000001367.log - OK Log file: d:\SG1\E0000001368.log - OK Log file: d:\SG1\E0000001369.log - OK Log file: d:\SG1\E00.log ERROR: Cannot open log file (d:\SG1\E00.log). Error -1032.
Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use) after 368.625 seconds.
The fifth step is to perform a VSS backup of the database. Please consult with your backup vendor to ensure that a successful FULL backup is performed. Please also make sure that a consistency check of the backup is performed.
The sixth step is to restore the VSS backup. When you perform the restore, you should select the option to restore to file system. This may require that you restore to the file system of an Exchange server, so it may be necessary to ensure that sufficient free space exists on a volume on the Exchange server where the restore will be performed.
For SCR and CCR restore to the original server. In our example we will say that we are restoring to the original server at x:\Restore.
For LCR we can restore to the CopyEdbFilePath. In our example you would restore to d:\SG1-LCR. This will prevent us from having to run a copy operation at a later time.
If multiple databases are being restored, I recommend that databases be restored individually.
At this point we now have the EDB file on the file system, and we will use it for the seeding operation.
The seventh step is to ensure that the target paths are ready to have the database moved in place. The paths referenced in these steps can be obtained from the output gathered in step 3.
For SCR – ensure that the logFolderPath, systemFolderPath, and edbFilePath are empty on the SCR target.
For CCR – ensure that the logFolderPath, systemFolderPath, and edbFilePath are empty on the passive node.
For LCR – ensure that the copyLogFolderPath, copySystemFolderPath, and copyEdbFilePath are empty.
At this point the destination paths are empty and ready for the database to be moved.
We now need to create the directory structure where logs, system, and database files will be copied.
For SCR and CCR - create the log, system, and database folder. In our example logs, system, and database files are located at d:\SG1. Therefore on the SCR target or CCR passive node I would create the directory structure d:\SG1.
For LCR - we would create the copy folder. In our example we have copies placed in d:\SG1-LCR. Therefore on the server I would create the directory structure D:\SG1-LCR.
If you are using nested folders you need to create the entire directory structure.
The eighth step is to move the restored database to the target directory. This can be accomplished in a few different ways, but I will make a recommendation below.
For CCR and SCR:
From the source server map a drive to the drive$ share of the target. For example, I would map the drive Y: to \\SCRTarget\d$\ using our example.
Open a command prompt and navigate to the restore directory. In our example this is X:\Restore
Use Eseutil to copy the database from the source directory to the target directory. The sample command using our example is:
eseutil /y SG1-DB1.edb /d y:\SG1-DB1.edb
Here is the expected output from this command:
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.02 Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating COPY FILE mode... Source File: SG1-DB1.edb
Destination File: y:\SG1-DB1.edb
Copy Progress (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................
Operation completed successfully in 13.281 seconds.
At this point the copy has been seeded on the target server.
For LCR, this step is not necessary as the restoration to file system was performed to the LCR location.
Information on the usage of Eseutil can be found here. http://technet.microsoft.com/en-us/library/aa998249(EXCHG.80).aspx
The ninth step is to verify the health of the copied database. We need to ensure that the database was not corrupted as a part of the copy process.
For SCR and CCR:
Log on locally to the SCR target or CCR passive node, open a command prompt, and navigate to the database directory. In our example this would be d:\SG1.
Use Eseutil /k to perform a checksum of the database:
eseutil /k SG1-DB1.edb
The following output will be observed when the command completes:
Initiating CHECKSUM mode... Database: SG1-DB1.edb Temp. Database: TEMPCHKSUM3888.EDB
File: SG1-DB1.edb
Checksum Status (% complete)
514 pages seen 0 bad checksums 0 correctable checksums 129 uninitialized pages 0 wrong page numbers 0x4676 highest dbtime (pgno 0x86) 65 reads performed 4 MB read 1 seconds taken 4 MB/second 2755 milliseconds used 42 milliseconds per read 78 milliseconds for the slowest read 15 milliseconds for the fastest read
Operation completed successfully in 0.140 seconds.
We are interested in ensuring that there are 0 bad checksums (bolded line above).
For LCR, this command should be run locally on the machine.
Open a command prompt and navigate to the copy database directory. In our example, this would be d:\SG1-LCR.
The last step in the process is to resume the storage group copy::
(SCR): Get-StorageGroup –Server <SourceServerName> | Resume-StorageGroupCopy –StandbyMachne <SCRTargetName>
(CCR / LCR): Get-StorageGroup –Server <SourceServerName> | Resume-StorageGroupCopy
(Note: These command resume storage group copy for all storage groups. If you have a storage group that has copy suspended for another reason it may be necessary to resume single storage groups).
When replication has resumed successfully, you can note the following events in the application log indicating that replication began copying log files.
Event Type: Information Event Source: MSExchangeRepl Event Category: Action Event ID: 2084 Date: 3/16/2010 Time: 10:12:50 AM User: N/A Computer: SERVER Description: Replication for storage group SERVER\Storage Group SCR or CCR has been resumed.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Information Event Source: MSExchangeRepl Event Category: Service Event ID: 2114 Date: 3/16/2010 Time: 10:13:19 AM User: N/A Computer: SERVER Description: The replication instance for storage group SERVER\Storage Group SCR or CCR has started copying transaction log files. The first log file successfully copied was generation 31201.
The following are links to references from this post.
· Enable-StorageGroupCopy (http://technet.microsoft.com/en-us/library/aa996389(EXCHG.80).aspx)
· Enable-DatabaseCopy (http://technet.microsoft.com/en-us/library/aa996389(EXCHG.80).aspx)
· Suspend-StorageGroupCopy (http://technet.microsoft.com/en-us/library/aa998182(EXCHG.80).aspx)
· Get-StorageGroup (http://technet.microsoft.com/en-us/library/aa998331(EXCHG.80).aspx)
· Get-MailboxDatabase (http://technet.microsoft.com/en-us/library/bb124924(EXCHG.80).aspx)
· ESEUTIL (http://technet.microsoft.com/en-us/library/aa998249(EXCHG.80).aspx)
· Resume-StorageGroupCopy (http://technet.microsoft.com/en-us/library/bb124529(EXCHG.80).aspx)
Recently I have been asked how a file server / file share cluster can be utilized to host the file share witness directory for an Exchange 2010 Database Availability Group. I think it is important to note that this is not a recommended deployment.
With Exchange 2010 changing the witness directory is as simple as recognizing that the server hosting the witness is unavailable for an extended period of time and running the set-databaseavailabilitygroup command with the –witnessServer and specifying a remaining Exchange server <or> another server where the Exchange Trusted Subsystem group has been granted local administrator rights.
Due to the permissions model necessary for Exchange 2010 these a Windows 2003 File Share cluster should not be utilized for these functions. Utilizing a Windows 2003 File Share cluster causes incompatibilities with the set-databaseavailabilitygroup commandlet.
Node Configuration
Using server manager the Exchange Trusted Subsystem was added to the Local Administrator group of each node in the cluster.
Cluster Configuration
The cluster services were established utilizing the appropriate cluster quorum model and the desired number of nodes.
An empty service and application was created. Inside the empty service and application a client access point was created. An appropriate name and IP address were specified. (For this example the name is FileServer1).
The shared storage that will host the file share witness was already available in the Available Storage group. The desired disks were added to the service or application group.
At this time it’s important to note the drive letter associated with the disk where you want the witness directory hosted. (For this example the drive letter is E).
DAG Configuration
In our example the name of our dag is DAG. The domain where the DAG is created is exchange.msft.
By default when only a witness server is specified the default DAG directory is c:\DAGFileShareWitness\DAG-FQDN. (In our example c:\DAGFileShareWitness\DAG.exchange.msft.)
This is where the configuration issue arises. The C drive is not valid in the service or application that we created in cluster. Only the E drive is a valid drive. Therefore, we will have to utilize the E drive in the witness path. This will require us to set both the WitnessServer and WitnessDirectory attribute.
To configure the DAG to utilize the FileServer1 name and a directory on the E drive which is valid in the group, run the following command:
set-databaseavailabilitygroup –identity DAG –witnessServer FileServer1 –witnessDirectory e:\DAGFileShareWitness\DAG.exchange.msft
At this time when you review the service or application holding the name and disk resource, you will see an additional file server resource created. The share is now available on the file server cluster.
Although Exchange 2010 no longer deploys a cluster resource model we still use Windows Failover Clustering service for certain functions.
When a Windows 2008 / 2008 R2 cluster is created, the cluster core resources are groups together in the ‘Cluster Group’. THe Cluster Group is a hidden group that contains the following resources:
You can see the cluster core resources in failover cluster manager by selecting the cluster name in the upper left hand pane. In the center pane, expand the cluster core resources section.
The cluster core resource group can also be seen using cluster.exe (or in Windows 2008 R2 cluster powershell extensions).
Windows 2008 / Windows 2008 R2: Cluster.exe DAG.company.com group
cluster.exe dag.company.com group Listing status for all available resource groups:
Group Node Status -------------------- --------------- ------ Cluster Group DAG-1 Online Available Storage DAG-1 Offline
Windows 2008 R2: Get-ClusterGroup –Cluster DAG.company.com
PS C:\Users\Administrator> Get-ClusterGroup -Cluster DAG.company.com
Name OwnerNode State ---- --------- ----- Cluster Group dag-1 Online Available Storage dag-1 Offline
From an Exchange 2010 perspective you do not really need to manage the cluster core resources. As members join and depart the cluster this resource group will be automatically moved to a remaining member. Each member of the DAG should have the ability to arbitrate and fully bring online the cluster core resources.
When a cluster is created in Windows 2008 or Windows 2008 R2, the cluster service enumerates all network ports found on the nodes. These network ports are then combined into cluster networks. You can view the cluster networks in failover cluster manager by expanding the cluster name and expanding networks.
You can also view the cluster networks using cluster.exe or powershell.
Windows 2008 / Windows 2008 R2: cluster.exe dag.company.com network
cluster.exe dag.company.com network Listing status for all available networks:
Network Status ---------------------------------------- ----------- Cluster Network 2 Up Cluster Network 4 Up Cluster Network 1 Up
Windows 2008 R2: get-clusternetwork –cluster DAG.company.com
Get-ClusterNetwork -Cluster DAG.company.com
Name State ---- ----- Cluster Network 1 Up Cluster Network 2 Up Cluster Network 4 Up
A cluster network has three settings:
You can see these settings in failover cluster manager by getting the properties of a cluster network.
You can also view the network role either by using cluster.exe or powershell.
Windows 2008 / Windows 2008 R2: cluster.exe dag.company.com network "Cluster Network 1” /prop
cluster dag.company.com network "Cluster Network 1" /prop
Listing properties for 'Cluster Network 1':
T Network Name Value -- -------------------- ------------------------------ ----------- SR Cluster Network 1 Name Cluster Network 1 MR Cluster Network 1 IPv6Addresses MR Cluster Network 1 IPv6PrefixLengths MR Cluster Network 1 IPv4Addresses 10.0.0.0 MR Cluster Network 1 IPv4PrefixLengths 24 SR Cluster Network 1 Address 10.0.0.0 SR Cluster Network 1 AddressMask 255.255.255.0 S Cluster Network 1 Description D Cluster Network 1 Role 3 (0x3) D Cluster Network 1 Metric 1200 (0x4b0) D Cluster Network 1 AutoMetric 1 (0x1)
Windows 2008 R2: get-clusternetwork –cluster DAG.company.com | fl name,role
Get-ClusterNetwork -Cluster DAG-1.company.com | fl name,role
Name : Cluster Network 1 Role : 3
Name : Cluster Network 2 Role : 1
Name : Cluster Network 4 Role : 1
The role of the networks can also be viewed in the registry of each node. This information is located at: HKEY_LOCAL_MACHINE\Cluster\Networks. Each cluster network is represented by a subkey which is the GUID of the network. Expanding the GUID, you will see sub-values including Name and Role.
[HKEY_LOCAL_MACHINE\Cluster\Networks\2cd2b920-0a2a-4851-bb24-de02d4a70b7e] @="class mscs::TmNetworkInfo" "Id"="2cd2b920-0a2a-4851-bb24-de02d4a70b7e" "Name"="Cluster Network 2" "Signature"="NETW" "Description"="" "Role"=dword:00000001 "Priority"=dword:ffffffff "Transport"="TCP/IP" "Ignore"=dword:00000000 "Address"="192.168.0.0" "AddressMask"="255.255.255.0" "IPv6Address"="" "State"=dword:00000003 "Metric"=dword:0000044c "AutoMetric"=dword:00000001
The role value can contain three different values depending on the cluster network settings. The values are:
In order for an IPv4 resource to be brought online it must be associated with a network that is configured to “Allow cluster network communications on this network” and to “Allow clients to connect through this network”. If for any reason the “Allow clients to connect through this network” option is not enabled, the IPv4 resource associated with that network will not be able to be brought online.
On an Exchange 2010 DAG member, when attempting to move the cluster core resources to another DAG member the resources may fail to come online. Specifically the IPv4 resource fails to come online which results in the network name resource failing to come online (due to dependency).
If using Failover Cluster Manager and attempting to bring online the IPv4 resource in the cluster core resources group, the following pop up error is displayed:
A review of the system log shows event 1223:
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 5/10/2010 1:14:42 PM
Event ID: 1223
Task Category: IP Address Resource
Level: Error
Keywords:
User: SYSTEM
Computer: dagNode.company.com
Description:
Cluster IP address resource 'IPv4 Static Address 2 (Cluster Group)' cannot be brought online because the cluster network 'Cluster Network 2' is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.
This Event 1223, described above, indicates that the effective setting for Cluster Network 2 is “Allow cluster network communications on this network” but does not have “Allow clients to connect through this network” set. However, when reviewing the settings in failover cluster manager for Cluster Network 2 you might see that both “Allow cluster network communications on this network” and “allow clients to connect through this network” are enabled.
The Microsoft Exchange Replication Service is responsible for assisting to maintain the cluster network configuration. There is an issue in the current Replication Service where settings are not changed. This essentially causes a difference between the setting inside the cluster and the setting displayed in Failover Cluster Management tools.
Workaround:
A quick and easy workaround for this issue is to simply reset the state of the network. There are multiple ways to accomplish this and I will outline each below. Step zero before proceeding with any other steps is to note the cluster network that is displayed in the above event since that is the network that will need to be reset (in this example Cluster Network 2).
Windows 2008 / Windows 2008 R2 – Using Failover Cluster Management Tool
The network state can be reset using Failover Cluster Manager
Next we need to enable the network for “Allow clients to connect through this network”.
The network has been reset and cluster core resources should successfully arbitrate to any DAG member with a network port in this network.
Windows 2008 / Windows 2008 R2: Using cluster.exe
cluster.exe dag.company.com network “Cluster Network 2” /prop role=1
Next, we need to enable the network for “Allow clients to connect through this network”.
cluster.exe dag.company.com network “Cluster Network 2” /prop role=3
The network has now been reset and cluster core resources should successfully arbitrate to any DAG member with a network port in this network.
Windows 2008 R2: Using powershell
Get-clusternetwork –cluster DAG.company.com –name “Cluster Network 2” | % {$_.role=1}
Next, enable the network for “Allow clients to connect through this network”.
Get-clusternetwork –cluster DAG.company.com –name “Cluster Network 2” | % {$_.role=3}
LONG TERM FIX
This issue will be fixed in Exchange 2010 Service Pack 1. The issue will not be fixed in Exchange 2010 RTM.
==========================================
Updated – 6/2/2010
Updated to list Exchange 2010 SP1 confirmed to contain fix.
http://blogs.msdn.com/ntdebugging/archive/2010/04/22/etw-storport.aspx
A fantastic new way in Windows 2008 and Windows 2008 R2 to gather information on how your server is performing with your storage…
I recently had a case come across my desk where end users reported that after changing certain attributes in the directory these attributes were reverted to original or incorrect values.
In this case what we had was a centralized portal for information where end users could request that updates be made to their accounts. For example, if I was married I could request that my last name be updated in the directory to my new last name. This of course would imply that if I had email addresses based off first and last name that I would also want those addresses updated.
In Exchange 2003 the recipient update service (RUS) would scan the directory for object changes based on the objects USN. If we found an object that was changed after the USN stamped on the RUS, we knew we would need to take action against it. In this case the RUS always utilized a single Exchange server to calculate addresses and a single domain controller to read objects and write objects.
In Exchange 2007 and Exchange 2010 the RUS has become a process initiated by a task rather then scanning the directory for changes. By default when any task is run that alters a user object, such as set-mailbox or set-user, the recipient update service is called by the task. In this case calling the recipient update service means utilizing any Exchange Server’s system attendant and utilizing any domain controller – we are no longer limited to having a single Exchange server or single domain controller responsible for these tasks.
When the task set-user or set-mailbox contacts the RUS it provides a complete set of user attributes as read from the directory. The RUS then calculates valid email addresses, and provides them back to the set task. Once this information is provided the set task then commits the changes to the directory. Proxy addresses are always calculated and sent to the task which will re-write them to the directory. Other attributes, such as name etc, are only written to the directory if they were specified in the set command to be changed.
In the example of set-user and set-mailbox we have provided the –domainController switch so that a static domain controller could be utilized. For a mailbox server the task will always attempt to find a server with the mailbox role installed (hence having a system attendant where the RUS lives) in the same Active Directory site. If one is not present, the task will look to adjacent Active Directory sites.
Let’s take a look at how this changing from a directory scan to a task based system could cause the symptoms I outlined above.
I have a program that allows a user to change their name information and their ability to be seen in the address list. My email address policies are set to stamp information based on First.Last@domain.com (PRIMARY) and alias@domain.com (SECONDARY).
Environment:
Example:
In this instance a user named BARBARA SMITH wants to change her last name to JONES. Her account currently has the following attributes relevant to our example:
givenName: BARBARA
sn: SMITH
msExchHideFromAddressList: FALSE
proxyAddresses (2): smtp:Barbara@domain.com; SMTP: Barbara.Smith@domain.com
The application has received the request to process the change. The following command is called:
set-user –identity Barbara –lastName Jones –domainController DC-2
At this point the task contacts the directory and reads the attributes. It then contacts an Exchange server, and passes in the current attributes plus the attributes that changed. At this time the RUS evaluates this information and determines that an email address change should occur. The email address change should include a primary SMTP address of Barbara.Jones@domain.com. This information is provided back to the task.
The task then executes a write operation against DC-2 since the domain controller is statically specified. The attributes written to the directory are as follows:
sn: JONES
proxyAddresses (3): smtp:Barbara@domain.com; SMTP:Barbara.Jones@domain.com; smtp:Barbara.Smith@domain.com
The next step of the application is to reset the hide from address book value. This happens regardless of whether or not the value needs to be updated. The following command is called:
set-mailbox –identity BARBARA –hiddenFromAddressList:$FALSE
At this point the task contacts the directory and reads the attributes. In this case domainController was not specified, and the task chooses at random DC1. The task reads the following values from DC1.
You’ll note that the values are the original values – this is because DC1 has not had time to fully replicate the changes from DC2.
Because I used the set-mailbox command, these attributes are passed into the RUS along with the attribute to change – in this case hiddenFromAddressList. The RUS evaluates the attribute set, and determines that the email addresses that should be on this user are barbara@domain.com and barbara.smith@domain.com.
The task receives this feedback, and on DC-1 updates the following attributes:
msExchHideFromAddressList
proxyAddresses
After the domain controllers have had time to converge if I were to look at the attributes of the user here is what I see:
msexchHideFromAddressList: FALSE
proxyAddresses (2): smtp:Barbara@domain.com; SMTP: barbara.smith@domain.com
It would essentially appear that the last name change was successful but the recipient update service did not update the user successfully. In actuality, after tracing the issue, it was determined that one command utilized the static domain controller, the other command did not, and this resulted in the RUS evaluating two sets of different information for the same user and thus appearing as if changes did not occur that should have occurred.
In the end this issue was rectified by ensuring that when multiple set operations are included in sequence that all set operations specify the same domain controller.
In Exchange 2010 after a database availability group (DAG) has been created it is necessary for administrators to add database copies to members. This is generally accomplished using the add-mailboxdatabasecopy command.
When add-mailboxdatabasecopy is run, if the copy is successfully created an online seeding operation automatically occurs. The online seeding operation that occurs utilizes the public / MAPI interface to initially seed the database.
It may be desired in certain instances to have the initial seeding operation occur over one of the private / replication networks defined for the DAG. This is currently not possible since add-mailboxdatabasecopy does not implement the –Network parameter to specify an alternate network be utilized.
You can though work around this and achieve the initial seed over the private / replication network. To do this:
Run add-mailboxdatabasecopy using the –seedingPostponed switch. This will add the copy but not attempt the initial seeding operation. It will also place replication in a suspended state.
Follow this up with update-mailboxdatabasecopy –Network and specify the private / replication DAG network. This will allow the seeding operation to occur over the replication network. Upon successful seed the replication instance will be automatically resumed.
In Exchange 2007 / Exchange 2010, the maximum number of log files that can be reached before the log file sequence is exhausted is 7fffffec (2147483628). That’s a lot of log files!
In an attempt to warn administrators of when they are approaching the maximum log file, so a planned outage can be taken to reset the log generation, we implemented an ExBPA rule to inspect current log generation and provide a warning if approaching the maximum log file.
Here is a sample output.
Current transaction log generation for storage group 'SG22' on server EVS02 is nearing maximum value. Action must be taken to ensure that the databases do not stop.
Unfortunately the rule incorrectly calculates this value based on Exchange 2003. In Exchange 2003 the maximum log generation is FFFFF (1048575).
At this time this rule can be safely ignored for Exchange 2007 / Exchange 2010 servers. Application log messages can be monitored to ensure you are not approaching maximum log generation.
In a previous blog post I outlined a failure behavior of the file share witness resource on Windows 2008 clusters.
http://blogs.technet.com/timmcmic/archive/2009/01/22/exchange-2007-sp1-ccr-windows-2008-clusters-file-share-witness-fsw-failures.aspx
Ultimately there could exist a condition where the cluster enters a lost quorum state due to the File Share Witness resource being in a FAILED status even though the witness directory was fully available.
In Windows 2008 R2 a design change was made. When the File Share Witness host server becomes unavailable, the File Share Witness resource will still fail in cluster and cause the Cluster Core Resources to move between nodes. In this case assuming the File Share Witness host server is still not available, the resource remains in a failed state. If it becomes necessary to utilize the File Share Witness to maintain Quorum, and the witness resource is in a failed state, cluster will attempt to online the witness resource. If the online is successful the witness share is alive and accessible – quorum is maintained. If the online is not successful, the witness share is not alive and accessible – a lost quorum condition is encountered.
A hotfix has been released (KB978790) that back ports this change from Windows 2008 R2 to Windows 2008.
You can download this hotfix by navigating to http://support.microsoft.com and requesting the knowledge base article 978790. There is a link to request and download this fix in the upper left hand corner. (Note: When downloading the fix the package is marked for the product Windows Vista, but this is actually for Windows 2008).
This hotfix is applicable to Exchange 2007 and Exchange 2010 when utilized with Windows 2008.
Great blog post on installing rollup updates on DAG members.
http://www.ucblogs.net/blogs/exchange/archive/2009/12/15/Installing-Exchange-2010-rollups-on-DAG-servers.aspx