When implementing an Exchange 2010 Database Availability Group it may become necessary to perform a database seeding operation. The operation is typically performed as part of adding a mailbox database copy to a DAG member but may also be performed to recover from database divergence.
Seeding is most often performed using the update-mailboxdatabasecopy command. During seeding, the target replication service sends a seeding request to the source replication service on the DAG replication port (64327 by default). The source replication service then initiates a local ESE streaming backup session to the Information Store service. Pages are read from the source database by the source replication service and transmitted to the target replication service. The target replication service then writes the pages to the target database. There are sometimes where this process fails or for various reasons cannot be utilized. This means an alternate way of seeding the database is needed.
One method is to perform a manual offline seeding. In this operation, the source database is dismounted, verified, to be in a clean shutdown state, and then manually copied offline to the target. This can obviously be inconvenient, since the source database has to be down while the copy procedure is being performed.
Another method is to use a VSS backup of the database to seed the database copy. You can use VSS to backup the database, and VSS to restore the database. (There are no longer streaming backups for Exchange 2010).
When using an Exchange-aware VSS application, there are typically four destinations for a restore (note, your backup software may not enable all the options):
1. Original mailbox database.
2. Alternate mailbox database.
3. Recovery mailbox database.
4. File system.
To use the VSS backup and restore method, you would choose to restore to the file system.
The following steps outline a high level process on how to utilize a VSS backup and restore to file system to complete an online offline database seed operation.
The first step is to enable replication for the mailbox database. This step is accomplished by utilizing the add-mailboxdatabasecopy command with the –seedingPostponed parameter. This command will add the copy and inform all replication services the copy is present. Log truncation will also be suspended since copy status is not healthy for the database. If –seedingPostponed is not specified the database seeding operation will automatically be performed.
add-mailboxdatabasecopy –identity <DBNAME> –mailboxServer <DAGMember> –seedingPostponed:$TRUE
[PS] C:\Windows\system32>Add-MailboxDatabaseCopy -Identity DAG-DB3 -MailboxServer MBX-3 -SeedingPostponed:$TRUE WARNING: Replication is suspended for database copy 'DAG-DB3' because the database copy needs to be seeded.
If you have already added a mailbox database copy for the database proceed to the second step.
The second step is to ensure that the mailbox database copy is in a suspended state. Mailbox database copies can be suspended in bulk or one at a time. The following is an example command to suspend a single mailbox database copy:
Suspend-mailboxdatabasecopy –identity <Database\Server>
[PS] C:\Windows\system32>Suspend-MailboxDatabaseCopy -Identity DAG-DB3\MBX-3
Confirm Are you sure you want to perform this action? Suspending mailbox database copy "DAG-DB3" on server "MBX-3". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a
Prior to proceeding the get-mailboxdatabasecopy status command should be utilized to verify a status of suspended. The following is an example command to verify copy status for a single database copy:
Get-mailboxdatabasecopystatus –identity <Database\Server>
[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State ---- ------ --------- ----------- -------------------- ------------ DAG-DB3\MBX-3 Suspended 16 191 11/29/2010 7:49:53 AM Failed
The third step is to note the important paths that are necessary to complete the rest of these steps. Specifically, we are interested in the mailbox database log path and database file path. To get all the paths for the mailbox database on the source server, use the following command:
get-mailboxdatabase –identity <Database> | fl name,logFilePrefix,logFolderPath,edbFilePath
[PS] C:\Windows\system32>Get-MailboxDatabase -Identity DAG-DB3 | fl name,logFilePrefix,logFolderPath,edbFilePath
Name : DAG-DB3 LogFilePrefix : E04 LogFolderPath : c:\DAG\DAG-DB3 EdbFilePath : c:\DAG\DAG-DB3\DAG-DB3.edb
The forth step is to verify that the source log file sequence is in order. If the source log file sequence has been manually manipulated, and if any log gaps are present, this results in a failure of the seed operation. This step ensures that log files are in sequence on the source machine.
To ensure that the log sequence on the source machine is in the correct order, perform the following operations:
1) Open a command prompt and navigate to the log file directory of the mailbox database. This path can be found from the output gathered in step 3 above.
2) Run the following eseutil command:
eseutil /ml <LogFilePrefix>
The log file prefix can be found from the output gathered in step 3.
When you run this command it will scan every log file found in the source directory. If any gaps or errors are identified, you cannot continue with these steps. If the command completes and errors on the last log file in the series this is expected, as the EXX.log is currently open for writing and cannot be scanned. The following is sample output that you should receive for a mailbox database that is online.
[PS] C:\DAG\DAG-DB3>eseutil /ml E04
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 14.01 Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Verifying log files... Base name: E04
Log file: C:\DAG\DAG-DB3\E04000000BF.log - OK Log file: C:\DAG\DAG-DB3\E04000000C0.log - OK Log file: C:\DAG\DAG-DB3\E04000000C1.log - OK Log file: C:\DAG\DAG-DB3\E04000000C2.log - OK Log file: C:\DAG\DAG-DB3\E04000000C3.log - OK Log file: C:\DAG\DAG-DB3\E04000000C4.log - OK Log file: C:\DAG\DAG-DB3\E04000000C5.log - OK Log file: C:\DAG\DAG-DB3\E04000000C6.log - OK Log file: C:\DAG\DAG-DB3\E04000000C7.log - OK Log file: C:\DAG\DAG-DB3\E04000000C8.log - OK Log file: C:\DAG\DAG-DB3\E04000000C9.log - OK Log file: C:\DAG\DAG-DB3\E04000000CA.log - OK Log file: C:\DAG\DAG-DB3\E04000000CB.log - OK Log file: C:\DAG\DAG-DB3\E04000000CC.log - OK Log file: C:\DAG\DAG-DB3\E04000000CD.log - OK Log file: C:\DAG\DAG-DB3\E04000000CE.log - OK Log file: C:\DAG\DAG-DB3\E04000000CF.log - OK Log file: C:\DAG\DAG-DB3\E04.log ERROR: Cannot open log file (C:\DAG\DAG-DB3\E04.log). Error -1032.
Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use) after 1.469 seconds.
The fifth step is to perform a VSS backup of the database. Please consult with your backup vendor to ensure that a successful FULL backup is performed. Please also make sure that a consistency check of the backup is performed.
The sixth step is to restore the VSS backup. When you perform the restore, you should select the option to restore to file system (it may be necessary to consult your backup vendor). This may require that you restore to the file system of the Exchange Server, so it may be necessary to ensure that sufficient free space exists on a volume on the Exchange Server where the restore will be performed.
If multiple databases are being restored I recommend that databases be restored individually.
Please ensure that no recovery operations are performed on the database (options like roll forward recovery / replay logs / etc should be avoided).
At this point we now have the EDB file on the file system and we will use it for the seeding operation.
In our example we’ll restore to c:\RESTORE.
The seventh step is to ensure that the target paths are ready to have the database moved in place. The paths referenced in this step can be obtained from step 3.
In this example we will ensure the path c:\DAG\DAG-DB3 is empty on the target server.
If the paths already existed they are now ready to have the restored database moved to them.
If the paths do not exist they should be manually created. If you are using nested folders you need to create the entire directory structure.
The eighth step is to move the restored database to the target directory. This can be accomplished in a few different ways, but I will make a recommendation below.
From the source server map to the drive$ share of the target. For example, I would map the Y drive to \\MBX-3\C$\DAG\DAG-DB3 using our example.
net use y: <path>
C:\>net use y: \\MBX-3\c$\DAG\DAG-DB3 The command completed successfully.
On the source server open a command prompt to the directory where the data was restored. In this example c:\Restore
Use eseutil to copy the database from the source directory to the target directory. A sample command:
eseutil /y <Source.EDB> /d <Target.EDB> Here is the output expected from the command using our example:
C:\Restore>eseutil /y DAG-DB3.edb /d y:\DAG-DB3.edb
Initiating COPY FILE mode... Source File: DAG-DB3.edb Destination File: y:\DAG-DB3.edb
Copy Progress (% complete)
0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ...................................................
Total bytes read = 0x18810000 (411107328) (392 MB) Total bytes written = 0x18810000 (411107328) (392 MB)
Operation completed successfully in 29.16 seconds.
At this point the copy has been seeded on the target server.
In place of the network administrators may consider a portable storage device for the transportation of the database to the target server. (Note: In this case eseutil /y would be used to copy the data to the portable storage and from the portable storage to the target server).
The ninth step is to verify the health of the copied database. We need to ensure that the database was not corrupted as a part of the copy process.
On the target server open a command prompt and navigate to the location of the database file. In our example this is c:\DAG\DAG-DB3.
Use the eseutil /k to perform a checksum of the database:
eseutil /k <Database.edb>
The following output will be observed when the command completes:
C:\DAG\DAG-DB3>eseutil /k DAG-DB3.edb
Initiating CHECKSUM mode... Database: DAG-DB3.edb Temp. Database: TEMPCHKSUM4804.EDB
Checksum Status (% complete)
12546 pages seen 0 bad checksums 0 correctable checksums 5571 uninitialized pages 0 wrong page numbers 0x62c97 highest dbtime (pgno 0x7c)
6273 reads performed 392 MB read 6 seconds taken 65 MB/second 5895209 milliseconds used 939 milliseconds per read 3437 milliseconds for the slowest read 93 milliseconds for the fastest read
Operation completed successfully in 6.672 seconds.
We are interested in ensuring that there are 0 bad checksums.
The last step in the process is to resume the mailbox database copy.
The following command can be used to resume the mailbox database copy:
Resume-MailboDatabaseCopy –identity <Database\Server>
[PS] C:\>Resume-MailboxDatabaseCopy DAG-DB3\MBX-3
Post a resume the following events can be noted in the application log on the target server.
Log Name: Application Source: MSExchange Search Indexer Date: 12/6/2010 5:30:16 AM Event ID: 109 Task Category: General Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225). Reason for full crawl: Catalog doesn't exist.
Log Name: Application Source: MSExchangeRepl Date: 12/6/2010 5:30:17 AM Event ID: 2114 Task Category: Service Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: The replication instance for database DAG-DB3 has started copying log files. The first log file copied was generation 191.
Log Name: Application Source: MSExchange Search Indexer Date: 12/6/2010 5:30:18 AM Event ID: 108 Task Category: General Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: Exchange Search Indexer has enabled indexing for the Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).
Log Name: Application Source: MSExchangeIS Mailbox Store Date: 12/6/2010 5:30:27 AM Event ID: 1000 Task Category: General Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: Attempting to start the Information Store "DAG-DB3".
Log Name: Application Source: MSExchangeRepl Date: 12/6/2010 5:30:37 AM Event ID: 2157 Task Category: Service Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: The replication instance for database DAG-DB3 has copied and replayed multiple logs.
Log Name: Application Source: MSExchange Search Indexer Date: 12/6/2010 5:32:56 AM Event ID: 110 Task Category: General Level: Information Keywords: Classic User: N/A Computer: MBX-3.exchange.msft Description: Exchange Search Indexer completed a full crawl (indexing) of Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).
The administrator can monitor the post seeding activity using the get-mailboxdatabasecopystatus <Database\Server> command.
In this example we will use get-mailboxdatabasecopystatus DAG-DB3\MBX-3.
Initially the administrator will note a STATUS of RESYNCHRONIZING and a CONTENTINDEXSTATE of CRAWLING.
[PS] C:\>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State ---- ------ --------- ----------- -------------------- ------------ DAG-DB3\MBX-3 Resynchronizing 20 0 12/5/2010 8:58:13 PM Crawling
After all delta log files have copied to the target and replay has begun, the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of CRAWLING.
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State ---- ------ --------- ----------- -------------------- ------------ DAG-DB3\MBX-3 Healthy 0 50 12/6/2010 5:27:17 AM Crawling
Due to the online offline database seed the content index files were not copied from the target server and were not restored from backup. Therefore the search service on the target will initialize a new content index for this store and being indexing from the source. If the database is large content indexing could be in a CRAWLING state for some time.
After the content index has been successfully built the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of HEALTHY.
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex Length Length State ---- ------ --------- ----------- -------------------- ------------ DAG-DB3\MBX-3 Healthy 0 0 12/6/2010 5:27:17 AM Healthy
At this point the database seeding operations are now 100% complete and the database and fully participate in DAG functions.
Excellent article to undersantd offline database seeding when we enable DAG replication over WAN. We used to follow the offline Exchange Database copying of SCR source server to target server and enable the SCR replication in Exchange 2007 SCR. Should we use same method in exchange 2010 to copy the offline database to target DAG server and run add-mailboxdatabasecopy cmd let to enable replication ??.
I want to use this procedure to offline seed a failed database replication in a remote DAG member. Step 7 says that the target path has to be empty but the old copy of the database is there. How do I remove it?
You can just delete the file if it is no longer necessary for replication etc.
Hi Tim - what if the second step does not show suspended, but failed and suspended instead? Will this still work?
What a great question. Surprisingly this is the second time I've had this question this week - go figure!.
Suspended, failed and suspended, or failed would all be valid status here. The real point of having one of these status is to ensure that no log truncation can occur either through circular logging <or> through the backup process. Any of these status will not allow for log truncation.