Blog of the EMEA SBS Team

Troubleshooting Microsoft Exchange Information Store Service and Disaster Recovery.

[Today’s post comes to us courtesy Karan Rustagi and Girish Rajan.]

We have seen many a times that MSExchangeIS would not start or hang while starting. There could be several reasons for this to happen but in this post we will discuss one of the possibilities which we see very often.

Like in most issues, the the first step toward troubleshooting this issue is to check the Event Viewer and look for any related errors. If you get the following error while starting the service then proceed with reading the post.

Event Type: Error
Event Source: MSExchangeSA
Event Category: (14)
Event ID: 9175
Description: The MAPI call 'OpenMsgStore' failed with the following error: The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server is down for maintenance. The MAPI provider failed. Microsoft Exchange Server Information Store ID no: 8004011d-0526-00000000

This could happen if any of the Databases (Mailbox Store and Public Folder Store) are inconsistent. There are several ways to recover from this problem. Few of them are:

(1) Restore the Databases from backup
(2) Replay the logs files to recreate the Databases. Also known as Soft Recovery.
(3) Repair the Databases using Eseutil command.
Also known as Hard Recovery. It is known as Hard Recovery because following this process may cause some data loss.

Always Consider options (1) & (2) before going for Option 3. Also remember to take a backup of your databases before you go for options (3).

STEPS:

(For the purpose of simplicity we have kept default naming convention for logs, databases and their path.)

1. Let’s check the location of Exchange databases, log files and chk file.

(a) Look for the BIN directory where your Exchange is installed. Usually this would be located under “C:\Program files\Exchsrvr”. We use Eseutil to repair the Databases which is located under Bin directory.

(b) Now look for location where you have stored your Exchange Databases. By Default they would be located under “C:\Program Files\Exchsrvr\MDBDATA”. You can confirm it by checking the location manually.

(i) Start Exchange System Manager.
(ii) Open the administrative group that contains the database that you want to check.
(iii) Under Storage Group, right-click the mailbox store or the public folder store that you want to check, and then click Properties.
(iv) Click the Database tab. Now you should see the location EDB and STM files which are known as Exchange Databases.

(c) Now check the location for Exchange LOG files and CHK file. By default they are located under MDBDATA folder. If you want to check them manually:

(i) Start Exchange System Manager.
(ii) Expand the server container, and then locate the storage group that contains the files. By Default is First Storage Group in SBS
(iii) Right-click the First Storage Group.
(iv) Click Properties from the context menu.
(v) Click the General tab.
(vi) Note down the location of Transaction log location and System Path Location.

Note: System Path Location specifies the path of CHK file

2. Now we will check the state of databases using Eseutil command. Open Command Prompt and change you working directory to location where exchange databases are stored.

HINT: Read Step 1. (b) to find the location

3. Type the following command:

Set path=<LOCATION OF BIN DIRECTORY>

HINT: Read Step 1. (a) to find the location

Confirm that path has been set correctly by typing ESEUTIL at the prompt. It should give you similar output.

Press ENTER now and it should bring you back to command prompt.

NOTE: In this example we have used the SET command to temporarily set the path of command prompt window to run Eseutil command. There are several other ways to do it like running the Eseutil command from BIN directory and then specify the complete path of databases OR change the Environment Variables of the system to include BIN directory in Path.

4. Now type the following Command:

Eseutil /mh priv1.edb

This command will give different stats about the databases but what we are looking at is a check called “State”:

Dirty Shutdown means that databases are in inconsistent state and cannot be mounted. They can become inconsistent because of several reasons like abrupt shutdown of store.exe because of power failure etc.

If in your case state reports Dirty Shutdown; please continue reading the post.

5. We will repair the databases now using the “/p” switch. Please take an offline backup before following this step.

Type the following Command:

Eseutil /p priv1.edb

Now click on OK if you have taken a backup of databases and ready to repair the databases.

NOTE: When you repair a database (Eseutil /p), the amount of free disk space required depends on the number of corrupt pages in the database. Normally, 25 percent of the file being repaired is a conservative estimate of the amount of free disk space required.

6. Once the command completes you will see following output:

NOTE: The total time taken by this process depends upon the size of exchange databases and availability of system resources. It may take few hours to a day to complete. Incase if you want to know if it is working or not then open Task Manager and check for the CPU and Memory Usage of a process called ESEUTIL; it should change constantly.

7. Once the databases are repaired; Delete the Log Files, CHK file and Temp.edb(If Exist)

HINT: Read Step 1. (c) to find the location

Do a double check on if the log and chk files have been deleted as databases will not mount if it finds the old files.

NOTE: Every Database has a signature which should match the signature on Log files. Once we run Eseutil to repair the databases it changes the database signature and databases will not mount because log files and database have different signature now.

8. Do an offline defrag using “/d” Switch.

Type the following command:

Eseutil /d priv1.edb

NOTE: The amount of free disk space needed to defragment a database (Eseutil /d) is 110 percent of the size of the file being defraged.

9. At this stage we are done with Mailbox Stores (priv1.edb + priv1.stm); let’s follow Step 4-8 for pub1.edb which is Public Folder Database. Incase if running “Eseutil /mh” against pub1.edb says “Clean Shutdown” against state then you may skip repairing public folder databases.

10. Now open Services Console and Start the Microsoft Exchange Information Store Service.

11. Now open Exchange System Manager and check if the databases are mounted or not. Incase if they are not mounted then right click and mount the stores:

12. Next step is to run ISINTEG against the databases. We need to manually dismount the stores before running ISINTEG and let the Microsoft Exchange Information Store Service running..

Type the following command:

Isinteg –s <ServerName> -fix –test alltests

Press 1

Press Y

Now it should start running the command and will end with the following output:

Run ISINTEG command once again and this time choose Public Folder Store.

13. Now go back to Exchange System Manager and Mount the databases. At this stage I would once again recommend you to take the backup.

The above mentioned steps could also be followed if you are not able to mount the stores with following errors:

EVENT VIEWER:

Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Event ID: 9518
User: N/A
Description:
Error Database is in inconsistent state starting Storage Group /DC=XXXXXX/DC=XXXXXX/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=XXXXX/CN=Administrative Groups/CN=first administrative group/CN=Servers/CN=XXXXXX/CN=InformationStore/CN=First Storage Group on the Microsoft Exchange Information Store. MDB failed to start.

Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Event ID: 9519
User: N/A
Computer:
Description:
Error Database is in inconsistent state starting database "First Storage Group\Mailbox Store (XXXXXX)" on the Microsoft Exchange Information Store.
EXCHANGE SYSTEM MANAGER:
The database files in this store are inconsistent. ID no: c1041739 Exchange System Manager

NOTE: Please make sure that Microsoft Exchange Information Store Service is stopped before running Eseutil command against databases (Step 4) and start it once the offline defrag completes (Step 8).

What do the above steps exactly do?

The Eseutil [/p] repair mode corrects database problems at the page and Extensible Storage Engine (ESE) table levels but not at the application level. After repairing a database using Eseutil, ISInteg should be run to repair the database at an application level. To understand what database page level, ESE table levels, and application levels mean, see Database Recovery Strategies on Technet.
For more information about syntax and instructions for using Eseutil /P, see How to Run Eseutil /P (Repair) in Different Scenarios.

During repair, it may be necessary to discard rows from tables or even entire tables. After completing the ESE-level repairs, it is necessary to perform an application-level repair to correct problems that may now exist at the application level because of missing data. The Information Store Integrity (ISInteg) utility can be used to perform this Exchange application-level analysis and repair. The following example explains how Eseutil repair works.

For example, a table in the database stores messages for all mailboxes. A separate table is used for each user's Inbox folder. Suppose that a message is lost when using Eseutil to repair the message table. Eseutil will not correlate the message with the reference to it in each Inbox folder, because Eseutil does not understand the cross-table schema of the application. ISInteg is needed to compare the repaired message table with each Inbox folder, and to remove a lost message from the Inbox.

In short, Eseutil looks at each Exchange database page and table and ensures consistency and integrity within each table. ISInteg, recommended to be run after Eseutil, repairs a database at the application level and ensures the integrity of the relationships between tables.

Hence, repairing databases involves the following three stages, in this order:

1. Eseutil is run in /P mode to perform a database page-level and table-level repair.
2. Eseutil is run in /D mode to fully rebuild indexes and defragment the database.
3. ISInteg is then run to repair the database at the application level.

Note: A successful repair does not necessarily mean that a database will always be useable. The loss of system metadata may leave a database unmountable or empty. When a database is not repairable, you can restore data from backup or create a new database.