Why Exchange Databases Might Remain Dirty After ESEUTIL /R Recovery

Why Exchange Databases Might Remain Dirty After ESEUTIL /R Recovery

  • Comments 33
  • Likes

You’re trying to recover your Exchange database from the “dirty shutdown” state to “clean shutdown.”  You use ESEUTIL /R and after several ill-fated attempts the databases remain dirty.  What’s happening?  Danijel Klaric. a Microsoft Premier Field Engineer based in Germany, sheds some light on why the Exchange 2007/2010 databases might remain dirty after a seemingly “successful” ESEUTIL /R recovery, and provides pointers on how to solve this.  Enjoy his article!


Normally I wouldn't think that this following topic would ever be a problem, but as I faced the issue two times in the last three weeks, I thought it would be good to shine some light on the subject.

Both customers are using Exchange 2010 with Native Data Protection, and have enabled a seven day lag on one of their database copies to enable their admins to recover items past the “single item recovery” period, or to recover deleted folder structures. Remember that single item recovery doesn't preserve the folder structure as mails moved to the dumpster are stored in a flat hierarchy.

The Symptom

They thought they were following appropriate Microsoft guidelines to use logs needed to recover the database from the “dirty shutdown” state to “clean shutdown”, so that the database would be usable as a recovery database for extracting the needed content.

1) Collect the database and the required logs

First grab a copy of the edb file and needed logs (i.e. logs needed from database perspective and logs they would like to roll forward to). This can be accomplished by suspending the database copy, and then copying the needed files to a separate location or create a snapshot of the volume which can be reverted to at a later time.

2) Check database state

Run the following: eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb"

eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb"

Check the database header for log files generation needed for the recovery, in this case: Log Required 124-124 (0x7c-0x7c) which means file E000000007C . These log files are needed at a minimum to recover the database to a clean shutdown without data loss.

Side note: If you try to recover the database with only “Log Required” logs and NOT “Log Commited” logs it will throw the following error:

Side note: If you try to recover the database with only “Log Required logs” and NOT “Log Commited” logs it will throw the error:

You can continue with the recovery using /a. And the last committed transaction will be removed from the database and the database will have a clean and consistent state. Otherwise just add the log files mentioned in the “log committed” field to your log directory and this error will disappear.

3) Check consistency of needed logs

Before starting to apply the logs you would like to, it´s important to check if all logs are consistent and available. If you need to handle a large number of logs and have used the Windows Explorer to copy the files, occasionally I’ve seen that some files are missing when sorting them in the Windows Explorer view.

To ensure that you will not fail afterwards when performing the recovery, take a moment and check these. Use eseutil /ml together with your log directory path and log prefix, in my example E00.

Log file check within directory: eseutil /ml c:\DBRecovery\E00

Log file check within directory: eseutil /ml c:\DBRecovery\E00

4) Recover database state to “clean shutdown”

After they finished both checks successfully, they started the recovery using:

eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb"
eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb"

It failed with an Error “-1216”. They did some Internet research and found that this could be fixed by using the “/i"

So their next try was:

eseutil /R E00 /l c:\DBRecovery /d "c:\DBRecovery\Mailbox Database 0436312751.edb" /i

eseutil /R E00 /l c:\DBRecovery /d "c:\DBRecovery\Mailbox Database 0436312751.edb" /i

And YES it results in “Operation completed successfully in 0.140 seconds.” Then they checked the Application event log and YES, there is no Error.

Application event log

Their last check before mounting the database was eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb."  After all that work - STILL a “dirty shutdown” Sad smile.

Note:

The get quick access to their data, they executed the eseutil command using /P for a repair to get the database into a clean shutdown. Microsoft strongly discourages using eseutil /P because it may lead to data loss. The /P should be a measure of extreme desperation when there’s no way at all to recover your database via /R.

Troubleshooting Steps – How did we solve it?

They showed me their eseutil command:

eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb” /i

I saw three things I wouldn't expect to see:

  1. “/i” is a parameter for skipping missing log stream attached databases. In older versions of Exchange one log stream was used per storage group. As a storage group could have contained multiple databases, the “/i” option was provided when only one out of multiple databases has to be recovered.   As Exchange 2010 doesn’t use storage groups any more, this parameter shouldn’t be used any more.
  2. “/d” parameter with the fully qualified path to the EDB file.  You HAVE to use the directory path and NOT the full path to the edb file.
  3. They also didn’t specify a system path, so the actual path of the cmd prompt is used. This directory is used to check for, or to create, an E0x.chk file. I always recommend specifying the same path as the edb and log directory to keep the files together, and ensure that the files you expect to be used are the ones which are always used.

First, I asked them to remove the “/i” parameter and to try it again, they told me that this resulted in the previously mentioned error:  “-1216 (JET_errAttachedDatabaseMismatch)”.

Checking the EventLog after removing the “/i” parameter, it stated that the database file has not been found at path “C:\DBRecovery\Mailbox Database 0436312751.edb\Mailbox Database 0436312751.edb" which makes sense.

Checking the EventLog

After removing the file name and ending “\”, and optionally adding the system directory parameter to the command, we ran eseutil /R E00 /l “c:\DBRecovery” /s “c:\DBRecovery” /d “c:\DBRecovery”.  Running this modified command, everything worked as expected and the database ended up in a “clean shutdown” without any error message.

eseutil /R E00 /l “c:\DBRecovery” /s “c:\DBRecovery” /d “c:\DBRecovery”.

Note: If the database still stays in “dirty shutdown”, check the specified system directory for an e0x.chk file from your previous attempts. This checkpoint file can be safely removed for this recovery process if only logs that are needed for your recovery exist in the log directory. Following that, the command should complete successfully.

So why was this happening?

I was wondering why two different customers would make the “same” mistake using eseutil /r, so I started some Internet research and found sites providing examples using “eseutil /r /d” with the full path to the edb. This will simply not work.  Most of these mention that sometimes this will not work – and recommend using /p to perform a repair of the database. As mentioned, this shouldn’t be the recommended way as long as the edb file and the required logs exist and are available to be used.

Another possible misunderstanding would be that eseutil also incorporates another “MODES OF OPERATION” which use “/d” as a parameter –> Database defrag. So if you use “eseutil /d” to defrag a database you have to specify a complete file path.

Hope this helps. And after reading through this, just sit back and use the ESEUtil parameter which you should try out with your Exchange 2007/2010 server (without any risk):  “eseutil /ESE.”  And give them their credits providing such a solid database Smile.

Thanks goes to Alexandre Costa and James Cameron reviewing this content.


Posted by Frank Battiston, MSPFE Editor

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Great articles! Very informative !!! Thanks for clarifying, that was so ease to understand.

  • Good one. Thanks for sharing this info.

  • Cool article Danijel. Also, I totally didn't know about the /ESE trick. Pretty cool & nerdy at the same time :D

  • Hi Daniel, very good write-up on this topic. I have encountered this scenarios several times.

    Thanks!

  • Wow!!!!  It is a very detailed description about how Exchange Server recovery can be possible, but in case if you need effective third party tool then you can go for Stellar Phoenix Exchange Server Recovery. I used it and it worked like a second life for my database.

  • awesome :-)...

    Nice one to read and understand...

  • Great article! Easy to have customers follow the step by step instructions.

  • Great Information for recovering Exchange database via ESEUTIL commands. However in case where Exchange Mailbox database is highly corrupted & Exchange server is even crashed then you can call for trusted Exchange Recovery solution ( www.pcvita.com/exchange-database-recovery.html ). ESEUTL will run only when Exchange server is live otherwise no way-out is left.

  • Lastly, I had a unusable .ost file which was not working due to sudden server crash. I needed to recover that .ost file because some crucial information was in that file. I had searched lots of tools over internet but enable to find best one.<br />

    One of the user suggest me to use SysTools OST to PST converter software and i tried that many times and happily recovered all items of .OST file into PST file format.<br />

    People can search this tool directly here: <a href="www.microsoftosttopstconversion.scanostfile.com/.../www.microsoftosttopstconversion.scanostfile.com </a>

  • It's really very informative article about Exchange server Recovery,If you want to know more about other option for Exchange Server Recovery, then I would like to suggest you for an professional Exchange server recovery tool like Kernel for Exchange Server recovery Tool. By using this effective software one can very easily repair EDB files of Exchange server without any hassle.

  • If we receive “-1216 (JET_errAttachedDatabaseMismatch)” best way to force them to replay the logs is mounting the database on Exchange server with required log files plus 15 - 20 log files in sequence. Just make sure you delete the Chk file before mounting it.

    Note : this is applicable only to Exchange 2007 and above. If we Still unable to mount its possibly page corruption which don't have other option than hard repair "eseutil /p"

  • Thanks .. it was really useful ..

  • Work with outlook conversion with the aid of ost to pst converter

    OST to PST Converter Toolbox works with ost and pst data

    Visit next resource www.osttopstconverter.convertertoolbox.com

  • Very Informative article on Exchange dirty shutdown which is a very common issue with Exchange server. Eseutil utility has innumerous commands to serve distinct need of the users. But if any case these inbuilt commands fails to bring in results then a third party tool always comes in use. Visit here: http://www.exchangerecover.com/recovery-tool-for-exchange.html

  • Thank you, thank you, thank you