Enterprise Communications Support (formerly known as Exchange Messaging Support) has recently seen an increase in support incidents around the -519 Jet_errLogSequenceEnd issue. In this blog I will explain this issue in detail, why it may occur, how to recognize it, and what to do next.
Exchange has had a robust data engine that provides an ACID database from the early days; the transaction-based JET database engine ensures all database changes are Atomic, Consistent, Isolated and Durable. We accomplish this with the use of transaction log files. In Exchange Server 2000 and 2003, we introduced the concept of Storage Groups. Storage Groups can contain up to 5 databases, but share one set of common Transaction Logs. To differentiate between Storage Groups, all Transaction Log file names contain a prefix, followed by a 5 digit hexadecimal number, like so:
Where nn is the number of the Storage Group, typically 01, 02, and so on. As transactions (changes such as new e-mail, mail being read, tasks created, views built, etc) occur Exchange records them in sequentially numbered logs. This allows us to recover from certain database problems by knowing exactly which log comes next in a replay sequence. In Exchange 2000 and 2003, the transaction log file name length is 8 characters long. Since 3 of the characters form the file name prefix, 5 remain. Thus, the largest possible hexadecimal number we can represent is
The actual largest number ESE in 2000/2003 will allow is FFFF0. That number in decimal is 1,048,560, representing the maximum number of log files we can write sequentially before running out.
With over one million 5 MB log files available, in some cases, it can take as long as a few years to hit this condition. When the transaction log sequence is exhausted, the Microsoft Jet database engine returns error -519, JET_errLogSequenceEnd to the Information Store. Depending on which version of Exchange 2000 or 2003 you are on, this error will result in slightly different symptoms. These symptoms are described in the following Knowledge Base articles:
830408 Store databases are dismounted without warning or users cannot log on to their mailboxes in Exchange Server 2003 or in Exchange 2000 Server
896001 An event is not logged in the Application log before the last available transaction log in the sequence is used in Exchange 2000 Server
In the latest revisions for Exchange 2000 and 2003, we added some fixes to warn you of this impending problem and prevent it from occurring. Store will warn you when you are nearing the end of the log sequence with the following event:
Event Type: Warning Event Source: ESE Event Category: Logging/Recovery Event ID: 514 Description: Information Store (2748) SG2: Log sequence numbers for this instance have almost been completely consumed. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.
If you have databases in Storage Groups that have been online contiguously for several years, you should be monitoring your transaction log sequence and the Application Log for this event. When you see this, it's time to reset the transaction log sequence.
Notice that if you miss the ESE 514 warning your databases will dismount and generate the following events:
Event ID: 1159Event Type: ErrorEvent Source: MSExchangeISEvent Category: GeneralDescription: Database error 0xfffffdf9 occurred in function JTAB_BASE::EcEscrowUpdate while accessing the database "First Storage Group\Mailbox Store (SERVER)".
Event ID: 9518Event Type: ErrorEvent Source: MSExchangeISEvent Category: GeneralDescription: Error 0xfffffddc starting Storage Group Path_of_Storage_Group on the Microsoft Exchange Information Store. Storage Group - Initialization of Jet failed.
OK, great, what the heck does 0xfffffddc mean? Glad you asked! You can look up Exchange error codes here: Microsoft Exchange Server Error Code Look-up
Note that error translates to Jet_errLogSequenceEndDatabasesConsistent:
Err 0xfffffddc -# for hex 0xfffffddc / decimal -548 JET_errLogSequenceEndDatabasesConsistent esent98.h# /* databases have been recovered, but all possible log# generations in the current sequence are used; delete all# log files and the checkpoint file and backup the databases# before continuing */
If you attempt to mount databases in this condition you will discover another cause for this common error in the Exchange System Manager:
An internal processing error has occurred. Try restarting the Exchange System Manager or the Microsoft Exchange Information Store service, or both.ID no: c1041724Exchange System Manager
Now that I've explained the issue and how it will be reported in the event logs, here's how to correct this problem:
KB 830408 has a workaround section that describes how to reset transaction log sequence manually. However, we have included this functionality in the Microsoft Exchange Troubleshooting Assistant "Reset log generation number task". We talk about that here:
MSExchangeIS 9518 (0xfffffddc): Exceeded the Maximum Number of Transaction Logs Available for this Storage Group
Microsoft recommends using the Troubleshooting Assistant (ExTRA) because it automates the process of verifying the health of your databases and resetting the transaction log sequence. Download the Microsoft Exchange Troubleshooting Assistant v1.0 and follow these instructions to reset transaction log sequence: http://technet.microsoft.com/en-us/library/aa998611(EXCHG.80).aspx
We also recommend monitoring all your Exchange 2000 and 2003 Storage Groups' transaction log sequences to avoid a potential temporary outage of service. Monitor your application logs for the ESE 514 events.
Now, while it is theoretically possible to hit this condition in Exchange 2007, it will probably take a really long time. For Exchange Server 2007 we decreased the log file size to 1 MB but increased the transaction log file name length to 11 digits, meaning we can go up to
Due to the way ESE math works our upward limit is actually 7fffffec log files, but that is still a HUGE number. After figuring in the change in size to 1 MB, that's still about 409 times the number of logs we could generate in 2000/2003. Read more about this improvement in the following TechNet Magazine article located here.
Corbin MeekEnterprise Communications Support