Dude where's my PFE? : Exchange 2003

Defeated by Unexpected Transaction Log File growth

jeffstok — Sun, 25 Jan 2009 20:23:00 GMT

Applies to Exchange 2003, concepts apply to 2007 as well.

I've bumped into a few cases recently where the customer had unexpected transaction log file growth that caused the server to dismount a storage group due to lack of disk space. In this post I'll attempted to explain why this occurs, how to troubleshoot it, etc.

The short of it is transaction log file growth usually occurs because of a repeating transaction. It can be a looping message, a mis-behaving client, or a corrupt message. Looping messages I've seen done by users setting up special things on their Outlook clients. Consider the following example:

A user leaves for the weekend. They are expecting an important email, so they put in a forward rule to forward all email to their mobile phone's email address. They either 1)mis-type that address, or 2)their phone's email box doesn't accept messages above a certain size. In the event of 1), every message sent to the user is going to hit the mail servers of the phone provider and bounce with an invalid address. This NDR will come back and hit the mailbox of the user, where the forward rule will forward the NDR to the phone, which will bounce and come back to the inbox, where it will forward the NDR to the phone...... In the event of 2), any message above the size limit will trigger the loop above (unless the ISP's mail server knows not to append the offending email as an attachment to the NDR).

This is a real world example I've personally run into. Users can and will do all kinds of bizarre things that under the light of day seem obtuse, but in the heat of the moment make sense.

So how do you track this down?

The normal troubleshooting path I take for this type of problem is:

1. Run Exmon. Tell me if a single user is taking something silly like 50% of the servers resources. If you're spooling out transaction logs like it's nobody's business and Exmon shows a user at 50%+ and they are in the same Storage Group as the spooling transaction logs, then chances are you've found your man. If Exmon doesn't point anything out of the ordinary, then proceed to step 2:

2. Go to your Exchange System Manager, drill down to the Storage Group that you're seeing the transaction log growth on. Expand each database and visit the logins area. Add columns for MSG Ops, Folder Ops, Total Ops, and sort by high/low and see if you have one user towering above the rest. Do this for each database. If you've got a single user standing out, again, this is very likely your culprit. Log into their mailbox, see if there is something stuck in the Outbox, or check their active client for any client-side rule that may be at fault. Worse comes to worse, disable the user's mail.

3. User Scott Oseychik's guide on Transaction Log analysis to figure out what the offending message might be:

http://blogs.msdn.com/scottos/archive/2007/07/12/rough-and-tough-guide-to-identifying-patterns-in-ese-transaction-log-files.aspx

This is an excellent guide and needs no further clarification.

4. If this doesn't work out for you at this point, call into support, it could be a problem with a mobile device syncing or an OWA session trying to process a corrupt message (I've seen both scenarios). Only a series of store dumps collected with adplus will tell us that.

I hope this helps in your troubleshooting efforts.

How to get rid of 9646 events

jeffstok — Tue, 25 Nov 2008 23:16:00 GMT

Applies to Exchange 2003

Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Event ID: 9646
Description:
Mapi session "/o=<org>/ou=First Administrative
Group/cn=Recipients/cn=<userName>" exceeded the maximum of 32 objects of type "session".

Seen these in your environment? Sometimes they are caused by desktop search engines opening too many MAPI sessions, other times they are because your Exchange server is keeping open connections from the client that the client for whatever reason thinks are closed.

For example, say you have users connecting to Exchange over a poor network connection or VPN. When Outlook connects, it establishes MAPI sessions. If the users drops the VPN connection without closing Outlook first, those connections are going to stay open on the Exchange server for 2 hours. If the user connects again, more connections get added to the Exchange server for that user. See where we're going with this? If you max out, new connections will fail, resulting in an unhappy end user.

So how do you fix this?

Well, one of three ways.

You can try to prevent your clients from doing so many connections by educating your users, making changes in Outlook 2007, correctly configuring Network Accelerators to not keep connections open, etc.

You can tell Windows/Exchange that 2 hours is far too long to keep a session open without activity. To do this you follow the instructions in this document, specifically the TCP KeepAliveTime, set that to 5 minutes.

KB324270 How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;324270

Or finally, and the last resort, you can add additional allowable sessions by following KB842022. Note that this is a last resort, as in many cases you are merely delaying your pain for later. Note the warning: "If you do this, try to determine the minimum value that you can use so that the client program can run without problems. If you raise the limit too high, the client program might affect the performance of the Exchange Server computer."

Event ID 9646 is logged in the application event log of your Exchange Server 2003 computer when a client opens many MAPI sessions.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;842022.

Repeating 623 version store error.

jeffstok — Sun, 26 Oct 2008 22:38:00 GMT

Applies to Exchange 2003

I had a case a couple weeks ago I thought I'd write about. What was happening is the Version Store would run out of memory and a 623 error would throw. Version Store buckets allocated would climb from 4 to over 2000 in less than 5 minutes. The store would then rollback its transactions for a bit, recover, run for 10-15 minutes and repeat the whole cycle over.

This is atypical 623 behavior to say the least.

What we ended up doing to fix it was capture an adplus dump, 3 actually, triggered at Version Store buckets allocated crossing 1600. We captured 3 dumps at 1 minute intervals.

The 1st dump caught the problem transaction, the last 2 were both capturing rollbacks, so this was a quick ramp up.

Turns out the problem was being caused by a bad meeting request being processed over and over again. We tried all kinds of ways to delete the message, all of which caused Version Store buckets allocated to climb. A MFCMapi hard delete ended up doing the trick.

How to fix smashed schema in Exchange 2003

jeffstok — Thu, 09 Oct 2008 00:53:00 GMT

Dan and I and some other engineers wrote up a blog post you can find here on how to recover from a smashes schema scenario on your Exchange Servers.

It's pretty succinct so I don't have anything to add to it, it's an interesting read though.