Mike Lagase

Saving the Exchange world one day at a time.....

Excessive paging on Exchange 2007 servers when working sets are trimmed

Excessive paging on Exchange 2007 servers when working sets are trimmed

  • Comments 17
  • Likes

For a list of current recommendations to help alleviate these issue, click here

Recently, there has been a rash of performance issues on Exchange 2007 Mailbox servers where they become unresponsive due to excessive paging. Previously this was tracked down to .NET garbage collection not occurring properly which caused managed services to consume excessive amounts of memory. Applying http://support.microsoft.com/kb/942027 was the fix for this. If you have installed SP1 for Exchange 2007, we currently recommend to apply .NET 2.0 SP1 to the server which contains this fix. This is not a hard block, but a warning during the pre-req checks during setup to apply this.

Even if you have this hotfix installed, excessive paging was still occurring. After looking in to this further, we noticed that the working set for store.exe was getting trimmed causing us to page that trimmed memory to the paging file. Here is what you will see if you experience this in performance monitor. The red line is Memory\Pages/sec when the trim operations occur.

image

Here is a screenshot of all processes working sets.

image

If you look at all of the processes, they are all getting trimmed at the same time. With Exchange 2007, we have lifted the ceiling on store cache and can use upwards of 80-90% of the memory on a server. So if you have 32GB of RAM on the server, the store process itself will take longer to warm-up the cache than it will with only 16GB of RAM. During this time, clients may experience slower than normal responses while the cache is being populated. Looking at the above data, store.exe is using over 20GB of RAM and when we reach a certain peak, the working set gets trimmed causing this cache to also get trimmed. This is where performance dies when this working set is being paged to disk. Once the paging has occurred, we then have to reload that data back in to the working set as that is needed to perform a current operation. If the roller coaster ride begins, the server is going to fall over as you can see at midnight that even performance monitor couldn't even connect to the server. Once paging storm died down 30 minutes later, we were back working again, albeit slow.

Memory Planning considerations can be found in the help file at http://technet.microsoft.com/en-us/library/bb738124.aspx 

So you may ask, what is causing this? Well, in Windows 2003, if a driver makes a call to allocate a considerable amount of memory, specifically MMAllocateContiguousMemory, the working sets of processes need to get trimmed to give this call the memory that it needs. This can actually be any driver on the server that has the need to request these memory allocations. Detecting what driver it is much harder than you would think as you have to perform some type of kernel debugging to get to the bottom of it.

Currently in Windows 2003, we will trim about 1/4 of the working sets for each process to satisfy these requests and as you can see, it appears to be much more than 1/4 of the working set being trimmed, more like 3/4. Luckily, the below hotfix helps with this situation greatly when these trimming operations occurs as with the hotfix applied, we only trim 8,192 pages of each working set instead of a percentage.

A Windows Server 2003-based computer becomes unresponsive because of a memory manager trimming operation that is caused by an indeterminate module that requests lots of memory
http://support.microsoft.com/kb/938486 

As you can see below after the hotfix is applied, the store working set remained steady throughout it's lifetime which is a huge improvement over the previous screenshots.

image

Shown below are the working sets from all processes. When these trim operations occur, we trim memory in a step stair pattern, thus preventing your server from all of this excessive paging. This more or less prevents faulty drivers from hanging/crashing your server.

image

ExBPA over time will be updated to detect drivers that may aggravate this problem, but at least there is some protection built in to the Operating system now to help this situation with this hotfix applied.

Currently, you can request this hotfix and the link is in the aforementioned article to get it. 

For a list of current recommendations to help alleviate these issue, click here

Mike

Comments
  • PingBack from http://geeklectures.info/2007/12/19/excessive-paging-on-exchange-2007-servers-when-working-sets-are-trimmed/

  • Very good article! Thanks for the information!

    Christian Schindler(MCA-M)

  • So do you recommend installing the KB938486 hotfix proactively on a Server 2003 computer running Exchange 2007 SP1 or only if this problem is observed?

  • I would put this hotfix on proactively to prevent a faulty 3rd party driver from taking down your entire exchange server.

  • Great post.  Do you know if this issues occurs on Windows 2008 w/ Exchange 2007 SP1?

    Thanks.

    -matt

  • This hotfix has been rolled in to Windows 2008, so this problem should not occur there.

  • Sounds good.  I'm eager to build out my Exchange 2007 + Windows 2008 platform and it's good to know I won't have to worry about this particular issue.

    Thanks.

    -matt

  • We have a similar problem with SAP/Oracle running on Windows 2003 32 Bit (/3GB Option and PAE active).

    4-5 times a day, the memory of every single process gets trimmed, the system is unresponsive for minutes.

    For example, Mem Usage of Oracle crashes from 2 GB down to less than 100 MB.

    This seeems to be a problem for each kind of database.

  • I am not an SQL expert by any means, but if you are having working set trimming issues on your server, then I would go ahead and request the hotfix, install it, and see how it goes. http://blogs.msdn.com/psssql/archive/2007/05/31/the-sql-server-working-set-message.aspx has more information on working set trimming problems and if you have GBs of RAM, this hotfix is definately going to help.

  • There are several hotfixes, which solve problems with memory trimming.

    But none of them helped us.

    The problem has been solved with Service Pack 2 for Windows 2003.

    We applied the SP one week ago, and since then this strange behaviour did nor occur again.

    Thx for your interest.

  • Mike posted a fantastic explanation for an excessive paging problem that you might get on Exchange 2007

  • Mike-

    Applied this patch on all my servers (2003 SP2, Exchange 2007 SP1).  Experienced an issue today where I couldn't login to the server via ILO or RDP.. Nothing.  Had to hard power the server off as well.  None of the services would really stop on the server, but mailbox access was working.. Weird..!

    Server has only 60 mailboxes, running 8GB of ram.  I'm curious as to why the PF Usage is 7.62 GB..  I checked the page file, and its set to only be 2048-4096 max.  Maybe I dont understand the windows pagefile.. How is this possible, and is it harmful?

    Skip

  • It is normal for the pagefile to allocate a certain amount of memory for certain processes and has been a very common question here lately. The only time that there is cause for concern is when we the pages/sec increases on a server for an extended period of time possibly due to working set trimming going on. Online Defrag, Content Indexing, Streaming backups, etc. can cause an increase in memory usage on any given Exchange server, but that just makes sense due to the amount of work going on. If the memory consumed by the store process should get very high and the amount of available RAM gets low, then the Operating System could start trimming working sets if a process/driver should request memory to do work that the OS does not have available. This can cause the store process or all processes to get their working sets trimmed causing huge performance issues affecting your client connectivity.

    See http://msexchangeteam.com/archive/2008/08/06/449484.aspx for more information.

  • Five Microsoft Exchange Server backup worst practices Third-party Exchange Server 2007 backup and restore

  • I've applied all the fixes here and still getting issues with excessive paging.

    8Gb RAM and currently 17.2Gb pagefile! What's weird is the pagefile is set to a min and max size of 12Gb.

    The working set of the store process is flat as a pancake. The server looks like it's undergoing back pressure right now - I have app server owner complaining their mails are getting dropped by the exchange server.

    The %Usage peak (_Total) for the Paging File counter is showing 100% utilisation.

    The server only hosts around 200 mailboxes, this is pretty crazy.

    They all use Enterpriuse Vault and we have around 16 Blackberry users.

    Could really do with some pointers on where to go from here...

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment