Why is Exchange Store.exe so RAM hungry?

re: Why is Exchange Store.exe so RAM hungry?

drebin — Mon, 02 Aug 2004 20:48:00 GMT

Wow.

The was the worst thing I've ever read.

I can't believe you just tried to justify a program eating 1.5GB (thats 1,500,000,000 bytes, you know)..

If you're car gastank is on empty, do you tell yourself "Is this affecting my driving at all?" or "Is the car still running fine?"

Just because it isn't affecting anything right now, that doesn't mean you don't have to do something about it!! How is one to do capacity planning - when your application will eat whatever memory it finds???

Wow. That's just horrible.

re: Why is Exchange Store.exe so RAM hungry?

Nino Bilic — Mon, 02 Aug 2004 21:03:00 GMT

Thanks for your comment :)

The truth of the matter is - the 1.5 GB figure is quite extreme. I did see it on very busy servers with 1000s of mailboxes though, and it was not a cause of the problem at all.

The whole point of this post was to shed some light on this as I have heard people rebooting their servers just because "Store was utilizing XYZ MBs" - so not because there was a performance problem on the server, but because of perception that Store is taking up more RAM than some specific number.

That in itself is not a problem - so it does not have to be fixed by a reboot. Especially not when there are 4 GBs of RAM in the machine for example (was true in the case where Store took 1.5 GB), and other applications that run on the server can get all the RAM they need.

re: Why is Exchange Store.exe so RAM hungry?

Nino Bilic — Mon, 02 Aug 2004 21:07:00 GMT

Sorry, just saw teh "capacity planning" part of the questuion - the behavior of Store (taking up more and more RAM if it needs it and if it is available) is not something that you necessarily need to try "compensate" for by adding more RAM to the machine. The process that Store goes through should prevent it from "starving" the OS and other applications on the server - so they have no RAM. So - no matter how much RAM you have on the server (unless it is very small for server application - where you will probably see problems with Exchange or no Exchange) - Store will act accordingly and stop growing after certain point.

re: Why is Exchange Store.exe so RAM hungry?

Nick Wade — Mon, 02 Aug 2004 21:17:00 GMT

When I was responsible for building and running in Exchange 2000 Clusters; if I put 2GB or 3GB of RAM in a box and Store *didn't* use 1.2GB on average (well, obviously user load is a factor here) then I KNEW I had a problem with that box. 1.2GB of RAM and idling on 3 to 4% CPU is a happy cluster node that'll run for months in that state - reboot - are you mad?

When you're supporting gobs of users on the one Exchange server, shouldn't it be a dedicated server, and shouldn't it use as much physical RAM as possible? In memory ops are so much more performant than those that must refer out to the page...

Next stop - fast fast FAST transaction log volumes :-) - then 2000 / 3000 corp users *concurrent* per box should be no problem at all.

Cheers

re: Why is Exchange Store.exe so RAM hungry?

Nick Wade — Mon, 02 Aug 2004 21:24:00 GMT

P.S. Oops, nearly forgot

Nino - good, reassuring article, good tips, thanks! ;-)

Them there are fightin' words there....

E-Bitz - SBS MVP the Official Blog of the SBS — Tue, 03 Aug 2004 08:15:00 GMT

Why is Exchange Store.exe so RAM hungry?

Welcome to Flaphead.com @ Home — Tue, 03 Aug 2004 11:50:00 GMT

Why is Exchange Store.exe so RAM hungry?

Welcome to Flaphead.com @ Home — Tue, 03 Aug 2004 11:51:00 GMT

re: Why is Exchange Store.exe so RAM hungry?

Jeff Centimano — Tue, 03 Aug 2004 11:54:00 GMT

The first comment (while being way too harsh and a little unprofessional) is typical of new Exchange admins. I run into this quite often as a consultant. Your article will be added to my "Knowledge Gems" folder to distribute to other newbie admins when this question comes up in the future. Thanks for taking the time to write this article. Keep up the good work.

re: Why is Exchange Store.exe so RAM hungry?

drebin — Tue, 03 Aug 2004 12:31:00 GMT

Jeff, for the record, I am no longer an Exchange admin - but have done it since the olden days. Secondly, sorry you cried over my post. The concept of having a poorly behaving product is one thing, but to justify it's behaviour was a little over the top for me.

To assume that Exchange is the only thing on that box is crazy. There could be other connectors - or what about backup software and a local tape drive. Imagine the backup taking 10 hours because it can't get enough RAM to do it more efficiently?

At least something like SQL Server let's you have some control over how much RAM you want to give it.. but this main post was a little out of control in my book.. Sorry I offended.

re: Why is Exchange Store.exe so RAM hungry?

Nino Bilic — Tue, 03 Aug 2004 23:04:00 GMT

drebin,

I am not sure - I guess it is possible I was not clear enough in my post - obviously it would be bad if Store does not care if anything else is running on the machine and just takes up RAM as it wants. That's why it said:

1. Exchange Store will grab as much RAM as it can if it thinks it needs it, yes. But - we constantly monitor the performance of the system in regards to memory usage and we can use this data to infer when we need more memory and when other applications or the OS needs more memory. We then use this data to act accordingly. This scheme allows the system to act as if there is explicit control when in fact it is actually a few autonomous applications cooperating in a disconnected manner.

In other words - we will decrease our (store) memory utilization as needed on the server too. Hope that clears it up?

re: Why is Exchange Store.exe so RAM hungry?

drebin — Wed, 04 Aug 2004 12:34:00 GMT

Understood - and sorry to make a stir.. :-)

Clearing up the Store.exe mystery

Andy Baran's Blog — Thu, 19 Aug 2004 22:43:00 GMT

re: Why is Exchange Store.exe so RAM hungry?

Joey Winn — Tue, 24 Aug 2004 14:04:00 GMT

Thank you all for your comments / information. I'm one of those 'new' Exchange Admins. I've only been doing it for about three years. (That's 'new', right?)

Well I actually DID encounter some problems with store.exe causing user-side memory errors (Exchange 2003). Two company users received a "90% memory used" - type of error message on their workstation. Sure enough, Store.exe had used up a huge amount of RAM (about 650Megs, out of our 1Gig available) which resulted in our server utilizing about 90% of the RAM available. No, there were no performance problems (unless you count the error and inability to send e-mails with any decent-sized attachment). Ok yeah, so there WERE performance problems. haha. Here comes my question....

Can I limit store.exe to a specific amount of alloted RAM (perhaps 500 Megs for example)?

Thanks. - joeywinn@yahoo.com
Joey Winn
Network Administrator
Bellomy Research, Inc.

re: Why is Exchange Store.exe so RAM hungry?

Nino Bilic — Tue, 24 Aug 2004 16:52:00 GMT

Joey,

There is no way to actually limit the amount of RAM that Exchange Store will use. So - you can not set a cap (limit) on it. That was always the case for Exchange - even in 5.5, even though the UI in Performance Optimizer suggested that you can do this - it would actually limit the buffers for the store rather than putting the cap on the memory use.

That being said - I would really suggest to open up a case with our support on this - if you are seeing memory errors.

re: Why is Exchange Store.exe so RAM hungry?

Colin Lee — Thu, 26 Aug 2004 00:33:00 GMT

One check I would do is to ensure that you have the /3GB and /Userva=3030 switch in the boot.ini as you have 1Gb or more of RAM.

http://support.microsoft.com/default.aspx?scid=kb;en-us;316739

re: Why is Exchange Store.exe so RAM hungry?

blast from the past — Thu, 26 Aug 2004 00:52:00 GMT

What Nino tried to explain above (memory backoff alogrithm) is called affectionately DBA which stands for Dynamic Buffer Allocation. Basically depending on the memory pressure being put on the system (and being sampled at a fairly aggressive rate), the store will release memory back to the system. For example, go to your Exchange server and start up IE or MMC. Both spike memory usage at thier load time. If you are watching the memory allocated to each process you will see Exchange store give up memory to the system (so it can then turn around and give it to IE / MMC). You will have to trust me.. this is how it works.

Now it wont release ALL of it at once, but will release it over (a relatively short) time. On the flip side, we don't aggressively eat it as it becomes available. Again based on memory pressure we will consume free memory in chunks. Think of it as Exchange's private memory manager that takes input from the entire system and is a benevolent participant.

The reason for doing it gradually is that you don't end up in a seesaw effect on the server where you are reacting to every absolute request for memory etc. Realize that the "memory" you are evicting is essentially the database cache and if you evict too aggressively and then re-allocate you end up with unpredictable and in-efficient IO patterns and your hit ratio on the cache drops (plus all the added IO's required to flush the cache pages to disk and the subsequent reads to fill the cache).

This now explains why on an Exchange server unused memory is essentially a wasted resource and if you can effectively approprirate and evict the resource, you net make your Exchange server as efficient as possible.