Configuring Hot-Add and Large Memory for Exchange 2003

re: Configuring Hot-Add and Large Memory for Exchange 2003

Dan Sheehan — Tue, 27 Dec 2005 17:00:28 GMT

Two questions on a great post.
#1 - My lack of knowledge - I thought you could use PAE or 3GB, but not both. This is really confusing the consultants out there that I talk to because we understood this to be an either/or scenario. Can you go into a little more depth as to how the two different memory mangement methods can work together on the same box?
FYI - Most servers I run into do have the /NOPAE switch set in the BOOT.INI. So it sounds like that doesn't do anything w/o the EXECUTE command?

#2 - Do you generally recommend that dedicated mailbox servers with 4GB of RAM have the ESE cache set to 1.2 gig (the max recommended)? I have heard differing opinions on this, and I generally leave systems at the default setting until I know for a fact they should be modified. the ExBPA does't really make a recommendation either way (I wish it did). I know we can monitor the server like the KB suggests, but I was interested in a general overall recommendation.

re: Configuring Hot-Add and Large Memory for Exchange 2003

Igor — Tue, 27 Dec 2005 20:44:55 GMT

All our exchange servers installed on
HP ML370 G4 servers with PCI Express support
and 4 Gb RAM.
We have windows 2003 sp1 and exchange 2003 sp1 installed (both enterprise editions).
Is PAE switch must be enabled in this situation? Is /3GB switch must be enabled too?
Thanks.

re: Configuring Hot-Add and Large Memory for Exchange 2003

Paul Bowden [MSFT] — Thu, 29 Dec 2005 21:49:22 GMT

Dan...

#1 - No, you can set the /3GB switch and have the PAE kernel loaded together. However, both of these squeeze kernel memory, and that's why it's important to tune the server appropriate. The PAE kernel uses wider page table entries (PTE) so that it can accommodate physical memory above the 4GB boundary. As an aside, these wider PTEs are also needed to support the Data Execution Prevention (DEP) feature in Windows 2003 SP1. The /3GB switch modifies the 4GB of virtual address space allocation for each process so instead of each app getting 2GB of user and 2GB of kernel space, the app gets 3GB of user and only 1GB of kernel space. Because Exchange uses a single store.exe process, it requires the larger user address space to scale. You can think of the PAE kernel as changeing physical memory allocation, and the /3GB switch changing virtual memory allocation.

#2 - Great question. As a general rule, we recommend leaving the cache size at its default of ~900M. While it's true that a scale-up server will perform better with a larger cache size (i.e. 1.2GB), you run the risk of depleting the virtual address space which can cause a service outage. As with most tuning parameters, you have to balance performance vs. stability. You are correct that ExBPA doesn't provide a concrete recommendation right now. This is mainly because there continue to be different schools of thought on tuning this parameter. Here at Microsoft, we're running one large server with a 1.2GB cache (>4000 users), whereas all other Exchange servers are running with the default cache size. This one server is running closer 'to the wire' in terms of stability, and we have seen one instance where the lack of virtual memory caused an outage. NOTE: Just because you have 4GB of physical RAM in a server, it doesn't mean that it's safe to increase cache settings. At the end of the day, you'll run out of virtual memory in the process before physical memory. The amount of virtual memory is constrained by the 32-bit OS, which is a big reason why we're looking to 64-bit with E12.

re: Configuring Hot-Add and Large Memory for Exchange 2003

Paul Bowden [MSFT] — Thu, 29 Dec 2005 21:53:12 GMT

Igor. Because the ML370 supports hot-add memory and DEP, you'll find that the PAE kernel will automatically load with Windows 2003 SP1. Therefore, you don't need to explicitly set the /PAE switch. However, assuming that this is a mailbox server, you will need to set the /3GB switch manually in the Boot.ini. Your best bet is to run the ExBPA tool (www.exbpa.com) as this will programmatically analyze many aspects of your server (in fact, it analyzes nearly 2500 settings and variables), and make recommendations for tuning.

re: Configuring Hot-Add and Large Memory for Exchange 2003

Dan Sheehan — Tue, 03 Jan 2006 18:13:06 GMT

#1. We should leave the /NOPAE switch in the BOOT.INI files since most customers do not put more than 4GB of RAM in their boxes, and they don't honestly care about DEP on dedicated & locked down servers. I.E. They do not appear to get any benefit from adding it on 4GB machines other than DEP support.

#2. Understood - I think the fact that you all are only running the mod on one server speaks volumes. I would be interested to know if the virtual memory issues you all saw had anything to do with the fact that the server was on a cluster (assuming it was since ITG clusters every production mail server from what I know).

Thanks for the continued advice and support Paul!

Exchange hotfix for kernel memory exhaustion issues is now released

You Had Me At EHLO... — Mon, 27 Feb 2006 21:48:16 GMT

EXECUTIVE SUMMARY
&nbsp;
A few weeks ago,&nbsp;we released three CXP flashes&nbsp;on the subject of...