Troubleshooting memory issues on Exchange when using /3GB

Troubleshooting memory issues on Exchange when using /3GB

  • Comments 2
  • Likes

Hi all,

Windows Server 2003 x86 systems running Exchange require the use of the /3GB switch in the boot.ini to ensure that there is plenty of address space available for the Store.exe process. In brief, the /3GB switch is used to provide more virtual memory to user mode processing. It signals the OS to use 3 GB of the 4 GB virtual address space for each user mode process, as opposed to the normal 2 GB.. This leaves only 1 GB of the virtual address space for kernel mode processing, therefore the amount of Paged Pool Memory, Non-Paged Pool Memory and Available System PTEs on the server are reduced. As a result, there is often fine tuning required for problem free operations.

The theoretical maximums for Paged and Non-Paged Pool on Server 2003 32bit with the /3GB switch are:

Paged Pool = 256MB

Non-Paged Pool = 128MB

For a full explanation of how the /3GB switch works, please refer to the following AskPerf Blog article:

Memory Management - Demystifying /3GB

Issues caused by the /3GB switch will typically show up as some type of system resource depletion. A vast majority of the cases we see in this area are due to either Paged or Non-Paged Pool depletion. If this is occurring, there will be Event 2020 (Paged Pool depletion) or Event 2019 (Non-Paged Pool depletion) events in the System Event Log. We can also run low on Available System PTEs (Page Table Entries). Customers are often first made aware of problems in the Exchange environment when they see errors that HTTP.sys is refusing connections.

Making the following changes will almost always get the server back to a serviceable state. This is NOT to be considered a resolution, but is intended to get the server back online and serving mail quickly – additional data collection and analysis (perfmon, poolmon and possibly a complete memory dump) will be required to properly diagnose the underlying issue:

1. Make sure there is no more than 4GB of physical RAM in the system (i.e. do not have more RAM and use /MAXMEM or /BURNMEMORY as memory structures are still created at boot time in order to hide/remove memory when using these parameters).

2. Change the system to use /USERVA=2970 setting in the Boot.ini.

3. Set the system to use /BASEVIDEO in the Boot.ini and the display driver to Standard VGA.

4. To allow the Operating System to gracefully manage and allocate memory, set Paged Pool and System Pages to 0 for “system managed” by setting the following registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\control\Session Manager\Memory Management
SystemPages=0
PagedPoolSize=0
PoolUsageMaximum=60 (decimal)

5. Ensure you have the most current hotfix updates for Store.exe and Exifs.sys.

6. HTTP.sys is designed to stop servicing requests if the system comes within 20 MB of the non-paged pool limitation. As per KB 934878, you can set the EnableAggressiveMemoryUsage registry key to allow the system to get closer to the Non-Paged Pool limit before HTTP.sys stops servicing requests:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters
EnableAggressiveMemoryUsage=1 (DWORD)

7. Ensure settings are in place to capture a complete memory dump if necessary as per the following KB article:

972110 How to generate a kernel dump file or a complete memory dump file in Windows Server 2003

8. Reboot the system

 

When Microsoft is engaged for troubleshooting these issues, we often find that the highest consumers of Non-Paged Pool are Exchange / EXIFS related pool tags (AuxL or Flst). This is typical of a heavily loaded Exchange server. When you see this it is a good indicator that there are more users than the server can handle. You should consider the following:

- Bring another Exchange server online and redistribute the user load

- Move to a Server 2008 64bit solution for your Exchange server

- If there is plenty of paged pool available, you can move AuxL and Flst to paged pool as was the default behavior in SP1.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters
AuxFreeListInPagedPool=1 (DWORD)
FreeListInPagedPool=1 (DWORD)

Helpful Articles / References

934878  Users receive a "The page cannot be displayed" error message, and "Connections_refused" entries are logged in the Httperr.log file on a server that is running Windows Server 2003, Exchange 2003, and IIS 6.0

http://support.microsoft.com/default.aspx?scid=kb;EN-US;934878

912376  How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;912376.

823440  Use of the /3GB switch in Exchange Server 2003 on a Windows Server 2003-based system

http://support.microsoft.com/default.aspx?scid=kb;EN-US;823440

810371  Using the /Userva switch on Windows Server 2003-based computers that are running Exchange Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;810371

AskPerf Blog - Memory Management - Demystifying /3GB

http://blogs.technet.com/b/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx

Robin Aylesworth

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • I have found disabling unused network and HBA ports help with NPP, as well as comparing usage under different driver versions to scale Exchange 2003 clusters very high.

  • Great Post Tim!    On a few months ago, during a exchange server 2003 troubleshooting memory issue, we had found that system was run with lower free PTE's.. changing 2900 from 2970 resolved the issue.