This blog provides some guidance on how to optimize Exchange 2000/2003 on Windows OS for paged and non-paged pool issues.
Symptoms
Event ID 2020 Event Type: Error Event Source: Srv Event ID: 2020 Description: The server was unable to allocate from the system paged pool because the pool was empty. Event ID 2019 Event Type: Error Event Source: Srv Event ID: 2019 Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.
Other symptoms of pool exhaustion on the system include application or process hangs, out of resource errors reported by drivers or applications, the server becomes slow or refuses additional requests and connections, or all of the above!
Understanding Pool
When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool. Pool memory is allocated statically during Windows startup. Available pool memory depends on several factors to include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.
Pool memory is not the amount of RAM on the system. It is a segment of the virtual memory or address space that Windows reserves on boot. These pools are finite because x86 OS can only address 2^32==4GB. By default, Windows uses 2GB for applications and 2GB for kernel.
These pools are used by either the kernel directly, indirectly by its support of various structures due to application requests on the system (CreateFile for example), or drivers installed on the system for their memory allocations made via the kernel pool allocation functions.
NonPaged means that this memory when allocated will not be paged to disk and thus resident at all times, which is an important feature for drivers. Paged conversely can be paged out to disk. In the end though, all this memory is allocated through a common set of functions, most common is ExAllocatePoolWithTag.
System PTEs
Page Tables are built for each process address space. The Page Table maps logical virtual addresses for a process to physical memory locations. System Page Table Entries (PTEs) are used to map system pages such as I/O space, Kernel stacks, and memory descriptor lists. Tuning the memory by using the USERVA switch in conjunction with the /3GB switch can often stave off PTE depletion issues.
/USERVA
A boot.ini switch used for more precise tuning of user and kernel virtual memory space in the Windows Server 2003. Use this switch in conjunction with /3GB switch in the Boot.ini file to tune reduce the User-mode space, allowing the difference to be returned to Kernel mode. Standard configuration is to use /USERVA=3030. However there might be several situations where you need to allow for more PTEs to become available. On an Exchange server, the value of USERVA should not be lower than 2970 without consulting Windows Performance team.
/3GB
A boot.ini switch that allocates 1 GB to the kernel and 3 GB to the User-mode space. Using this switch reduces the memory available for Nonpaged Pool, Paged Pool, & System Page Table Entries (PTEs).
/PAE
A Boot.ini switch. When more than 4GB of physical memory is used on the system, the process of paging memory to the disk increases dramatically, and performance may be negatively impacted. The Windows memory managers use PAE to provide more physical memory to the operating system. This reduces the need to swap the memory in and out of the page file and results in increased performance. All the memory management and allocation of the PAE memory is handled by the memory manager independently of the running programs. It is now a best practice to allow the PAE kernel to load on an Exchange 2003 server.
Pool Tags
A pool tag is a four-byte character that is associated with a dynamically allocated chunk of pool memory. The tag is specified by a driver when it allocates the memory. The routine ExAllocatePoolWithTag is called to allocate pool memory. Pool tags are useful for identifying which drivers are allocating nonpaged and paged pool memory.
By the Numbers
Review the chart at the bottom of this document to identify the default boot up of Paged Pool, NonPaged Pool, and Free System PTE as it applies to your system configuration.
A typical large-scale Exchange 2003 server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention. Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.
In general, a system should always have around 10,000 free System PTE’s. If the value gets below 5,000, then the system could hang temporarily.
64-bit resolves this, right?
Not quite. While 64-bit programs can take advantage of the 16-TB tuning model (8 terabytes User and 8 terabytes Kernel) that 64-bit OS has, 32-bit programs still use the 4-GB tuning model (2 GB User and 2 GB Kernel). This means that 32-bit processes that run on 64-bit versions of Windows run in a 4-GB tuning model. So by running 32-bit programs on 64-bit OS, you could still run into limitations.
Troubleshooting Memory Depletion
Here are some basic troubleshooting steps to identifying the problem with Pool Memory depletion on Exchange 2000/2003 server.
Common Causes of Pool Depletion
NOTE: Size of the PF store does not matter – it is the connections to the store
Data Collection
If experiencing symptoms, collect the following data:
Common Best Practices
The following are some best practices for managing paged and nonpaged pool issues on Exchange 2000/2003.
NOTE: Before performing these steps, run Poolmon on the server at difference intervals to keep track of system changes
NOTE: There are other settings and adjustments that can be made to the system to potentially improve Paged and NonPaged Pool performance. Those adjustments should only be made after completing the above steps and consulting with Microsoft Services.
Poolmon is Windows Support Tool that displays data that the operating system collects about memory allocations from the system paged and non-paged kernel pools, and the memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag.
NOTE: Pool tagging is permanently enabled on Windows Server 2003. However, in earlier OS versions (including XP), you must use Gflags.exe to enable pool tagging.
1. Open a command prompt and view the available options: Poolmon /? -s Display session pool -n [Logfile] Take a pool snapshot (Logfile maybe specified, default is poolsnap.log) -c [LocalTagFile] Display driver information using LocalTagFile -g [PoolTagFile] Display driver information using PoolTagFile -itag Include the tag -xtag Exclude the tag -e Display totals -t Sort by tags -a Sort by allocs -u | -b Sort by Bytes -f Sort by free -d Sort by diff -m Sort by each -l Highlight -p First turns on nonpaged, second turns on paged -( | -) Increase parenthesis -r Print memory summary information 2. Copy the Pooltag.txt file from the Windows Debugging tools directory to the system running poolmon. This will allow you to map the driver to the tag. 3. Next run Poolmon -n –b -g Review the poolsnap.log. The tags should be sorted by byte usage.