Determining the Maximum Sizes of Kernel Memory Pools

clinth@microsoft.com — Fri, 27 Mar 2009 05:23:34 GMT

While onsite with customers, I have found several more cases where the Windows Server 2003 32-bit is running dangerously low on kernel memory. Here lately it has been a lack of pool paged memory. In any case, you can use my earlier blog posts to estimate the kernel memory size. With that said, when you run into this issue, you need to really know what the real maximum sizes are. In this blog post, I will show you two relatively easy ways to get this information.

The memory pool usage in pool paged memory and pool non-paged memory should not exceed 80% of their respective maximums. The pool usage values are exposed the performance counters, “\Memory\Pool Paged Bytes” and “\Memory\Pool NonPaged Bytes”, but their maximum sizes can only be found by debugging the kernel. I know that debugging the kernel sounds very intimidating, but I assure you it is easy, but *dangerous* if you mess it up, so try this in a test environment first.

Getting Maximum Sizes for Pool Paged Bytes and Pool Non-Paged Bytes:

Install the Microsoft Debugging Tools for Windows. This is a free download from http://download.microsoft.com. Installation of these tools does not require a reboot.
Use Microsoft WinDBG or Process Explorer (both Microsoft tools):

WinDBG method:

Open WinDBG.

Ensure the symbol path is set: Click File, “Symbol File Path…”. Ensure the following path is in place without the double quotes: “SRV*C:\symbols*http://msdl.microsoft.com/download/symbols”. Click OK.
Click File, “Kernel Debug…”. Select the Local tab, then click OK.
Select “No” to any Workspace dialog boxes.
At the “lkd>” prompt, type “!vm”
“PagedPool Maximum” is the maximum size for the Pool Paged memory.
“NonPagedPool Max” is the maximum size for the Pool Non-Paged memory.

Process Explorer method: This is preferred if you need to get the maximum pool sizes often.

Download Process Explorer from:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx or run directly from http://live.sysinternals.com.
No installation is required.
Run procexp.exe.
Ensure the symbol path is set: Click Options, “Configure Symbols…”
Set the Symbols path to (without the double quotes):
“SRV*C:\symbols*http://msdl.microsoft.com/download/symbols”
Set the Dbghelp.dll path to the installation directory of the Debugging Tools for Windows. Default location is: “C:\Program Files\Debugging Tools for Windows\dbghelp.dll” Click OK.
Click View, “System Information…”
“Paged Limit” is the maximum size for the Pool Paged Memory.
“Nonpaged Limit” is the maximum size for the Pool Non-Paged memory.
If you need to get the maximum pool sizes again, then just follow steps 2, 6 and 7.

Identifying 32-bit Kernel Memory Issues

clinth@microsoft.com — Tue, 02 Oct 2007 00:55:00 GMT

I'm a Premier Field Engineer (PFE) and I go onsite with customers on a regular basis to conduct Health Checks. This is my first blog posting on TechNet, but I figured it would be on something important versus "Hello World". ;-)

More and more I am seeing customers who are not aware of kernel memory issues on the 32-bit Windows architecture. If you are running 32-bit Windows 2000 or 32-bit Windows 2003, then check the kernel memory. Lack of kernel memory can lead to system-wide hangs which seem unexplainable, so this is a serious issue. Always generally use and/or recommend 64-bit or Windows Server 2008 server to avoid these issues (Windows Server 2008 and Windows Vista have automatically adjusting kernel memory pool sizes). This issue and other performance issues are addressed in the Vital Signs workshop (written by Shane Creamer), which can be delivered by my team, Premier Field Engineering (PFE) - just contact your Technical Account Manager (TAM) if you are interested in this course.

Here is a kernel memory chart for Windows 2003 Server:

Memory

Default ( /PAE for 6-16GB )

/3GB

1GB

Free System PTE: 51k

Paged Pool: 282MB

Non Paged Pool: 212MB

Free System PTE: 32k

Paged Pool: 163MB

Non Paged Pool: 131MB

2GB

Free System PTE: 196k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 16k

Paged Pool: 262MB

Non Paged Pool: 131MB

3GB

Free System PTE: 195k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 14k

Paged Pool: 262MB

Non Paged Pool: 131MB

4GB

Free System PTE: 106k

Paged Pool: 336MB

Non Paged Pool: 285MB

Free System PTE: 15k

Paged Pool: 258MB

Non Paged Pool: 154MB

6GB

Free System PTE: 186k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 239MB

Non Paged Pool: 131MB

8GB

Free System PTE: 182k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 225MB

Non Paged Pool: 131MB

12GB

Free System PTE: 175k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 196MB

Non Paged Pool: 131MB

16GB

Free System PTE: 167k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 169MB

Non Paged Pool: 131MB

Check your 32-bit servers to see if they are within 80% of the memory pool sizes for Pool Paged Memory and Pool non-Paged memory according to the chart above. The Free Page Table Entries (PTEs) listed in the chart above is the starting amount based on the specification of the hardware and boot.ini switches. Here are the performance counters to monitor:

Memory\Free System Page Table Entries

Memory\Pool Nonpaged Bytes

Memory\Pool Paged Bytes

For example, if I have a server that is a Windows 2003 SP1 (SP1 is needed to accurately see the Free PTE data in performance monitor) with 4GB of memory and /3GB switch turned on, then Free System PTE’s start off at 15,000 (according to the chart). If the value gets below 5,000, then the system could hang temporarily. This system has a Paged Pool memory maximum of 258MB and Non-Paged Pool maximum memory of 154MBs. If the respective “Memory\Pool Paged Bytes” and “Memory\Pool Nonpaged Bytes” counter values come within 80% of these maximum pool sizes, then the system could hang temporarily – in this case if the values go over 206MB’s and 123MB’s respectively, then it’s a critical issue.

Fix for Win2003 SP1 systems with /3GB and low on PTE’s: If the system is low on PTE’s, running Windows 2003, and using /3GB switch, then consider using the /USERVA switch to give back some of the memory to the kernel. Note, this only works for Free System PTE issues. For more information on the USERVA switch, go to:

How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB

http://support.microsoft.com/kb/316739

Lack of Paged Pool or non-Paged Pool Memory: If the system is low on Paged Pool or non-Paged pool memory, then first consider opening a support case with Microsoft to address this. Alternatively, you can use a free and public tool called Poolmon.exe to see what DLL’s are using kernel memory (see the article below). Most kernel memory leaks can be tracked back to a usermode process. To identify which user mode process is responsible, reboot the system (so you start off with a clean system), start a performance monitor log intending to run for a week or more capturing the Memory and Process objects, then analyze the perfmon log looking for memory leaks and/or handle leaks in one or more of the processes.

How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks

http://support.microsoft.com/kb/177415

In summary, always consider 64-bit and always keep an eye on kernel memory.

In addition, I and a few of my colleages wrote a tool that can help identify these and many other performance issues by analyzing performance counter logs. The tool is located at http://www.codeplex.com/PAL. Codeplex.com is Microsoft's open source web site.

Clint Huffman's Windows Performance Blog : memory

Determining the Maximum Sizes of Kernel Memory Pools

Getting Maximum Sizes for Pool Paged Bytes and Pool Non-Paged Bytes:

WinDBG method:

Process Explorer method: This is preferred if you need to get the maximum pool sizes often.

Identifying 32-bit Kernel Memory Issues