Identifying 32-bit Kernel Memory Issues

clinth@microsoft.com — Tue, 02 Oct 2007 00:55:00 GMT

I'm a Premier Field Engineer (PFE) and I go onsite with customers on a regular basis to conduct Health Checks. This is my first blog posting on TechNet, but I figured it would be on something important versus "Hello World". ;-)

More and more I am seeing customers who are not aware of kernel memory issues on the 32-bit Windows architecture. If you are running 32-bit Windows 2000 or 32-bit Windows 2003, then check the kernel memory. Lack of kernel memory can lead to system-wide hangs which seem unexplainable, so this is a serious issue. Always generally use and/or recommend 64-bit or Windows Server 2008 server to avoid these issues (Windows Server 2008 and Windows Vista have automatically adjusting kernel memory pool sizes). This issue and other performance issues are addressed in the Vital Signs workshop (written by Shane Creamer), which can be delivered by my team, Premier Field Engineering (PFE) - just contact your Technical Account Manager (TAM) if you are interested in this course.

Here is a kernel memory chart for Windows 2003 Server:

Memory

Default ( /PAE for 6-16GB )

/3GB

1GB

Free System PTE: 51k

Paged Pool: 282MB

Non Paged Pool: 212MB

Free System PTE: 32k

Paged Pool: 163MB

Non Paged Pool: 131MB

2GB

Free System PTE: 196k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 16k

Paged Pool: 262MB

Non Paged Pool: 131MB

3GB

Free System PTE: 195k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 14k

Paged Pool: 262MB

Non Paged Pool: 131MB

4GB

Free System PTE: 106k

Paged Pool: 336MB

Non Paged Pool: 285MB

Free System PTE: 15k

Paged Pool: 258MB

Non Paged Pool: 154MB

6GB

Free System PTE: 186k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 239MB

Non Paged Pool: 131MB

8GB

Free System PTE: 182k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 225MB

Non Paged Pool: 131MB

12GB

Free System PTE: 175k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 196MB

Non Paged Pool: 131MB

16GB

Free System PTE: 167k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 169MB

Non Paged Pool: 131MB

Check your 32-bit servers to see if they are within 80% of the memory pool sizes for Pool Paged Memory and Pool non-Paged memory according to the chart above. The Free Page Table Entries (PTEs) listed in the chart above is the starting amount based on the specification of the hardware and boot.ini switches. Here are the performance counters to monitor:

Memory\Free System Page Table Entries

Memory\Pool Nonpaged Bytes

Memory\Pool Paged Bytes

For example, if I have a server that is a Windows 2003 SP1 (SP1 is needed to accurately see the Free PTE data in performance monitor) with 4GB of memory and /3GB switch turned on, then Free System PTE’s start off at 15,000 (according to the chart). If the value gets below 5,000, then the system could hang temporarily. This system has a Paged Pool memory maximum of 258MB and Non-Paged Pool maximum memory of 154MBs. If the respective “Memory\Pool Paged Bytes” and “Memory\Pool Nonpaged Bytes” counter values come within 80% of these maximum pool sizes, then the system could hang temporarily – in this case if the values go over 206MB’s and 123MB’s respectively, then it’s a critical issue.

Fix for Win2003 SP1 systems with /3GB and low on PTE’s: If the system is low on PTE’s, running Windows 2003, and using /3GB switch, then consider using the /USERVA switch to give back some of the memory to the kernel. Note, this only works for Free System PTE issues. For more information on the USERVA switch, go to:

How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB

http://support.microsoft.com/kb/316739

Lack of Paged Pool or non-Paged Pool Memory: If the system is low on Paged Pool or non-Paged pool memory, then first consider opening a support case with Microsoft to address this. Alternatively, you can use a free and public tool called Poolmon.exe to see what DLL’s are using kernel memory (see the article below). Most kernel memory leaks can be tracked back to a usermode process. To identify which user mode process is responsible, reboot the system (so you start off with a clean system), start a performance monitor log intending to run for a week or more capturing the Memory and Process objects, then analyze the perfmon log looking for memory leaks and/or handle leaks in one or more of the processes.

How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks

http://support.microsoft.com/kb/177415

In summary, always consider 64-bit and always keep an eye on kernel memory.

In addition, I and a few of my colleages wrote a tool that can help identify these and many other performance issues by analyzing performance counter logs. The tool is located at http://www.codeplex.com/PAL. Codeplex.com is Microsoft's open source web site.

Clint Huffman's Windows Performance Blog : poolmon.exe

Identifying 32-bit Kernel Memory Issues