Memory Management - Understanding Pool Resources

Following up on our Memory Management 101 post, we're moving on to a discussion of Pool Resources and Pool Resource Depletion.  First of all - what are Pool Resources?  When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool.  Each of these pools start at an initial size that is based upon the amount of physical memory present in the system.  Pool memory is a subset of available memory and is not necessarily contiguous.  If necessary, these pools can grow up to a maximum size that is determined by the system at boot time.

So - what distinguishes Paged Pool and NonPaged Pool memory?  The first difference is that Paged Pool is exactly what its name implies - it can be paged out.  The NonPaged Pool cannot be paged out.  Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any Interrupt Request Level (IRQL).  The IRQL defines the hardware priority at which a processor operates at any given time (there's a link to a document covering Scheduling, Thread Context and IRQL's in the Additional Resources section at the end of this post).

Getting back to our Pool Resources, it is important to remember that these resources are finite.  The table below outlines some sample maximum values for Paged / NonPaged Pool on x86 systems that are not configured with the /3GB switch in the system's boot.ini file.  We'll cover /3GB and its effects on memory in a future post.  We'll also cover Kernel Changes to Windows Vista separately.  It's important to note that x64 systems don't suffer from the same Virtual Address Space limitations! 

Windows 2000

 * If Terminal Services is installed on Windows 2000, Paged Pool is lowered down to 160 MB unless a registry change is made to the server to set the Paged Pool Size to its maximum value (see below).

Windows 2003 SP1

On Windows 2003 systems, Terminal Services are enabled by default. 

On both Windows 2000 and Windows 2003, the HKLM\System\CurrentControlSet\Control\Session Management\Memory Management\PagedPoolSize value can be set to 0xFFFFFFFF (or resetting the value to 0) to ensure that the Virtual Address Space used for Paged Pool is maximized.

Also - here's the theoretical maximums for pre-Vista Operating Systems:

 * depends on whether or not /3GB is enabled

Now that we know what the maximum value ranges should look like, here's how to verify what those values look like on your own system using Process Explorer:

1. Download Process Explorer from the Microsoft Sysinternals Site.
2. Unzip the contents of the Zip file to C:\ProcessExplorer
3. Run ProcExp.exe
4. The first thing you'll want to do is configure the Symbols and the dbghelp.dll path.  If you leave the default path alone (see the image below), you'll get an error message that you haven't configured symbols - even if you have the Symbol path filled in (the Procexp.chm file in the ProcessExplorer folder provides instructions).
   
5. To get the most out of Process Explorer however, you'll need to install the Microsoft Debugging Tools (we'll be needing these later anyway when we start getting into troubleshooting).  This is important because you can specify the proper version of dbghelp.dll.  Once you have Process Explorer fully configured (see the image below) for the settings, you're ready to check out your Pool Resources:
   
6. Within Process Explorer, select View ... System Information ... and look at the Kernel Memory Section.  The Paged Limit and NonPaged Limit show the current maximum values for the system you are examining.
   

Now, let's examine what type of items reside in each of these pools.  Within the NonPaged pool, you would find handles that are used by applications in the user-mode space as well as Kernel-Mode drivers (typically ending in a .sys file extension).  Examples of Paged Pool items are Token Objects, Kernel-Mode drivers and the Registry.

Here's where things get really interesting - what happens to a system when these Pool Resources get depleted?  Some of the most common symptoms exhibited are:

• the machine becomes sluggish
• users can no longer log on to the machine
• console access is sluggish
• users cannot connect to file shares or other shared resources
• system hangs including the console itself being unresponsive

Symptoms such as this are usually the first indication that there is something causing an issue with the machine. 

If the NonPaged Pool on a server has become depleted, the machine will log an Event in the System Log as shown below:

Event ID 2019
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.

Paged Pool Depletion is logged as an Event 2020:

Event ID 2020
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Description: The server was unable to allocate from the system paged pool because the pool was empty.

What are these error messages telling us beyond the fact that there is an issue with Pool Depletion?  A common misunderstanding of this message is that the problem is being caused by the Server Service (srv.sys).  Usually the Server Service is the first component to experience the issue because it is trying to satisfy a request and cannot allocate the appropriate Pool Memory.

We'll cover 2019 & 2020 troubleshooting in greater depth in a future post including how to use tools such as Poolmon and Perfmon in conjunction with a Memory Dump to find the culprits!  However, to get you started, I've included links to a great post by our CPR team on their blog as well as some more general information regarding Pool Memory.  The links are below.

Coming soon - Memory Tuning, Troubleshooting Memory Issues and Using the /3GB switch

Additional Resources:

• Microsoft® Windows® Internals, Fourth Edition: Microsoft Windows Server™ 2003, Windows XP, and Windows 2000 (Chapter 7 covers Memory Management)
• Scheduling, Thread Context, and IRQL
• NT Debugging - Troubleshooting 2019 / 2020 errors

 - CC Hameed