Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
In previous Pushing the Limits posts, I described the two most basic system resources, physical memory and virtual memory. This time I’m going to describe two fundamental kernel resources, paged pool and nonpaged pool, that are based on those, and that are directly responsible for many other system resource limits including the maximum number of processes, synchronization objects, and handles.
Here’s the index of the entire Pushing the Limits series. While they can stand on their own, they assume that you read them in order.
Pushing the Limits of Windows: Physical Memory Pushing the Limits of Windows: Virtual Memory Pushing the Limits of Windows: Paged and Nonpaged Pool Pushing the Limits of Windows: Processes and Threads Pushing the Limits of Windows: Handles Pushing the Limits of Windows: USER and GDI Objects – Part 1 Pushing the Limits of Windows: USER and GDI Objects – Part 2
Pushing the Limits of Windows: Physical Memory
Pushing the Limits of Windows: Virtual Memory
Pushing the Limits of Windows: Paged and Nonpaged Pool
Pushing the Limits of Windows: Processes and Threads
Pushing the Limits of Windows: Handles
Pushing the Limits of Windows: USER and GDI Objects – Part 1
Pushing the Limits of Windows: USER and GDI Objects – Part 2
Paged and nonpaged pools serve as the memory resources that the operating system and device drivers use to store their data structures. The pool manager operates in kernel mode, using regions of the system’s virtual address space (described in the Pushing the Limits post on virtual memory) for the memory it sub-allocates. The kernel’s pool manager operates similarly to the C-runtime and Windows heap managers that execute within user-mode processes. Because the minimum virtual memory allocation size is a multiple of the system page size (4KB on x86 and x64), these subsidiary memory managers carve up larger allocations into smaller ones so that memory isn’t wasted.
For example, if an application wants a 512-byte buffer to store some data, a heap manager takes one of the regions it has allocated and notes that the first 512-bytes are in use, returning a pointer to that memory and putting the remaining memory on a list it uses to track free heap regions. The heap manager satisfies subsequent allocations using memory from the free region, which begins just past the 512-byte region that is allocated.
The kernel and device drivers use nonpaged pool to store data that might be accessed when the system can’t handle page faults. The kernel enters such a state when it executes interrupt service routines (ISRs) and deferred procedure calls (DPCs), which are functions related to hardware interrupts. Page faults are also illegal when the kernel or a device driver acquires a spin lock, which, because they are the only type of lock that can be used within ISRs and DPCs, must be used to protect data structures that are accessed from within ISRs or DPCs and either other ISRs or DPCs or code executing on kernel threads. Failure by a driver to honor these rules results in the most common crash code, IRQL_NOT_LESS_OR_EQUAL.
Nonpaged pool is therefore always kept present in physical memory and nonpaged pool virtual memory is assigned physical memory. Common system data structures stored in nonpaged pool include the kernel and objects that represent processes and threads, synchronization objects like mutexes, semaphores and events, references to files, which are represented as file objects, and I/O request packets (IRPs), which represent I/O operations.
Paged pool, on the other hand, gets its name from the fact that Windows can write the data it stores to the paging file, allowing the physical memory it occupies to be repurposed. Just as for user-mode virtual memory, when a driver or the system references paged pool memory that’s in the paging file, an operation called a page fault occurs, and the memory manager reads the data back into physical memory. The largest consumer of paged pool, at least on Windows Vista and later, is typically the Registry, since references to registry keys and other registry data structures are stored in paged pool. The data structures that represent memory mapped files, called sections internally, are also stored in paged pool.
Device drivers use the ExAllocatePoolWithTag API to allocate nonpaged and paged pool, specifying the type of pool desired as one of the parameters. Another parameter is a 4-byte Tag, which drivers are supposed to use to uniquely identify the memory they allocate, and that can be a useful key for tracking down drivers that leak pool, as I’ll show later.
There are three performance counters that indicate pool usage:
However, there are no performance counters for the maximum size of these pools. They can be viewed with the kernel debugger !vm command, but with Windows Vista and later to use the kernel debugger in local kernel debugging mode you must boot the system in debugging mode, which disables MPEG2 playback.
So instead, use Process Explorer to view both the currently allocated pool sizes, as well as the maximum. To see the maximum, you’ll need to configure Process Explorer to use symbol files for the operating system. First, install the latest Debugging Tools for Windows package. Then run Process Explorer and open the Symbol Configuration dialog in the Options menu and point it at the dbghelp.dll in the Debugging Tools for Windows installation directory and set the symbol path to point at Microsoft’s symbol server:
After you’ve configured symbols, open the System Information dialog (click System Information in the View menu or press Ctrl+I) to see the pool information in the Kernel Memory section. Here’s what that looks like on a 2GB Windows XP system:
2GB 32-bit Windows XP
As I mentioned in a previous post, on 32-bit Windows, the system address space is 2GB by default. That inherently caps the upper bound for nonpaged pool (or any type of system virtual memory) at 2GB, but it has to share that space with other types of resources such as the kernel itself, device drivers, system Page Table Entries (PTEs), and cached file views.
Prior to Vista, the memory manager on 32-bit Windows calculates how much address space to assign each type at boot time. Its formulas takes into account various factors, the main one being the amount of physical memory on the system. The amount it assigns to nonpaged pool starts at 128MB on a system with 512MB and goes up to 256MB for a system with a little over 1GB or more. On a system booted with the /3GB option, which expands the user-mode address space to 3GB at the expense of the kernel address space, the maximum nonpaged pool is 128MB. The Process Explorer screenshot shown earlier reports the 256MB maximum on a 2GB Windows XP system booted without the /3GB switch.
The memory manager in 32-bit Windows Vista and later, including Server 2008 and Windows 7 (there is no 32-bit version of Windows Server 2008 R2) doesn’t carve up the system address statically; instead, it dynamically assigns ranges to different types of memory according to changing demands. However, it still sets a maximum for nonpaged pool that’s based on the amount of physical memory, either slightly more than 75% of physical memory or 2GB, whichever is smaller. Here’s the maximum on a 2GB Windows Server 2008 system:
2GB 32-bit Windows Server 2008
64-bit Windows systems have a much larger address space, so the memory manager can carve it up statically without worrying that different types might not have enough space. 64-bit Windows XP and Windows Server 2003 set the maximum nonpaged pool to a little over 400K per MB of RAM or 128GB, whichever is smaller. Here’s a screenshot from a 2GB 64-bit Windows XP system:
2GB 64-bit Windows XP
64-bit Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 memory managers match their 32-bit counterparts (where applicable – as mentioned earlier, there is no 32-bit version of Windows Server 2008 R2) by setting the maximum to approximately 75% of RAM, but they cap the maximum at 128GB instead of 2GB. Here’s the screenshot from a 2GB 64-bit Windows Vista system, which has a nonpaged pool limit similar to that of the 32-bit Windows Server 2008 system shown earlier.
2GB 32-bit Windows Server 2008
Finally, here’s the limit on an 8GB 64-bit Windows 7 system:
8GB 64-bit Windows 7
Here’s a table summarizing the nonpaged pool limits across different version of Windows:
The kernel and device drivers use paged pool to store any data structures that won’t ever be accessed from inside a DPC or ISR or when a spinlock is held. That’s because the contents of paged pool can either be present in physical memory or, if the memory manager’s working set algorithms decide to repurpose the physical memory, be sent to the paging file and demand-faulted back into physical memory when referenced again. Paged pool limits are therefore primarily dictated by the amount of system address space the memory manager assigns to paged pool, as well as the system commit limit.
On 32-bit Windows XP, the limit is calculated based on how much address space is assigned other resources, most notably system PTEs, with an upper limit of 491MB. The 2GB Windows XP System shown earlier has a limit of 360MB, for example:
2GB 32-bit Windows XP
32-bit Windows Server 2003 reserves more space for paged pool, so its upper limit is 650MB.
Since 32-bit Windows Vista and later have dynamic kernel address space, they simply set the limit to 2GB. Paged pool will therefore run out either when the system address space is full or the system commit limit is reached.
64-bit Windows XP and Windows Server 2003 set their maximums to four times the nonpaged pool limit or 128GB, whichever is smaller. Here again is the screenshot from the 64-bit Windows XP system, which shows that the paged pool limit is exactly four times that of nonpaged pool:
2GB 64-bit Windows XP
Finally, 64-bit versions of Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 simply set the maximum to 128GB, allowing paged pool’s limit to track the system commit limit. Here’s the screenshot of the 64-bit Windows 7 system again:
Here’s a summary of paged pool limits across operating systems:
Because the kernel pools are used by almost every kernel operation, exhausting them can lead to unpredictable results. If you want to witness first hand how a system behaves when pool runs low, use the Notmyfault tool. It has options that cause it to leak either nonpaged or paged pool in the increment that you specify. You can change the leak size while it’s leaking if you want to change the rate of the leak and Notmyfault frees all the leaked memory when you exit it:
Don’t run this on a system unless you’re prepared for possible data loss, as applications and I/O operations will start failing when pool runs out. You might even get a blue screen if the driver doesn’t handle the out-of-memory condition correctly (which is considered a bug in the driver). The Windows Hardware Quality Laboratory (WHQL) stresses drivers using the Driver Verifier, a tool built into Windows, to make sure that they can tolerate out-of-pool conditions without crashing, but you might have third-party drivers that haven’t gone through such testing or that have bugs that weren’t caught during WHQL testing.
I ran Notmyfault on a variety of test systems in virtual machines to see how they behaved and didn’t encounter any system crashes, but did see erratic behavior. After nonpaged pool ran out on a 64-bit Windows XP system, for example, trying to launch a command prompt resulted in this dialog:
On a 32-bit Windows Server 2008 system where I already had a command prompt running, even simple operations like changing the current directory and directory listings started to fail after nonpaged pool was exhausted:
On one test system, I eventually saw this error message indicating that data had potentially been lost. I hope you never see this dialog on a real system!
Running out of paged pool causes similar errors. Here’s the result of trying to launch Notepad from a command prompt on a 32-bit Windows XP system after paged pool had run out. Note how Windows failed to redraw the window’s title bar and the different errors encountered for each attempt:
And here’s the start menu’s Accessories folder failing to populate on a 64-bit Windows Server 2008 system that’s out of paged pool:
Here you can see the system commit level, also displayed on Process Explorer’s System Information dialog, quickly rise as Notmyfault leaks large chunks of paged pool and hits the 2GB maximum on a 2GB 32-bit Windows Server 2008 system:
The reason that Windows doesn’t simply crash when pool is exhausted, even though the system is unusable, is that pool exhaustion can be a temporary condition caused by an extreme workload peak, after which pool is freed and the system returns to normal operation. When a driver (or the kernel) leaks pool, however, the condition is permanent and identifying the cause of the leak becomes important. That’s where the pool tags described at the beginning of the post come into play.
When you suspect a pool leak and the system is still able to launch additional applications, Poolmon, a tool in the Windows Driver Kit, shows you the number of allocations and outstanding bytes of allocation by type of pool and the tag passed into calls of ExAllocatePoolWithTag. Various hotkeys cause Poolmon to sort by different columns; to find the leaking allocation type, use either ‘b’ to sort by bytes or ‘d’ to sort by the difference between the number of allocations and frees. Here’s Poolmon running on a system where Notmyfault has leaked 14 allocations of about 100MB each:
After identifying the guilty tag in the left column, in this case ‘Leak’, the next step is finding the driver that’s using it. Since the tags are stored in the driver image, you can do that by scanning driver images for the tag in question. The Strings utility from Sysinternals dumps printable strings in the files you specify (that are by default a minimum of three characters in length), and since most device driver images are in the %Systemroot%\System32\Drivers directory, you can open a command prompt, change to that directory and execute “strings * | findstr <tag>”. After you’ve found a match, you can dump the driver’s version information with the Sysinternals Sigcheck utility. Here’s what that process looks like when looking for the driver using “Leak”:
If a system has crashed and you suspect that it’s due to pool exhaustion, load the crash dump file into the Windbg debugger, which is included in the Debugging Tools for Windows package, and use the !vm command to confirm it. Here’s the output of !vm on a system where Notmyfault has exhausted nonpaged pool:
Once you’ve confirmed a leak, use the !poolused command to get a view of pool usage by tag that’s similar to Poolmon’s. !poolused by default shows unsorted summary information, so specify 1 as the the option to sort by paged pool usage and 2 to sort by nonpaged pool usage:
Use Strings on the system where the dump came from to search for the driver using the tag that you find causing the problem.
So far in this blog series I’ve covered the most fundamental limits in Windows, including physical memory, virtual memory, paged and nonpaged pool. Next time I’ll talk about the limits for the number of processes and threads that Windows supports, which are limits that derive from these.
Someone at microsoft needs to code up a utility that automatically does the steps outlined in the tracing pool leaks section of this article, and returns the offending leaky driver/system file. People could then run the tool if they've been getting "insufficient system resources" errors, and that would help them fix their problem. Optionally they could upload the results to microsoft and microsoft could pass along such results to the software vendor in question or post advisories about those software products.
güzel davetiye örnekleri davetiye modelleri düğün davetiyeleri
Mark, awesome article, thank you very much!
Very nice article Mark.
I have a couple of questions,
1. Is it poosible to increase the Non Paged pool memory limit beyond 75% of RAM on Windows Server 2008 ?
2. If the answer to question 1 is yes, how can you do that ?
I have been trying to find the reason for a non paged pool leak on a windows server 2003 for quite some time now without much success. This happens randomly sometimes twice in a day. Here is the output from poolman when the non paged pool consumption was high. Is it possible to find from this output, the possible cause of the leak.
Memory:16771724K Avail:15716652K PageFlts: 76038 InRam Krnl: 2268K P:114036
Commit: 765728K Limit:33273512K Peak: 968688K Pool N:196572K P:1150
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
Irp Nonp 15114034 (3654) 14980131 (3578) 133903 62270528 ( 33776) 465
usbp Nonp 292346 ( 136) 159911 ( 136) 132435 41995160 ( 0) 317
Mdl Nonp 578590 ( 286) 307149 ( 293) 271441 34807680 ( 80) 128
HidU Nonp 291648 ( 136) 159541 ( 136) 132107 13738616 ( 0) 103
MmCm Nonp 2463 ( 0) 2305 ( 0) 158 8316176 ( 0) 52634
MFE0 Nonp 210464239 (3115) 210434889 (3080) 29350 7450848 ( 3568) 253
HidC Nonp 264459 ( 0) 132319 ( 0) 132140 3175736 ( 0) 24
LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384
Io Nonp 95706488 (65071) 95574219 (65071) 132269 2191368 ( 0) 1
HdCl Nonp 264402 ( 0) 132303 ( 0) 132099 2116896 ( 0) 16
TPLA Nonp 512 ( 0) 0 ( 0) 512 2097152 ( 0) 4096
TCPt Nonp 37255 ( 6) 37218 ( 6) 37 1664536 ( 0) 44987
File Nonp 13742294 ( 368) 13734330 ( 358) 7964 1213584 ( 1536) 152
VxSb Nonp 6 ( 0) 4 ( 0) 2 1171456 ( 0) 585728
Mm Nonp 1037 ( 0) 1025 ( 0) 12 1127512 ( 0) 93959
brcm Nonp 30 ( 0) 0 ( 0) 30 819184 ( 0) 27306
TCht Nonp 7652 ( 0) 7456 ( 58) 196 802816 (-237568) 4096
bRcm Nonp 8 ( 0) 0 ( 0) 8 685840 ( 0) 85730
naFF Nonp 299 ( 0) 1 ( 0) 298 656504 ( 0) 2203
Thre Nonp 155086 ( 7) 154036 ( 11) 1050 655200 ( -2496) 624
Pool Nonp 6 ( 0) 3 ( 0) 3 610304 ( 0) 203434
AfdC Nonp 149183 ( 30) 146111 ( 28) 3072 491520 ( 320) 160
LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256
Even Nonp 3895544 ( 354) 3889534 ( 350) 6010 291088 ( 192) 48
Devi Nonp 705 ( 0) 350 ( 0) 355 270104 ( 0) 760
NDpp Nonp 95 ( 0) 14 ( 0) 81 267520 ( 0) 3302
TCPC Nonp 11370 ( 6) 8215 ( 0) 3155 265952 ( 480) 84
Vad Nonp 2730397 ( 9) 2725265 ( 9) 5132 246336 ( 0) 48
Hal Nonp 194560 ( 341) 194536 ( 340) 24 199968 ( 368) 8332
Ntf0 Nonp 3 ( 0) 0 ( 0) 3 196608 ( 0) 65536
Ntfr Nonp 20653 ( 0) 17959 ( 0) 2694 173384 ( 0) 64
AfdE Nonp 148107 ( 30) 147506 ( 28) 601 168280 ( 560) 280
TCPc Nonp 243685 ( 53) 240390 ( 49) 3295 158160 ( 192) 48
Sema Nonp 6766238 (2360) 6763555 (2360) 2683 150648 ( 0) 56
MmCi Nonp 27458 ( 0) 26860 ( 0) 598 141344 ( 0) 236
Dump Nonp 9 ( 0) 1 ( 0) 8 135600 ( 0) 16950
RceT Nonp 1 ( 0) 0 ( 0) 1 131072 ( 0) 131072
Muta Nonp 999419 ( 189) 997544 ( 189) 1875 125856 ( 0) 67
CcSc Nonp 255342 ( 10) 254961 ( 9) 381 121920 ( 320) 320
MmCa Nonp 1262482 ( 11) 1261431 ( 10) 1051 107792 ( 112) 102
VadS Nonp 1697218 (1113) 1694239 (1117) 2979 95328 ( -128) 32
Vadl Nonp 2237198 ( 116) 2235790 ( 118) 1408 90112 ( -128) 64
NtFs Nonp 874028 ( 18) 872463 ( 5) 1565 72712 ( 520) 46
Ntfi Nonp 115369 ( 12) 115125 ( 0) 244 66368 ( 3264) 272
Lfsr Nonp 6 ( 0) 2 ( 0) 4 65536 ( 0) 16384
PooL Nonp 8 ( 0) 0 ( 0) 8 65536 ( 0) 8192
AmlH Nonp 1 ( 0) 0 ( 0) 1 65536 ( 0) 65536
There seems to be one kind of memory allocation that I haven't seen covered by your articles. How do virtualisation products such as Sun Virtual Box allocate their memory? It doesn't show up in any of the process-level memory counters in task manager / process explorer / perfmon. The only place it seems to be counted is in the overall "Cached" memory on the performance tab of task manager and the equivilent in other tools.
Looking at the Process Explorer in Windows Server 2003 guest in VMware. I am able to see the Paged Physical and Virtual values but the Paged Limit reads "no symbols" What gives?
You need to configure symbols in Process Explorer so it can access the symbols for the kernel image.
First, great post. I've used this information to try and trace what was using all the non-paged memory on my server 2003 system - it's an old system and still running SP0 so there could be many bugfixes which would explain my problem.
I was getting volsnap events stating that it couldn't allocate enough nonpaged memory to hold a bitmap for a snapshot of my C: drive.
volsnap uses 16KB blocks in it's bitmap, so on my 69GB drive, I calculated 552KB for the bitmap.
It took me a while to get the symbols for the system so that I could see the nonpage pool limit.
The system in question was using 100 MB of nonpaged pool memory. Procexp.exe gave a limit of 256 MB of nonpaged pool memory. More than enough free for the bitmap.
I was confused! I removed the shadowstorage for the C: drive from it's second volume D: so that it would just use the C: drive and the problem went away!
Do you think this is a bug in the SP0 VSS implementation or do you really think I was running out of nonpaged memory?
Great stuff, thanks.
I used to say an IT admin/support persons job is 50% Internet search, 50% compare settings side by side. But I am changing it to 33% Search, 33% Settings comparing and 34% Using tools created by Mark R.
Poolmon, Autoruns, Pagedefrag, ProcessExplorer, Disk2VHD, you name it...I love it!
Helpful article, thanks.
Also I would like to solicit opinion for the problem that I am debugging.
My software uses a Jungo driver for which I do not have a source code. At some point when I try to allocate memory it returns insufficient resource error.
I used a logger utility that came with the Jungo and extracted information that AllocateCommonBuffer fails which tells me that I am running out some memory resource somewhere. But I have plenty of physical memory ( 3 gb) which is available to me ( no special hardware hogging my memory making it unavailable to the system) . Since I am aware that kernel NonPaged pool is limited resource( 128 M on my system according to NonPagedPoolSize key in registery ) I have decided to use poolmon to see how I am running out of NonPaged memory and who is the culprit also this would tell me the Tag for the Jungo driver since I know the amount of memory I am allocating (8 Mb). Here is the strange part. According to the Poolmon no one is allocating 8Mb memory and there is plenty of Paged and NonPaged memory at the time AllocateCommonBuffer fails.
Now, I can hypothesize all kind of scenarios as to why the allocation fails. For example, maybe it needs something like NonPaged + Contiguous therefore it runs out of this flavor of memory or something. But I am not convinced by this kind of arguments since I do not have any way of verification of my hypothesis. I think the question I would like to ask is this : When I allocate 8Mb memory successfully with AllocateCommonBuffer where does the memory come from since I do not see anywhere Paged or NonPaged resources being reduced by this amount? Only indication of the allocation successful return of AllocateCommonBuffer and System PTE increased 2048 entries.
For those of you who have suggestions. I thank you for your time and attention.
I forgot to mention that I am using Windows 7.