Pushing the Limits of Windows: Paged and Nonpaged Pool

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Crawford — Fri, 27 Mar 2009 09:15:11 GMT

I get delayed write failures every once in a while, and usually the cause for me is a glitch in the hibernation process. I've also discovered that you're almost guaranteed one if you hibernate while a USB-connected NTFS volume is mounted (even if you resume with the same drive connected to the same port). I've also gotten it once or twice by breaking my file server while writing to files. Fortunately, it's never actually caused any data loss for me.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Carl Klemmer — Fri, 27 Mar 2009 16:27:02 GMT

Great article and great series, Mark. In 15 years of managing NT servers I've banged up against physical and virtual memory limitations plenty of times, but I've never had an issue that lead me to these pools. Is exhausting them a common problem? What kind of software most commonly leads to issues with these pools?

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

markrussinovich — Fri, 27 Mar 2009 16:46:28 GMT

@Carl: Thanks for the feedback. I've only run into pool leaks a few times in the last 10 years. Device drivers of any kind - hardware, file system filter, antivirus - can be guilty of it.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Jonathan — Fri, 27 Mar 2009 17:01:38 GMT

Server software that keeps many TCP/IP connections in the air can easily exhaust Non-paged pool. That happens on heavily-loaded ISA Servers for example - and is the reason why the next version of it is 64-bit only.

Crawford: Maybe it's because NTFS is always updated the last-access time of files? I think you can disable it with fsutil (fsutil behavior query disablelastaccess), though I don't know whether it's per-machine or per-volume.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

David — Fri, 27 Mar 2009 17:51:45 GMT

Another great article. I learn so much, I recommend these articles to everyone.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

LookingForSolutions — Fri, 27 Mar 2009 23:21:19 GMT

Earlier this week, I was troubleshooting a customer's server where IIS6 had stopped accepting new connections. After checking the entries in the SYSTEM32\LOGFILES\HTTPERR logs, I searched on Google and found out it was due to available non-paged pool memory being less than 20MB, at which point IIS6 stops accepting new connections. Rebooting the server fixed the problem for now. When/if the problem returns, I'll try the troubleshooting steps from this excellent blogpost!

FYI, the related KB's:

http://support.microsoft.com/kb/934878

http://support.microsoft.com/kb/820729

"The kernel NonPagedPool memory has dropped below 20MB and http.sys has stopped receiving new connections"

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Carl — Sat, 28 Mar 2009 03:03:57 GMT

It is my understanding that storing Outlook PST files on network drives can cause nonpaged memory pool issues on 32-bit file Windows 2003 file servers.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Miral — Mon, 30 Mar 2009 10:06:55 GMT

My 32-bit Vista machine gets *very* flaky when the non-paged pool hits 1GB or higher. (On a couple of occasions it truncated files that were being written at the time.)

In case anyone is interested: the leak was in the TdxA pool and was (possibly indirectly) caused by a poorly-written firewall/virusscan program. (I tracked this down using poolmon a few months ago.)

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

David — Mon, 30 Mar 2009 23:27:42 GMT

I've gotten the "Delayed Write Failed" error most of the times I've tried to use ntbackup on Windows Server 2003. Extremely unacceptable for such an important piece of software.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Nick — Mon, 30 Mar 2009 23:46:49 GMT

Mark -

The timing of your articles never ceases to amaze. Just two weeks ago I was fighting with a server experiencing pool exhaustion, largely due to someone placing the /3GB switch into it's boot options. Process Explorer was (yet again) an invaluable tool to solve the problem.

Thanks again for the great articles and utilities.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Anonymous Coward — Tue, 31 Mar 2009 14:04:43 GMT

Great article, Mark. I'm always amazed by the way you tackle these seemingly complicated things in such a straightforward manner.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Chad — Tue, 31 Mar 2009 20:52:22 GMT

Mark, great article & very timely for me as I'm getting 2019's every 4.5 days & I'm assuming it's 1 of many new print drivers that were recently loaded onto the server. I've begun using Poolman to try & diagnose which driver causes the leak, unfortunately when I attempt to use the poolmon -c switch to generate a "localtag.txt get the following error;

"Poolmon: No localtag.txt in current directory

Unable to load msvcr70.dll/msvcp70.dll, cannot create local tag file"

So I receive a number of "unknown driver" tags.....any ideas on how to help me further isolate this driver leak or get poolmon to successfully create a localtag.txt file?

Also, I fear it may be a event driven leak as I've been watching the nonpaged limit in process explorer for a couple of days & it's a very steady & not climbing.

Thanks again, the article was very straight-forward & helpful.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Clovis — Tue, 31 Mar 2009 23:03:32 GMT

Why is MPEG2 playback disabled if debugging is enabled? How do you debug MPEG2 drivers?

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

dirbase — Tue, 31 Mar 2009 23:32:06 GMT

Mark,

Thanks for this post. This reminds me of a topic I started in the sysinternals forum back in January http://forum.sysinternals.com/forum_posts.asp?TID=17486 where I showed that on several XP (SP2/SP3) systems with NTFS partitions searching for a file via explorer search can take up a large portion of both page and non paged pools, which are not really released after the search has ended.This is presumably due to NTFS.sys.

I'm still wondering if it's an expected behaviour. What do you think?

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

dirbase — Wed, 01 Apr 2009 14:19:03 GMT

Concerning my comment above about high pool usage following an explorer file search, since the MmSt paged pool tag is on top of the bytes list, I tend to think now that the basic reason is that the many searched files have been cached and the increased MmSt allocations correspond to their PTEs. This would explain why a repeat file search does not increase the pool usage.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

wtp — Thu, 02 Apr 2009 19:43:20 GMT

Mark:

We are getting back to the point IBM did in 1964 when it built the Stretch for the AEC - long words, meaning longer addresses and better control of memory (though I wish the next I7 or AMD similar would be a 100% 64-bit x86 rather than the 32 bits-with-bag-on-the-side models we've been seeing.

Once you *can* address 4 TB of cheap RAM, then setting the various amounts allowed various jobs, as we used to say (a "job" is an active application) or daemons (read "TSRs" or inactive pre-loads), we could program (we did it with 1s and with 0s, and sometimes we ran out of 1s!)* without sloppy coding relying on stacks(or, for that matter, C, a language designed as a 'portable' single-pass compiler for two reasons 1) because compiling a job on a PDP-7 multi-pass device meant 'ok, let's start it up and break for lunch, then see if we had any errors) and mainframes were run, in part, by bean counters, who gave you so many 'chocolate' dollars in time to get your work done) and move back to writing all programs large and small in a 'smart' Assembler, yep in assembly language,** that would allow the construction or use of repeatedly-used optimized structures as "pseudo ops"***

Which naturally brings up the question of why, guy, didn't you write all your sisinternals to run under 64 bit extensions?

wtp

*Tom Smith - filk writer "when I was a Boy"

**See Steve Gibson on assembler GRC.com

***Of course, that was in the days before microcode and the need to build 1 circuit per class of instructions, The stunningly simple code for DEC's 12-bit desktop machines fit on a 3x5 card, the absolutely awesome code for the PDP-10 room-filling "big iron" fit on an 8.5x11" sheet. Remember the study on VAXen and PDP-11s that produced RISC architecture? A lot of x86 code might also prove slow and redundant if given the same test, cutting down 3 vol.s of code manuals to , at most, 1.(wtp's opinion, anyway) This might make for a physically larger Windows OS that ran fast enough to reduce CPU speed to MHz again - not 4-core hyperthreaded 3 GHz.+.And the assembled code might be way smaller than anything the most efficient C (+,++,# or.net) could produce.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Chris — Thu, 02 Apr 2009 20:20:08 GMT

The thing I don't understand is this:

When I install Windows (any version) on any of my machines, a quick look at the Task Manager shows me that a significant amount of memory is being paged.

but when I install Ubuntu on that same hardware, the page drive is zero

what's the deal with that?

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Alex Ionescu — Fri, 03 Apr 2009 05:12:37 GMT

@Clovis It's disabled because the Protected Media Path (PMP) could be theoretically subverted by using the debugger APIs (which allow you to do things like patch the kernel from user-mode). When debugging PMP components, there's a special internal build which keeps the components running but allows debugging (which is internal to Microsoft and their DRM partners) -- this isn't something a user can do.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Friday — Sun, 05 Apr 2009 22:37:15 GMT

@Clovis: If they told you they'd have to sue you. :)

Oops, too late. :D

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

MPyle — Tue, 14 Apr 2009 22:05:00 GMT

Great article. Very clear.

But given that how does, Task-Manager define a per process count of Non-Paged-Pool memory usage?

I have a problem where we see a per process memory-pool rise.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

GS — Wed, 15 Apr 2009 18:29:43 GMT

So what do you if you find more then 1 driver matching symbol string? How do I identify which one is actually causing this?

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Timothy Davis — Thu, 16 Apr 2009 05:45:19 GMT

@GS and @Chad

You can use driver verifier to track pool usage by a driver. See [http://msdn.microsoft.com/en-us/library/ms792856.aspx] for more information

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

RW — Fri, 17 Apr 2009 20:50:41 GMT

Great article. I have a handle leak that accumulates in explorer.exe on my 4GB WinXP sp3 laptop. This occurs multiple times per week and is happening as I type this.

PoolMon shows:

Even Nonp 12848369 ( 427) 11400450 ( 430) 1447919 69507456 ( -144) 48

Total Handles 1,471,236 which of course means Windows is beginning to act up.

The techniques so far have not led me to tell what is leaking the handles. The Even tag I believes it is unknown.

Explorer.exe handles total is 1,437,260

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

LookingForSolutions — Fri, 17 Apr 2009 23:30:55 GMT

Sure enough, the problem which I mentioned in my previous post has returned. This time I used poolmon to track the culprit. It turned out to be a faulty version of the Microsoft MPIO driver (version 1.21 leaks nonpaged pool memory -> http://support.microsoft.com/kb/961640). This issue was happening on one (Windows Server 2003 x86) cluster node, while the other cluster node did not exhibit any problems. The other cluster node turned out to be using a different version of the MPIO driver which didn't have the memory leak bug.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

reporter — Thu, 23 Apr 2009 08:54:34 GMT

Symantec Antivirus software is one of the software which i have seen allocating humungous amount of paged pool (almost ~70MB). it's surprising how they got away with this so far given the huge instll base (both retail and corporate) they have.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

TechFreak — Thu, 23 Apr 2009 16:27:09 GMT

Good information, but nothing creative here.

For a deveoper like me, I rather liked the creativity and ingenuity of the idea in using pure Win32 and C++ to create Terabyte sized arrays on Win32 machines (without tweaking anything that too):

http://blogs.msdn.com/gpalem/archive/2008/06/05/huge-arrays-with-file-mapping.aspx

Thats what I call real pushing of limits. Whoever did it, I found it quite inspiring. Way to go Boys.

-AMiCo

NationalSemiConductors.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Andrew Fidel — Thu, 23 Apr 2009 18:50:21 GMT

Easier than installing the DDK just to get poolmon is to install the support tools for your OS. For XP SP2 it's available at http://www.microsoft.com/downloads/details.aspx?FamilyId=49AE8576-9BB9-4126-9761-BA8011FABF38

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Travis — Thu, 30 Apr 2009 18:50:23 GMT

Nice job. By far the best, most comprehensive guide on the subject.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

elbutre — Fri, 22 May 2009 18:12:25 GMT

i am running a windows 2008 64 bit server with 10gb of ram and exchange 2007 sp1.

the exchange monitoring tool on the server says

Paged pool memory on server uk-mailbox is over the error threshold of 220 MB. Current value: 234 MB.

The maximum non-paged pool memory on server uk-mailbox is over the warning threshold of 100 MB. This may cause system instability. Current value: 110 MB.

but having ran the process explorer tool it shows that it is nowhere near the limit of 128gb for 64bit 2008 server systems

i also wanted to look at poolmon to see if any of the applications are leaking but i cannot see that poolmon is supported on windows 2008 64 bit server. is there a version for this OS?

and should i be worried about the paged and non-paged pool memory warning?

there are no event id's in the system log and the server seems to be running fine. no crashes (yet)

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

Remko Weijnen — Sun, 31 May 2009 23:02:45 GMT

Mark,

How about the other important resources (eg System PTE's, Session View Space, Sesion Paged Pool, Desktop Heap), it would be great if you can discuss these as well!

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

John — Fri, 03 Jul 2009 22:14:04 GMT

Someone at microsoft needs to code up a utility that automatically does the steps outlined in the tracing pool leaks section of this article, and returns the offending leaky driver/system file. People could then run the tool if they've been getting "insufficient system resources" errors, and that would help them fix their problem. Optionally they could upload the results to microsoft and microsoft could pass along such results to the software vendor in question or post advisories about those software products.

re: Pushing the Limits of Windows: Paged and Nonpaged Pool

davetiye — Mon, 02 Nov 2009 12:49:37 GMT

güzel davetiye örnekleri davetiye modelleri düğün davetiyeleri