INTRODUCTION

This blog describes how to proactively manage and reduce the use of paged pool kernel memory that is consumed on a Windows Server 2003.

There is a nice blog about it by Tate (G.E.S. E.E.)

http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx

SYMPTOMS

In the System event log, you might see:

Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Date: date
Time: time
User: N/A
Computer: server name
Description: The server was unable to allocate from the system paged pool because the pool was empty.

This error in the System event log will be only shown if the Server service (srv.sys) detects it. Otherwise, you will not be flagged about a pool consumption issue.

Additionally, you might see the following symptoms:

-You might be able to ping the server

-You might not be able to open a network share

-You might not be able to login to the server via a Remote Desktop (RDP) session.

-Your server may stop accepting new user connections and you may receive this message "Windows cannot logon you because the profile cannot be loaded. Contact your network administrator. DETAIL - Insufficient system resources exist to complete the requested service."
-You may receive the following error: "Not enough storage available to process this command".

-Your server might become unresponsive and/or hang.

CAUSE

Several factors may deplete the supply of paged pool kernel memory.

Taking poolmon logs at different time intervals may help you to understand which driver is consuming the paged pool kernel memory.

What is Paged pool memory?

Paged pool memory is a kernel memory that gets calculated during boot time.

The primary factor that determines the max values is the amount of physical memory the
system has installed.

Here is a nice KB article that shows you the mathematical formula to calculate the kernel memory:

247904 How to Configure the Paged Address Pool and System Page Table Entry Memory Areas

http://support.microsoft.com/?id=247904

What are the theoretical maximums of Paged Pool memory?

x86 (32-bit) Windows Server 2003 RTM/SP1/SP2:

Paged pool memory 650MB

x64 (64-bit) Windows Server 2003 RTM/SP1/SP2:
Paged pool memory 128GB

Note: On the KB articles below, the theoretical maximum for 32-bit Windows Server 2003 show up as 470 MB which was true in the Windows 2000 timeframe but no longer correct in Windows Server 2003.

294418 Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003

http://support.microsoft.com/?id=294418

888732 Processor and memory capabilities of Windows XP Professional x64 Edition and of the x64-based versions of Windows Server 2003

http://support.microsoft.com/?id=888732

889654 How to determine the appropriate page file size for 64-bit versions of Windows Server 2003 or Windows XP

http://support.microsoft.com/?id=889654

What is the real world maximum of Paged Pool memory?

To determine the actual Maximum Paged and NonPaged pool memory of a specific
computer without a memory dump use the following steps:
- Download "Debugging Tools for Windows"
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

http://www.microsoft.com/whdc/devtools/debugging/installx64.mspx

and install by selecting "Custom" during the installation to C:\DEBUGGERS folder.
- Download Process Explorer from
http://www.sysinternals.com/ntw2k/freeware/procexp.shtml

then
a) Run procexp.exe
b) Click "Options", "Configure Symbols" to configure "dbghelp.dll Path:" and
"Symbols path:"
C:\debuggers\dbghelp.dll

 

to


C:\WebSymbols;srv*C:\WebSymbols*http://msdl.microsoft.com/download/symbols
c) Click "View", "System Information" (this will automatically trigger the creation of c:\websymbols and the download of ntoskrnl.pdb symbol file from the Internet. The first time the symbol file is downloaded you will need to wait a minute or two)
The maximum is displayed as "Page limit" in the "Process Explorer" tools from http://www.sysinternals.com

What Windows settings affect paged pool memory?

1. The /3gb switch in the boot.ini

For 32-bit Windows Server 2003 RTM/SP1/SP2 for those that require to use the /3gb switch, it will limit the amount of paged pool memory available.

32bit(x86) machines can address 2^32==4GB (gigabyte), Windows uses (by default) 2GB for applications and 2GB for kernel.

Of the 2GB for kernel there are other things we must fit in our 2GB such as Page Table Entries (PTEs)

So if we give 1GB to the Kernel and give 3GB to the User mode (applications), you are cutting the amount of kernel memory available.

316739 How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB

http://support.microsoft.com/?id=316739

2. PagedPoolSize

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
PagedPoolSize (dword) =00000000 (hex) (default)

If you need to increase the amount of Paged Pool available to the system, change it to:

PagedPoolSize (dword) =FFFFFFFF (hex) (default)

Note: A reboot is required for this setting to take effect.

3. SystemPages

On some OEM servers, you will notice when you start Windows Server 2003, the SystemPages registry will be set to a different value than 0 (default)

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
SystemPages (dword) =0000c000 (hex)

This will increase the Free System Page Table Entries (Free System PTE’s) but will lower the amount of paged pool memory available.

Change the value to SystemPages (dword)=00000000 (hex)

Note: A reboot is required for this setting to take effect.

4. DynamicMemory

On some OEM servers, you have the ability to HOT ADD MEMORY. If, hot-add memory is enabled, the operating system kernel must pre-allocate space to handle any future memory that may be added to the computer.

Therefore, kernel resources are allocated based on the capabilities of the computer instead of what is actually installed.

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
SystemPages=0000c000

This will increase the Free System Page Table Entries (Free System PTE’s) but will lower the amount of paged pool memory available.

Change the value to SystemPages=00000000

Note: A reboot is required for this setting to take effect.

More detail at
http://technet.microsoft.com/en-us/library/2b0d4c6e-92b7-410b-876b-367c4043b1c7.aspx
(or search on DynamicMemory if unable to disable Hot-add Memory in the BIOS)

913568 Decreased performance, driver load failures, or system instability may occur on Hot Add Memory systems that are running Windows Server 2003

http://support.microsoft.com/?id=913568

  1. Video driver

If the server is not a Terminal Server in Application Mode

You might want to use /basevideo in the boot.ini

Disable the OEM video driver

Use the VGA driver

 

Tools needed

-Poolmon.exe (currently supported)

Do you want the latest version, it’s included in the Windows Driver Kit (WDK, used to be known as Device Development Kit (DDK))

http://msdn.microsoft.com/en-us/library/aa469207.aspx

Note:  Poolsnap.exe and PMLA.exe are older tools that will collect the data but it’s not as detailed as Poolmon.exe

-Perfmon, remotely.

-Debugging Tools for Windows

Troubleshooting

There are several tools that are available to monitor paged pool kernel memory on Windows Server 2003

- Task Manager, Handle count in Task Manager Processes tab for each process

Click on Start, Run, TaskMgr.exe

Click on the “Processes tab”

Click on "View"

Click on "Select Columns..."

Check the box for “Handle Count”

Click on Ok

A high handle count (larger than 7,000) may indicate the offending process without any further troubleshooting.

Stop the process or service if it is not necessary

In Task Manager, Performance tab, under Kernel Memory(K) observe “Paged” kernel memory usage.

If the problem continues, then go on to the next data gathering steps.

- Run the Microsoft Platform Support Reporting (MPSReports) utility Setup/Perf version

816819 Description of the Microsoft Platform Support Reporting Utility

http://support.microsoft.com/?id=816819

 

Common root causes:

---------------------------

Large File copies

Antivirus software

Backup software

Quota management software

 

Hotfixes:

------------

Windows Server 2003 RTM

Preferred:

Install Service Pack 2 for Windows Server 2003 + hotfixes

Work-around:

304101 Backup program is unsuccessful when you back up a large system volume

http://support.microsoft.com/?id=304101

MmSt tag -

312362 Server is unable to allocate memory from the system paged pool

http://support.microsoft.com/?id=312362

MmSt tag -

867686 The system runs out of paged pool, the system uses a high amount of memory, or the system stops responding when you use Volume Shadow Copy Service in Windows Server 2003

http://support.microsoft.com/?id=867686

____tag - Volsnap.sys

886670 Event 2020 is logged and your Windows Server 2003-based file server stops responding

http://support.microsoft.com/?id=886670

Strg tag - Srv.sys

891395 A memory leak occurs on a dedicated Windows Server 2003-based printer server

http://support.microsoft.com/?id=831395

Gadd tag - Win32k.sys

Windows Server 2003 SP1

Preferred:

Install Service Pack 2 for Windows Server 2003 + hotfixes

Work-around:

304101 Backup program is unsuccessful when you back up a large system volume

http://support.microsoft.com/?id=304101

MmSt tag - ____.sys

312362 Server is unable to allocate memory from the system paged pool

http://support.microsoft.com/?id=312362

MmSt tag - ____.sys

833167 A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003

http://support.microsoft.com/?id=833167

___ tag – Volsnap.sys

891004 You may experience a paged pool memory leak when many clients create, copy, and delete files on an NTFS partition in Windows Server 2003

http://support.microsoft.com/?id=891004

____tag-Ntfs.sys

891957 Update is available that fixes various Volume Shadow Copy Service issues in Windows Server 2003

http://support.microsoft.com/?id=891957

MmSt tag – Volsnap.sys

894372 Support for Windows Server 2003 SP1 on Windows Storage Server 2003-based server appliances

http://support.microsoft.com/?id=894372

MmSt tag - ____.sys

906952 Event ID 2020 and event ID 1000 are logged and the registry size limit is exceeded on a computer that is running Terminal Services in Application Server mode and Windows Server 2003 or Windows 2000

http://support.microsoft.com/?id=906952

____ tag - Winspool.drv

913648 A new Volume Shadow Copy Service update is now available that fixes various Volume Shadow Copy Service problems in Windows Server 2003

http://support.microsoft.com/?id=913648

___ tag – Volsnap.sys

927435 A computer may run out of paged pool memory if the "Shadow Copies for Shared Folders" feature is enabled in Windows Server 2003

http://support.microsoft.com/?id=927435

___tag - srv.sys

935649 Error message when you try to log on to a Windows Server 2003-based terminal server: "Windows cannot load the user's profile but has logged you on with the default profile for the system"

http://support.microsoft.com/?id=935649

___tag - ____.sys

936087 A handle leak occurs in Windows Server 2003 when an application frequently maps and unmaps shared drives

http://support.microsoft.com/?id=936087

___tag - ___.sys

940307 A Microsoft Windows Server 2003-based computer may stop responding if an application frequently uses the mount manager to query volume information

http://support.microsoft.com/?id=940307

MntA tag – Mountmgr.sys

Windows Server 2003 R2

Preferred:

Install Service Pack 2 for Windows Server 2003 + hotfixes

Work-around:

942586 The computer may stop responding if you enable the Quota function of the File Server Resource Manager (FSRM) component on a Windows Server 2003 R2-based computer

http://support.microsoft.com/?id=942586

____tag - Quota.sys

Windows Server 2003 SP2

304101 Backup program is unsuccessful when you back up a large system volume

http://support.microsoft.com/?id=304101

MmSt tag - __.sys

312362 Server is unable to allocate memory from the system paged pool

http://support.microsoft.com/?id=312362

MmSt tag - __.sys

913648 A new Volume Shadow Copy Service update is now available that fixes various Volume Shadow Copy Service problems in Windows Server 2003

http://support.microsoft.com/?id=913648

____ tag - ___.sys

927338 Exchange 2003 may perform slowly or may stop responding after you enable the "Message Tracking" log and the "RPC Calls" category

http://support.microsoft.com/?id=927338

Toke tag - __.sys

927435 A computer may run out of paged pool memory if the "Shadow Copies for Shared Folders" feature is enabled in Windows Server 2003

http://support.microsoft.com/?id=927435

NtFB tag - srv.sys

935649 Error message when you try to log on to a Windows Server 2003-based terminal server: "Windows cannot load the user's profile but has logged you on with the default profile for the system"

http://support.microsoft.com/?id=935649

MmSt tag - _____

936087 A handle leak occurs in Windows Server 2003 when an application frequently maps and unmaps shared drives

http://support.microsoft.com/?id=936087

____tag - Basesrv.dll

940307 A Microsoft Windows Server 2003-based computer may stop responding if an application frequently uses the mount manager to query volume information

http://support.microsoft.com/?id=940307

MntA tag – Mountmgr.sys

942586 The computer may stop responding if you enable the Quota function of the File Server Resource Manager (FSRM) component on a Windows Server 2003 R2-based computer

http://support.microsoft.com/?id=942586

FMfn tag - Quota.sys

951749 Stop error message on a terminal server that is running Windows Server 2003 SP1 or Windows Server 2003 SP2: "Stop 0x000000AB (SESSION_HAS_VALID_POOL_ON_EXIT)"

http://support.microsoft.com/?id=951749

Gump tag – Win32k.sys

953325 A Windows Server 2003-based computer becomes unresponsive because the paged pool memory is exhausted when an application calls the GetFileAttributesEx and MoveFileEx functions on lots of files

http://support.microsoft.com/?id=953325

NtfF tag - Ntfs.sys

957492 When you try to start a terminal session to a Windows Server 2003-based computer, the computer may be unresponsive

http://support.microsoft.com/?id=957492

____tag - Oledlg.dll

961640 Nonpaged pool kernel memory leak on Windows 2003 with Multipathing solution installed

http://support.microsoft.com/?id=961640

____ tag -

File Servers (including Cluster File Servers):

------------------------------------------------------

“If the pool, poolmon.exe indicate that the MmSt tag (Mm section object prototype PTEs) is the largest consumer and paged pool memory has been depleted or the system is logging error event 2020s, there is a large probability that there are a very large number of files that are open on the server.

By default, the Memory Manager tries to trim allocated paged pool memory when the system reaches 80 percent of the total paged pool.

Depending on the system configuration, a possible maximum paged pool memory on a computer can be 343MB and 80 percent of this number is 274MB.

If the Memory Manager is unable to trim fast enough to keep up with the demand, the event that is listed in the "Symptoms" section of this article may occur. By tuning the Memory Manager to start the trimming process earlier (for example, when it reaches 60 percent), it would be possible to keep up with the paged pool demand during sudden peak usage, and avoid running out of paged pool memory.”

Terminal Servers:

----------------------

N/A

Exchange Server:

----------------------

912376 How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/?id=912376

Toke tag - ____.sys

912480An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory

http://support.microsoft.com/?id=912480

Toke tag - Store.exe

SQL:

-------

N/A

Macintosh clients:

------------------------

842355 How Services for Macintosh uses system resources in Windows Server 2003

http://support.microsoft.com/?id=842355

MmSt tag - ____.sys

Registry:

------------

292726 Registry Size Limit functionality has been removed from Windows Server 2003 and from Windows XP

http://support.microsoft.com/?id=292726

 

File Servers (including Cluster File Servers):

------------------------------------------------------

“If the pool, poolmon.exe indicate that the MmSt tag (Mm section object prototype PTEs) is the largest consumer and paged pool memory has been depleted or the system is logging error event 2020s, there is a large probability that there are a very large number of files that are open on the server.

By default, the Memory Manager tries to trim allocated paged pool memory when the system reaches 80 percent of the total paged pool.

Depending on the system configuration, a possible maximum paged pool memory on a computer can be 343MB and 80 percent of this number is 274MB.

If the Memory Manager is unable to trim fast enough to keep up with the demand, the event that is listed in the "Symptoms" section of this article may occur. By tuning the Memory Manager to start the trimming process earlier (for example, when it reaches 60 percent), it would be possible to keep up with the paged pool demand during sudden peak usage, and avoid running out of paged pool memory.”

Terminal Servers:

----------------------

N/A

Exchange Server:

----------------------

912376 How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/?id=912376

Toke tag - ____.sys

912480An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory

http://support.microsoft.com/?id=912480

Toke tag - Store.exe

SQL:

-------

N/A

Macintosh clients:

------------------------

842355 How Services for Macintosh uses system resources in Windows Server 2003

http://support.microsoft.com/?id=842355

MmSt tag - ____.sys

Registry:

------------

292726 Registry Size Limit functionality has been removed from Windows Server 2003 and from Windows XP

http://support.microsoft.com/?id=292726

 

Example:

------------

In this example, we are going to use the Microsoft SysInternals NotMyFault.exe to mimic a Paged pool leak.

NotMyFault is available at:

http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

Server: 32-bit (x86) Windows Server 2003 Service Pack 2 with all the security updates from Windows Update/Automatic Update.

Launch Task Manager and click on the Performance tab

Launching NotMyFault.exe

Changing the “Leak/second:” from 1000 to 10000

Click on “Leak Paged”

Keep an eyes on Task Manager, once it reaches around 300-400 Mb of paged pool memory, stop the leak.

 

Click on Start, Run, CMD.exe

Type “poolmon.exe” without the quotation marks and then press Enter.

 

Notice that it’s not sorted by Type or Bytes.

Click on P to sort by Paged

Click on B to sort by Bytes

 

In here we could see that Paged pool kernel memory is at 312376K (305 MB).

The highest tag is named Leak

Notice that the Leak tag is allocating (Allocs) 26 and not freeing (Frees) any 0.

So you know the tag, now what? Who is using this tag?

According to KB 298102, you want to do the following:

Click on Start, Run, CMD.exe

Type “c:” without the quotation marks and then press Enter.

Type “c:\windows\system32\drivers” without the quotation marks and then press Enter.

Type “findstr.exe /m /l Leak *.sys” without the quotation marks and then press Enter.

 

Notice that the driver is named MyFault.sys

What if you had a live debugger attached such as LiveKd.exe or KD.exe –kl

LiveKD is available from the Microsoft SysInternals website http://technet.microsoft.com/en-us/sysinternals/bb795535.aspx

Click on Start, Run, CMD.exe

Type “LiveKd.exe –y srv*c:\websymbols*http://msdl.microsoft.com/download/symbols” without the quotation marks and then press Enter.

Type “.logopen c:\PagedPool_Output” without the quotation marks and then press Enter.

Type “!vm” without the quotation marks and then press Enter.

This is what the O.S. might look like when it’s running ‘normally’.

This is what the O.S. might look like when it’s running with a leak.

Type “!poolused 4” without the quotation marks and then press Enter.

This is what the paged pool tags might look when it’s running ‘normally’.

This is what the paged pool tags might look when it’s running with a ‘leak’.