This blog describes how to proactively manage and reduce the use of paged pool kernel memory that is consumed on a Windows Server 2003.
There is a nice blog about it by Tate (G.E.S. E.E.)
In the System event log, you might see:
Event Type: ErrorEvent Source: SrvEvent Category: NoneEvent ID: 2020Date: dateTime: timeUser: N/A Computer: server nameDescription: The server was unable to allocate from the system paged pool because the pool was empty.
This error in the System event log will be only shown if the Server service (srv.sys) detects it. Otherwise, you will not be flagged about a pool consumption issue.
Additionally, you might see the following symptoms:
-You might be able to ping the server
-You might not be able to open a network share
-You might not be able to login to the server via a Remote Desktop (RDP) session.
-Your server may stop accepting new user connections and you may receive this message "Windows cannot logon you because the profile cannot be loaded. Contact your network administrator. DETAIL - Insufficient system resources exist to complete the requested service."-You may receive the following error: "Not enough storage available to process this command".
-Your server might become unresponsive and/or hang.
Several factors may deplete the supply of paged pool kernel memory.
Taking poolmon logs at different time intervals may help you to understand which driver is consuming the paged pool kernel memory.
What is Paged pool memory?
Paged pool memory is a kernel memory that gets calculated during boot time.
The primary factor that determines the max values is the amount of physical memory the system has installed.
Here is a nice KB article that shows you the mathematical formula to calculate the kernel memory:
247904 How to Configure the Paged Address Pool and System Page Table Entry Memory Areas
What are the theoretical maximums of Paged Pool memory?
x86 (32-bit) Windows Server 2003 RTM/SP1/SP2:
Paged pool memory 650MB
x64 (64-bit) Windows Server 2003 RTM/SP1/SP2:Paged pool memory 128GB
Note: On the KB articles below, the theoretical maximum for 32-bit Windows Server 2003 show up as 470 MB which was true in the Windows 2000 timeframe but no longer correct in Windows Server 2003.
294418 Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
888732 Processor and memory capabilities of Windows XP Professional x64 Edition and of the x64-based versions of Windows Server 2003
889654 How to determine the appropriate page file size for 64-bit versions of Windows Server 2003 or Windows XP
What is the real world maximum of Paged Pool memory?
To determine the actual Maximum Paged and NonPaged pool memory of a specific computer without a memory dump use the following steps: - Download "Debugging Tools for Windows" http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
and install by selecting "Custom" during the installation to C:\DEBUGGERS folder. - Download Process Explorer from http://www.sysinternals.com/ntw2k/freeware/procexp.shtml
thena) Run procexp.exeb) Click "Options", "Configure Symbols" to configure "dbghelp.dll Path:" and "Symbols path:"C:\debuggers\dbghelp.dll
C:\WebSymbols;srv*C:\WebSymbols*http://msdl.microsoft.com/download/symbolsc) Click "View", "System Information" (this will automatically trigger the creation of c:\websymbols and the download of ntoskrnl.pdb symbol file from the Internet. The first time the symbol file is downloaded you will need to wait a minute or two)The maximum is displayed as "Page limit" in the "Process Explorer" tools from http://www.sysinternals.com
What Windows settings affect paged pool memory?
1. The /3gb switch in the boot.ini
For 32-bit Windows Server 2003 RTM/SP1/SP2 for those that require to use the /3gb switch, it will limit the amount of paged pool memory available.
32bit(x86) machines can address 2^32==4GB (gigabyte), Windows uses (by default) 2GB for applications and 2GB for kernel.
Of the 2GB for kernel there are other things we must fit in our 2GB such as Page Table Entries (PTEs)
So if we give 1GB to the Kernel and give 3GB to the User mode (applications), you are cutting the amount of kernel memory available.
316739 How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory ManagementPagedPoolSize (dword) =00000000 (hex) (default)
If you need to increase the amount of Paged Pool available to the system, change it to:
PagedPoolSize (dword) =FFFFFFFF (hex) (default)
Note: A reboot is required for this setting to take effect.
On some OEM servers, you will notice when you start Windows Server 2003, the SystemPages registry will be set to a different value than 0 (default)
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory ManagementSystemPages (dword) =0000c000 (hex)
This will increase the Free System Page Table Entries (Free System PTE’s) but will lower the amount of paged pool memory available.
Change the value to SystemPages (dword)=00000000 (hex)
On some OEM servers, you have the ability to HOT ADD MEMORY. If, hot-add memory is enabled, the operating system kernel must pre-allocate space to handle any future memory that may be added to the computer.
Therefore, kernel resources are allocated based on the capabilities of the computer instead of what is actually installed.
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory ManagementSystemPages=0000c000
Change the value to SystemPages=00000000
More detail at http://technet.microsoft.com/en-us/library/2b0d4c6e-92b7-410b-876b-367c4043b1c7.aspx(or search on DynamicMemory if unable to disable Hot-add Memory in the BIOS)
913568 Decreased performance, driver load failures, or system instability may occur on Hot Add Memory systems that are running Windows Server 2003
If the server is not a Terminal Server in Application Mode
You might want to use /basevideo in the boot.ini
Disable the OEM video driver
Use the VGA driver
-Poolmon.exe (currently supported)
Do you want the latest version, it’s included in the Windows Driver Kit (WDK, used to be known as Device Development Kit (DDK))
Note: Poolsnap.exe and PMLA.exe are older tools that will collect the data but it’s not as detailed as Poolmon.exe
-Debugging Tools for Windows
There are several tools that are available to monitor paged pool kernel memory on Windows Server 2003
- Task Manager, Handle count in Task Manager Processes tab for each process
Click on Start, Run, TaskMgr.exe
Click on the “Processes tab”
Click on "View"
Click on "Select Columns..."
Check the box for “Handle Count”
Click on Ok
A high handle count (larger than 7,000) may indicate the offending process without any further troubleshooting.
Stop the process or service if it is not necessary
In Task Manager, Performance tab, under Kernel Memory(K) observe “Paged” kernel memory usage.
If the problem continues, then go on to the next data gathering steps.
- Run the Microsoft Platform Support Reporting (MPSReports) utility Setup/Perf version
816819 Description of the Microsoft Platform Support Reporting Utility
Common root causes:
Large File copies
Quota management software
Windows Server 2003 RTM
Install Service Pack 2 for Windows Server 2003 + hotfixes
304101 Backup program is unsuccessful when you back up a large system volume
MmSt tag -
312362 Server is unable to allocate memory from the system paged pool
867686 The system runs out of paged pool, the system uses a high amount of memory, or the system stops responding when you use Volume Shadow Copy Service in Windows Server 2003
____tag - Volsnap.sys
886670 Event 2020 is logged and your Windows Server 2003-based file server stops responding
Strg tag - Srv.sys
891395 A memory leak occurs on a dedicated Windows Server 2003-based printer server
Gadd tag - Win32k.sys
Windows Server 2003 SP1
MmSt tag - ____.sys
833167 A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003
___ tag – Volsnap.sys
891004 You may experience a paged pool memory leak when many clients create, copy, and delete files on an NTFS partition in Windows Server 2003
891957 Update is available that fixes various Volume Shadow Copy Service issues in Windows Server 2003
MmSt tag – Volsnap.sys
894372 Support for Windows Server 2003 SP1 on Windows Storage Server 2003-based server appliances
906952 Event ID 2020 and event ID 1000 are logged and the registry size limit is exceeded on a computer that is running Terminal Services in Application Server mode and Windows Server 2003 or Windows 2000
____ tag - Winspool.drv
913648 A new Volume Shadow Copy Service update is now available that fixes various Volume Shadow Copy Service problems in Windows Server 2003
927435 A computer may run out of paged pool memory if the "Shadow Copies for Shared Folders" feature is enabled in Windows Server 2003
___tag - srv.sys
935649 Error message when you try to log on to a Windows Server 2003-based terminal server: "Windows cannot load the user's profile but has logged you on with the default profile for the system"
___tag - ____.sys
936087 A handle leak occurs in Windows Server 2003 when an application frequently maps and unmaps shared drives
___tag - ___.sys
940307 A Microsoft Windows Server 2003-based computer may stop responding if an application frequently uses the mount manager to query volume information
MntA tag – Mountmgr.sys
Windows Server 2003 R2
942586 The computer may stop responding if you enable the Quota function of the File Server Resource Manager (FSRM) component on a Windows Server 2003 R2-based computer
____tag - Quota.sys
Windows Server 2003 SP2
MmSt tag - __.sys
____ tag - ___.sys
927338 Exchange 2003 may perform slowly or may stop responding after you enable the "Message Tracking" log and the "RPC Calls" category
Toke tag - __.sys
NtFB tag - srv.sys
MmSt tag - _____
____tag - Basesrv.dll
FMfn tag - Quota.sys
951749 Stop error message on a terminal server that is running Windows Server 2003 SP1 or Windows Server 2003 SP2: "Stop 0x000000AB (SESSION_HAS_VALID_POOL_ON_EXIT)"
Gump tag – Win32k.sys
953325 A Windows Server 2003-based computer becomes unresponsive because the paged pool memory is exhausted when an application calls the GetFileAttributesEx and MoveFileEx functions on lots of files
NtfF tag - Ntfs.sys
957492 When you try to start a terminal session to a Windows Server 2003-based computer, the computer may be unresponsive
____tag - Oledlg.dll
961640 Nonpaged pool kernel memory leak on Windows 2003 with Multipathing solution installed
____ tag -
File Servers (including Cluster File Servers):
“If the pool, poolmon.exe indicate that the MmSt tag (Mm section object prototype PTEs) is the largest consumer and paged pool memory has been depleted or the system is logging error event 2020s, there is a large probability that there are a very large number of files that are open on the server.
By default, the Memory Manager tries to trim allocated paged pool memory when the system reaches 80 percent of the total paged pool.
Depending on the system configuration, a possible maximum paged pool memory on a computer can be 343MB and 80 percent of this number is 274MB.
If the Memory Manager is unable to trim fast enough to keep up with the demand, the event that is listed in the "Symptoms" section of this article may occur. By tuning the Memory Manager to start the trimming process earlier (for example, when it reaches 60 percent), it would be possible to keep up with the paged pool demand during sudden peak usage, and avoid running out of paged pool memory.”
912376 How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server
Toke tag - ____.sys
912480An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory
Toke tag - Store.exe
842355 How Services for Macintosh uses system resources in Windows Server 2003
292726 Registry Size Limit functionality has been removed from Windows Server 2003 and from Windows XP
In this example, we are going to use the Microsoft SysInternals NotMyFault.exe to mimic a Paged pool leak.
NotMyFault is available at:
Server: 32-bit (x86) Windows Server 2003 Service Pack 2 with all the security updates from Windows Update/Automatic Update.
Launch Task Manager and click on the Performance tab
Changing the “Leak/second:” from 1000 to 10000
Click on “Leak Paged”
Keep an eyes on Task Manager, once it reaches around 300-400 Mb of paged pool memory, stop the leak.
Click on Start, Run, CMD.exe
Type “poolmon.exe” without the quotation marks and then press Enter.
Notice that it’s not sorted by Type or Bytes.
Click on P to sort by Paged
Click on B to sort by Bytes
In here we could see that Paged pool kernel memory is at 312376K (305 MB).
The highest tag is named Leak
Notice that the Leak tag is allocating (Allocs) 26 and not freeing (Frees) any 0.
So you know the tag, now what? Who is using this tag?
According to KB 298102, you want to do the following:
Type “c:” without the quotation marks and then press Enter.
Type “c:\windows\system32\drivers” without the quotation marks and then press Enter.
Type “findstr.exe /m /l Leak *.sys” without the quotation marks and then press Enter.
Notice that the driver is named MyFault.sys
What if you had a live debugger attached such as LiveKd.exe or KD.exe –kl
LiveKD is available from the Microsoft SysInternals website http://technet.microsoft.com/en-us/sysinternals/bb795535.aspx
Type “LiveKd.exe –y srv*c:\websymbols*http://msdl.microsoft.com/download/symbols” without the quotation marks and then press Enter.
Type “.logopen c:\PagedPool_Output” without the quotation marks and then press Enter.
Type “!vm” without the quotation marks and then press Enter.
This is what the O.S. might look like when it’s running ‘normally’.
This is what the O.S. might look like when it’s running with a leak.
Type “!poolused 4” without the quotation marks and then press Enter.
This is what the paged pool tags might look when it’s running ‘normally’.
This is what the paged pool tags might look when it’s running with a ‘leak’.