Getting to know the MmSt Pool Tag

Getting to know the MmSt Pool Tag

  • Comments 1
  • Likes

Hello readers,

  We have briefly discussed the MmSt pool tag before, but only tangentially. This post is to go much further in depth to the use of this tag and common issues we see related to it.

What is the MmSt pool tag?

The MmSt pool tag is used by the Memory Manager when reserving memory for section prototype PTEs. A prototype PTE is a structure that maps the physical location of a page or set of pages to a memory mapped file. The MmSt allocations reside in Paged Pool memory. MmSt allocations are most commonly seen when the system is under heavy file I/O and has many open files mapped into the memory cache. The Cache Manager should reduce its MmSt allocations when paged pool memory pressure occurs, but this may not always happen quickly enough to prevent some issues.

 

Common Symptoms

The symptoms for issues dealing with high MmSt paged pool usage are common to most other paged pool depletion issues including:

  • Server hangs
  • Event ID 2020 – Paged Pool depletion
  • Event ID 333 - An I/O operation initiated by the Registry failed
  • Users are unable to RDP or access file shares on the server
  • Backups or large file copies fail with insufficient resources
    • Error 1450: Insufficient resources exist to complete the requested server
    • Error 1130: Not enough server storage is available to process this command
  • SQL backups or consistency checks performing heavy file I/O will fail to complete

 

Most Common Root Causes for high MmSt pool usage

  • The Memory Manager cannot trim paged pool memory quickly enough before allocations start failing
  • Heavy file I/O because too many files or large files are opened simultaneously
  • Outlook .PST/Lotus Notes .NSF files being stored on the server and opened over the network
  • VSS (Volume Shadow Copy) snapshots aren’t dismounting
  • Large $MFT due to fragmentation or many files on the volume

 

Identifying the MmSt pool tag as the issue

You will need to use a tool such as Poolmon, Process Explorer, a memory dump, or a LiveKD dump to first confirm that you have a paged pool issue, and second to determine if MmSt is the tag consuming most of the paged pool memory. If it is in fact the MmSt tag causing the server to run out of paged pool memory then you will typically want to collect data to confirm this and find root cause.

 

  • Poolmon logging
    • Indicates total paged pool usage and which tags are consuming the most
    • How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
    • http://support.microsoft.com/kb/177415
  • Performance Monitor log
    • Confirm high pool usage
    • Find the trend in pool memory
    • Determine if the trend is related to high file I/O
  • Complete or Kernel memory dump
    • Check if the system failed to allocate paged pool memory and which tags were using the most paged pool memory
  • Process Explorer
    • Check the limits of the paged and nonpaged pool on the machine
  • RamMap
  • LiveKD
    • LiveKD can be used to collect a memory dump without bugchecking the machine

 

 

Poolmon Logging

Poolmon will be one of the primary tools you use to determine which pool tag is consuming most of the memory. A poolmon log indicates how much pool memory is being used by all paged and nonpaged pool memory tags. The poolmon log should run until the symptoms occur, and should end up looking something like this:

Pool Used:

                NonPaged                       Paged

Tag       Allocs     Used               Allocs     Used

MmSt      0             0                   8684   293147416   Mm section object prototype ptes , Binary: nt!mm

This indicates that the Paged pool tag MmSt is using approximately 293MB. Depending on the maximum paged pool limit on the machine, this could be too much and cause the system to fail additional pool allocations.

 

Process Explorer

Process Explorer can be used live on the your machines to determine the maximum paged and nonpaged pool memory limits without needing to dump the machine. It is the least intrusive method of accomplishing this. Internet connectivity to the Public Microsoft Symbol Server is required for this to work successfully.

 

Memory Dump Files

Collecting a memory dump file from the server while it is in a problematic state or consuming high paged pool memory will confirm if MmSt is the responsible pool tag. The debugger will indicate that there have been pool allocation failures if at some point the machine could not allocate pool memory.

  • Requires bugchecking the machine
  • LiveKD can be used as a substitute instead of bugchecking the machine
    • Some data cannot always be retrieved from dumps generated by LiveKD due to the manner in which LiveKD collects the dump
  • Debugger Commands
    • !vm
      • Displays Virtual Memory values
    • !poolused /t 10 4
      • Displays the top 10 paged pool consumers
  • Can be used to check the versions of the core binaries
  • Can be used to check $MFT size
  • Can be used to check if .PST files are open on the server

Sample from a dump file:

!vm

  • *** Virtual Memory Usage ***
  •         Physical Memory:     8386920 (  33547680 Kb)
  •         Page File: \??\C:\pagefile.sys
  •           Current:  16384000 Kb  Free Space:  16346204 Kb
  •           Minimum:  16384000 Kb  Maximum:     16385024 Kb
  •         Available Pages:     6756395 (  27025580 Kb)
  •         ResAvail Pages:      8190718 (  32762872 Kb)
  •         Locked IO Pages:           0 (         0 Kb)
  •         Free System PTEs:      48900 (    195600 Kb)
  •         ******* 40192 system cache map requests have failed ******
  •         Modified Pages:        89760 (    359040 Kb)
  •         Modified PF Pages:     89632 (    358528 Kb)
  •         NonPagedPool Usage:    70527 (    282108 Kb)
  •         NonPagedPool Max:     523263 (   2093052 Kb)
  •         PagedPool 0 Usage:     26793 (    107172 Kb)
  •         PagedPool 1 Usage:     14819 (     59276 Kb)
  •         PagedPool 2 Usage:     12600 (     50400 Kb)
  •         PagedPool 3 Usage:     13044 (     52176 Kb)
  •         PagedPool 4 Usage:     12444 (     49776 Kb)
  •         PagedPool Usage:       79700 (    318800 Kb)
  •         PagedPool Maximum:523264 (   2093056 Kb) <-Paged pool limit for this machine
  •         ********** 33134 pool allocations have failed **********

3: kd> !poolused /t 10 4
Sorting by Paged Pool Consumed

                         NonPaged                             Paged
Tag             Allocs             Used          Allocs           Used
MmSt             0                     0             3696056   334902032    Mm section object prototype ptes , Binary: nt!mm
MmPd 2 525328 2470 225509872 Mm page table commitment bitmaps , Binary: nt!mm

The values above indicate that the MmSt Paged pool tag had 3696056 memory allocations and is consuming approximately 334MB of Paged Pool memory.

 

RAMMAP

RamMap, a tool from Sysinternals, can be used to determine why high MmSt pool usage is occurring. RAMMap is an advanced physical memory usage analysis utility for Windows Vista and higher.

  • View the Mapped File value to determine if memory mapped files are causing the high MmSt usage
    • Go to the Physical Pages tab and sort the Use column, scroll down to Mapped File to determine which files are being mapped into memory, such as .PSTs

 

RAMMAP Screenshot showing Mapped File and Metafile (MFT):

clip_image002

RAMMAP screenshot showing memory mapped files.

clip_image004

 

Troubleshooting Steps and Resolutions

Implementing the steps in the first two bullets below will resolve a majority of these issues:

  • Apply the latest hotfixes for core OS files to fix known issues.
    • Ntoskrnl.exe
    • Ntfs.sys
    • Srv.sys
    • Storport.sys
    • Install latest VSS rollup if VSS is being used
  • Adjust Maximum Paged Pool Memory and Trimming
    • We often hit the paged pool limit because the trimming process can’t keep up once paged pool hits the default of 80% limit before we start trimming. Adjusting the PoolUsageMaximum from KB312362 changes the default throttle so that we can force trimming to start earlier. Adjusting the PagedPoolSize value will tell the system to reserve as much paged pool memory as possible at boot time. These two registry keys will often alleviate the pressure on the Memory Manager and give it sufficient time to trim paged pool before nearing the maximum limit.
    • Tuning Paged Pool Trimming and Maximum Paged Pool Values
      • 312362 Server is unable to allocate memory from the system paged pool
        http://support.microsoft.com/default.aspx?scid=kb;EN-US;312362
      • PoolUsageMaximum from KB312362 can safely be set to a decimal value of 40 which will cause trimming to start when paged pool reaches 40% of the maximum limit
      • PagedPoolSize should be set to 0xFFFFFFFF to ensure the system reserves as much paged pool memory as possible
    • Remove 4GT Tuning if it is not required
      • Check if /3GB is enabled on the machine and confirm it is necessary for the roles of the server.
      • Removing /3GB (only if it is not required on the server) will increase the paged pool limit dramatically. Adding the /USERVA switch will also slightly increase the maximum amount of paged pool.

 

Determining Root Cause

In addition to providing the latest binaries and adjusting the registry keys for paged pool memory as stated above, data will need to be collected to determine root cause and additional troubleshooting steps if required.

  • PST Files Stored on a File Server
  • Heavy File I/O
    • Common Causes
      • Overloaded file servers
      • Antivirus scans
      • Backups
      • Very large number of files being opened or copied
    • Use Performance Monitor logging to confirm if this is the issue
      • Check file I/O and paged pool usage for similar trends
    • Recommendations
      • Set virus scans and backups to run when there is less load on the system
      • It may be necessary to add an additional server to handle the file I/O load if adjusting the PoolUsageMaximum value doesn’t resolve the issue.
      • x64 OS may be the best recommendation if the server is unable to handle the load without being depleted of paged pool
  • Large $MFT
  • VSS Snapshots (Volume Shadow Copy)
    • Install the latest VSS updates
    • Reduce the size of the VSS Store
    • Reduce the number of snapshots
    • KB903234 - An update is available to optimize the way that the Shadow Copy Client accesses shadow copies in Windows Server 2003 and in Windows XP
  • Custom Applications

 

Useful Links

Poolmon Overview:
http://technet.microsoft.com/en-us/library/cc737099(WS.10).aspx

Finding Poolmon.exe:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=18546

How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks - http://support.microsoft.com/kb/177415

RAMMAP Askperf Blog:
http://blogs.technet.com/b/askperf/archive/2010/08/13/introduction-to-the-new-sysinternals-tool-rammap.aspx

Process Explorer:
http://technet.microsoft.com/en-us/sysinternals/bb896653

How to generate a kernel dump file or a complete memory dump file in Windows Server 2003:
http://support.microsoft.com/kb/972110

Rami Munder

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • thanks for the post. My high MmSt usage comes due to an Superfetch regression in Windows 7. Everytime a software uses FindFirstFile to query directories (TSVN and other tools) Superfetch heavily allocates MmSt memory. I sometimes have 3GB memory usage if all programs are closed. I have over 1GB paged pool usage (most by MmSt or File), 1GB in $MFT and the rest from the Windows services programs. This never happend in Vista only in Windows 7. In Windows 8 the paged pool usage (outstanding allocations) seems to be reduced.