Mike Lagase

Saving the Exchange world one day at a time.....

Detecting Low Virtual Memory Conditions in Windows 2008 and R2

Detecting Low Virtual Memory Conditions in Windows 2008 and R2

  • Comments 3
  • Likes

On Windows 2008 servers with Exchange 2007/2010 installed, there are times when you may run out of virtual memory for various reasons. One could be a memory leak in some application or simply not configuring the paging file correctly.

Once you run out of virtual memory on any given server, various applications may start failing/crashing on the server due to the inability to obtain memory to complete a specific function that is being called. In some cases, this could lead to a possible blue screen of death (BSOD).

For server based systems, the new Reliability Infrastructure helps automatically diagnose various operating system components. Of that infrastructure, Resource Exhaustion Detection and Resolution (RADAR) helps notify you when you are resources are reaching critical levels. RADAR is part of the Diagnostic Policy service that is installed on each server.

When RADAR detects that memory has reached a critical state, a 2004 event will be logged to the system log. An example of one of these events is shown below. As you can see, it has various information that provides overall memory consumption for various system resources, the top processes for memory consumption, file version information and paged/nonpaged pool memory that includes the top tags that could attribute to the memory problem. The bolded parts are the area of interest.

Log Name:      System
Source:        Microsoft-Windows-Resource-Exhaustion-Detector
Event ID:      2004
Task Category: Resource Exhaustion Diagnosis Events
Level:         Warning
Keywords:      Events related to exhaustion of system commit limit (virtual memory).
User:          SYSTEM
Description:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: store.exe (7580) consumed 11282399232 bytes, MSExchangeMailboxAssistants.exe (21200) consumed 590950400 bytes, and w3wp.exe (21092) consumed 562757632 bytes.
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
    <EventID>2004</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>3</Task>
    <Opcode>33</Opcode>
    <Keywords>0x8000000020000000</Keywords>
    <TimeCreated SystemTime="2010-09-03T10:47:01.431311400Z" />
    <EventRecordID>169289</EventRecordID>
    <Correlation ActivityID="{AC93AF3C-02AE-433D-8C22-FA32493FAD8C}" />
    <Execution ProcessID="1160" ThreadID="8312" />
    <Channel>System</Channel>
    <Computer>Exserver01.domain.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <UserData>
    <MemoryExhaustionInfo xmlns:auto-ns2="
http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
      <SystemInfo>
        <SystemCommitLimit>21261021184</SystemCommitLimit>
        <SystemCommitCharge>20993597440</SystemCommitCharge>
        <ProcessCommitCharge>19448094720</ProcessCommitCharge>
        <PagedPoolUsage>453672960</PagedPoolUsage>
        <PhysicalMemorySize>17176764416</PhysicalMemorySize>
        <PhysicalMemoryUsage>17025470464</PhysicalMemoryUsage>
        <NonPagedPoolUsage>422363136</NonPagedPoolUsage>
        <Processes>133</Processes>
      </SystemInfo>
    
<ProcessInfo>
        <Process_1>
          <Name>store.exe</Name>
          <ID>7580</ID>
          <CreationTime>2010-09-02T11:21:32.755807700Z</CreationTime>
          <CommitCharge>11282399232</CommitCharge>
          <HandleCount>5619</HandleCount>
          <Version>14.1.218.10</Version>
          <TypeInfo>1089</TypeInfo>
        </Process_1>
        <Process_2>
          <Name>MSExchangeMailboxAssistants.exe</Name>
          <ID>21200</ID>
          <CreationTime>2010-08-28T06:50:53.878440200Z</CreationTime>
          <CommitCharge>590950400</CommitCharge>
          <HandleCount>2664</HandleCount>
          <Version>14.1.218.10</Version>
          <TypeInfo>1090</TypeInfo>
        </Process_2>
        <Process_3>
          <Name>w3wp.exe</Name>
          <ID>21092</ID>
          <CreationTime>2010-08-31T08:25:12.245594900Z</CreationTime>
          <CommitCharge>562757632</CommitCharge>
          <HandleCount>2817</HandleCount>
          <Version>7.0.6002.18005</Version>
          <TypeInfo>67</TypeInfo>
        </Process_3>
        <Process_4>
          <Name>powershell.exe</Name>
          <ID>19692</ID>
          <CreationTime>2010-09-03T09:12:48.188589800Z</CreationTime>
          <CommitCharge>152682496</CommitCharge>
          <HandleCount>629</HandleCount>
          <Version>6.0.6002.18111</Version>
          <TypeInfo>136</TypeInfo>
        </Process_4>
        <Process_5>
          <Name>mmc.exe</Name>
          <ID>18768</ID>
          <CreationTime>2010-09-03T09:12:42.167067000Z</CreationTime>
          <CommitCharge>107646976</CommitCharge>
          <HandleCount>464</HandleCount>
          <Version>6.0.6002.18005</Version>
          <TypeInfo>144</TypeInfo>
        </Process_5>
        <Process_6>
          <Name>explorer.exe</Name>
          <ID>13396</ID>
          <CreationTime>2010-09-03T09:12:24.929288000Z</CreationTime>
          <CommitCharge>22032384</CommitCharge>
          <HandleCount>451</HandleCount>
          <Version>6.0.6002.18005</Version>
          <TypeInfo>152</TypeInfo>
        </Process_6>
      </ProcessInfo>
      <PagedPoolInfo>
        <Tag_1>
          <Name>MmSt</Name>
          <PoolUsed>216638928</PoolUsed>
        </Tag_1>
        <Tag_2>
          <Name>CM31</Name>
          <PoolUsed>103596032</PoolUsed>
        </Tag_2>
        <Tag_3>
          <Name>MmRe</Name>
          <PoolUsed>15907504</PoolUsed>
        </Tag_3>
      </PagedPoolInfo>
      <NonPagedPoolInfo>
        <Tag_1>
          <Name>SmMs</Name>
          <PoolUsed>161243168</PoolUsed>
        </Tag_1>
        <Tag_2>
          <Name>BCM0</Name>
          <PoolUsed>40694064</PoolUsed>
        </Tag_2>
        <Tag_3>
          <Name>Cont</Name>
          <PoolUsed>35498720</PoolUsed>
        </Tag_3>
      </NonPagedPoolInfo>
      <ExhaustionEventInfo>
        <Time>2010-09-03T10:47:18.540433800Z</Time>
      </ExhaustionEventInfo>
    </MemoryExhaustionInfo>
  </UserData>
</Event>

This helps you determine what resource was the possible offender without having to install any additional tools on the server to troubleshoot this. The best part is that you don’t have to wait for an additional event to occur as the information has already been collected and logged.

There is another place where events are logged which is under the Windows Resource Exhaustion Detector (Resource-Exhaustion-Detector) under Applications and Services Logs in the Event Viewer as shown below.

image

These events show much less information than the system event, but do show your commit limits and charges to the system too. Sample below.

Log Name:      Microsoft-Windows-Resource-Exhaustion-Detector/Operational
Source:        Microsoft-Windows-Resource-Exhaustion-Detector
Event ID:      1003
Task Category: Resource Exhaustion Detection Events
Level:         Warning
Keywords:      Events related to exhaustion of system commit limit (virtual memory).
User:          SYSTEM
Computer:      ExServer01.Domain.Com
Description:
The Windows Resource Exhaustion Detector received a notification that the computer is low on virtual memory.
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
    <EventID>1003</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>2</Task>
    <Opcode>22</Opcode>
    <Keywords>0x4000000020000000</Keywords>
    <TimeCreated SystemTime="2010-09-03T10:52:01.431065200Z" />
    <EventRecordID>180</EventRecordID>
    <Correlation ActivityID="{0B95CAB5-E004-4C92-BF5D-3BFA39FDF7EE}" />
    <Execution ProcessID="1160" ThreadID="8312" />
    <Channel>Microsoft-Windows-Resource-Exhaustion-Detector/Operational</Channel>
    <Computer>ExServer01.domain.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <UserData>
    <CommitLimitExhaustion xmlns:auto-ns2="
http://schemas.microsoft.com/win/2004/08/events"xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
  
   <SystemCommitLimit>21261021184</SystemCommitLimit>
      <SystemCommitCharge>21258543104</SystemCommitCharge>

    </CommitLimitExhaustion>
  </UserData>
</Event>

A couple of potential events that can be seen when memory resources are low are shown below.

  • MSExchangeRepl Service failing to read a log file for database copy due to an out of memory error condition.

    Log Name:      Application
    Source:        MSExchangeRepl
    Event ID:      2168
    Task Category: Service
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      ExServer01.Domain.Com
    Description:
    Log file F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log' for database copy EXServer MBX Store 001\ExServer01' couldn't be read. Error: Out of Memory (-1011)
    Event Xml:
    <Event xmlns="
    http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeRepl" />
        <EventID Qualifiers="49156">2168</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-09-02T11:21:13.000000000Z" />
        <EventRecordID>3190563</EventRecordID>
        <Channel>Application</Channel>
        <Computer>Exserver01.domain.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log</Data>
        <Data>EXServer MBX Store 001\ExServer01' </Data>
        <Data>Out of Memory (-1011)</Data>
      </EventData>
    </Event>

  • A Registry flush operation failing to write the SOFTWARE hive to disk

    Log Name:      System
    Source:        Microsoft-Windows-Kernel-General
    Event ID:      6
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      ExServer01.domain.com
    Description:
    An I/O operation initiated by the Registry failed unrecoverably.The Registry could not flush hive (file): '\SystemRoot\System32\Config\SOFTWARE'.
    Event Xml:
    <Event xmlns="
    http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Kernel-General" Guid="{a68ca8b7-004f-d7b6-a698-07e2de0f1f5d}" />
        <EventID>6</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2010-09-03T10:48:17.714333400Z" />
        <EventRecordID>169290</EventRecordID>
        <Correlation />
        <Execution ProcessID="4" ThreadID="92" />
        <Channel>System</Channel>
        <Computer>ExServer01.domain.com</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="FinalStatus">0xc000014d</Data>
        <Data Name="ExtraStringLength">36</Data>
        <Data Name="ExtraString">\SystemRoot\System32\Config\SOFTWARE</Data>
      </EventData>
    </Event>

Depending on the component used to instantiate a specific function will determine what component will log the event in the system log. Finding root cause for memory issues has become significantly easier with this new Reliability Infrastructure and I hope this blog helps show you some of the methods for troubleshooting these type issues.

Until next time!!!

Comments
  • You work a lot for this. Thanks from all people that consider it is a step forward.

  • Log Name:      System

    Source:        NfsServer

    Date:          07-Jun-13 10:37:53 AM

    Event ID:      1006

    Task Category: None

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      fs03.circasys.com

    Description:

    Server for NFS is not configured for either Active Directory Lookup or User Name Mapping.

    Without either Active Directory Lookup or User Name Mapping, Server for NFS cannot grant file access to users.

    Configure Server for NFS for either Active Directory Lookup or User Name Mapping using the Nfsadmin command-line tool.

    Event Xml:

    <Event xmlns="schemas.microsoft.com/.../event">

     <System>

       <Provider Name="NfsServer" />

       <EventID Qualifiers="49152">1006</EventID>

       <Level>2</Level>

       <Task>0</Task>

       <Keywords>0x80000000000000</Keywords>

       <TimeCreated SystemTime="2013-06-07T05:07:53.198Z" />

       <EventRecordID>165920</EventRecordID>

       <Channel>System</Channel>

       <Computer>fs03.circasys.com</Computer>

       <Security />

     </System>

     <EventData>

       <Data>\Device\NfsSvr</Data>

       <Binary>000000000100000000000000EE0300C0000000000201000001000000000000000000000000000000</Binary>

     </EventData>

    </Event>

  • Why this Error Occurs and what is the procedure to remove these type of errors

    Log Name:      System

    Source:        Microsoft-Windows-TerminalServices-Printers

    Date:          07-Jun-13 10:22:05 AM

    Event ID:      1111

    Task Category: None

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      fs03.circasys.com

    Description:

    Driver Send To Microsoft OneNote 2010 Driver required for printer Send To OneNote 2010 is unknown. Contact the administrator to install the driver before you log in again.

    Event Xml:

    <Event xmlns="schemas.microsoft.com/.../event">

     <System>

       <Provider Name="Microsoft-Windows-TerminalServices-Printers" Guid="{952773BF-C2B7-49BC-88F4-920744B82C43}" EventSourceName="UmrdpService" />

       <EventID Qualifiers="0">1111</EventID>

       <Version>0</Version>

       <Level>2</Level>

       <Task>0</Task>

       <Opcode>0</Opcode>

       <Keywords>0x80000000000000</Keywords>

       <TimeCreated SystemTime="2013-06-07T04:52:05.000Z" />

       <EventRecordID>165880</EventRecordID>

       <Correlation />

       <Execution ProcessID="0" ThreadID="0" />

       <Channel>System</Channel>

       <Computer>fs03.circasys.com</Computer>

       <Security />

     </System>

     <EventData>

       <Data>Send To Microsoft OneNote 2010 Driver</Data>

       <Data>Send To OneNote 2010</Data>

       <Binary>00000000BA0E0000</Binary>

     </EventData>

    </Event>

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment