On Windows 2008 servers with Exchange 2007/2010 installed, there are times when you may run out of virtual memory for various reasons. One could be a memory leak in some application or simply not configuring the paging file correctly.
Once you run out of virtual memory on any given server, various applications may start failing/crashing on the server due to the inability to obtain memory to complete a specific function that is being called. In some cases, this could lead to a possible blue screen of death (BSOD).
For server based systems, the new Reliability Infrastructure helps automatically diagnose various operating system components. Of that infrastructure, Resource Exhaustion Detection and Resolution (RADAR) helps notify you when you are resources are reaching critical levels. RADAR is part of the Diagnostic Policy service that is installed on each server.
When RADAR detects that memory has reached a critical state, a 2004 event will be logged to the system log. An example of one of these events is shown below. As you can see, it has various information that provides overall memory consumption for various system resources, the top processes for memory consumption, file version information and paged/nonpaged pool memory that includes the top tags that could attribute to the memory problem. The bolded parts are the area of interest.
Log Name: System Source: Microsoft-Windows-Resource-Exhaustion-Detector Event ID: 2004 Task Category: Resource Exhaustion Diagnosis Events Level: Warning Keywords: Events related to exhaustion of system commit limit (virtual memory). User: SYSTEM Description: Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: store.exe (7580) consumed 11282399232 bytes, MSExchangeMailboxAssistants.exe (21200) consumed 590950400 bytes, and w3wp.exe (21092) consumed 562757632 bytes. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" /> <EventID>2004</EventID> <Version>0</Version> <Level>3</Level> <Task>3</Task> <Opcode>33</Opcode> <Keywords>0x8000000020000000</Keywords> <TimeCreated SystemTime="2010-09-03T10:47:01.431311400Z" /> <EventRecordID>169289</EventRecordID> <Correlation ActivityID="{AC93AF3C-02AE-433D-8C22-FA32493FAD8C}" /> <Execution ProcessID="1160" ThreadID="8312" /> <Channel>System</Channel> <Computer>Exserver01.domain.com</Computer> <Security UserID="S-1-5-18" /> </System> <UserData> <MemoryExhaustionInfo xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events"> <SystemInfo> <SystemCommitLimit>21261021184</SystemCommitLimit> <SystemCommitCharge>20993597440</SystemCommitCharge> <ProcessCommitCharge>19448094720</ProcessCommitCharge> <PagedPoolUsage>453672960</PagedPoolUsage> <PhysicalMemorySize>17176764416</PhysicalMemorySize> <PhysicalMemoryUsage>17025470464</PhysicalMemoryUsage> <NonPagedPoolUsage>422363136</NonPagedPoolUsage> <Processes>133</Processes> </SystemInfo> <ProcessInfo> <Process_1> <Name>store.exe</Name> <ID>7580</ID> <CreationTime>2010-09-02T11:21:32.755807700Z</CreationTime> <CommitCharge>11282399232</CommitCharge> <HandleCount>5619</HandleCount> <Version>14.1.218.10</Version> <TypeInfo>1089</TypeInfo> </Process_1> <Process_2> <Name>MSExchangeMailboxAssistants.exe</Name> <ID>21200</ID> <CreationTime>2010-08-28T06:50:53.878440200Z</CreationTime> <CommitCharge>590950400</CommitCharge> <HandleCount>2664</HandleCount> <Version>14.1.218.10</Version> <TypeInfo>1090</TypeInfo> </Process_2> <Process_3> <Name>w3wp.exe</Name> <ID>21092</ID> <CreationTime>2010-08-31T08:25:12.245594900Z</CreationTime> <CommitCharge>562757632</CommitCharge> <HandleCount>2817</HandleCount> <Version>7.0.6002.18005</Version> <TypeInfo>67</TypeInfo> </Process_3> <Process_4> <Name>powershell.exe</Name> <ID>19692</ID> <CreationTime>2010-09-03T09:12:48.188589800Z</CreationTime> <CommitCharge>152682496</CommitCharge> <HandleCount>629</HandleCount> <Version>6.0.6002.18111</Version> <TypeInfo>136</TypeInfo> </Process_4> <Process_5> <Name>mmc.exe</Name> <ID>18768</ID> <CreationTime>2010-09-03T09:12:42.167067000Z</CreationTime> <CommitCharge>107646976</CommitCharge> <HandleCount>464</HandleCount> <Version>6.0.6002.18005</Version> <TypeInfo>144</TypeInfo> </Process_5> <Process_6> <Name>explorer.exe</Name> <ID>13396</ID> <CreationTime>2010-09-03T09:12:24.929288000Z</CreationTime> <CommitCharge>22032384</CommitCharge> <HandleCount>451</HandleCount> <Version>6.0.6002.18005</Version> <TypeInfo>152</TypeInfo> </Process_6> </ProcessInfo> <PagedPoolInfo> <Tag_1> <Name>MmSt</Name> <PoolUsed>216638928</PoolUsed> </Tag_1> <Tag_2> <Name>CM31</Name> <PoolUsed>103596032</PoolUsed> </Tag_2> <Tag_3> <Name>MmRe</Name> <PoolUsed>15907504</PoolUsed> </Tag_3> </PagedPoolInfo> <NonPagedPoolInfo> <Tag_1> <Name>SmMs</Name> <PoolUsed>161243168</PoolUsed> </Tag_1> <Tag_2> <Name>BCM0</Name> <PoolUsed>40694064</PoolUsed> </Tag_2> <Tag_3> <Name>Cont</Name> <PoolUsed>35498720</PoolUsed> </Tag_3> </NonPagedPoolInfo> <ExhaustionEventInfo> <Time>2010-09-03T10:47:18.540433800Z</Time> </ExhaustionEventInfo> </MemoryExhaustionInfo> </UserData> </Event>
This helps you determine what resource was the possible offender without having to install any additional tools on the server to troubleshoot this. The best part is that you don’t have to wait for an additional event to occur as the information has already been collected and logged.
There is another place where events are logged which is under the Windows Resource Exhaustion Detector (Resource-Exhaustion-Detector) under Applications and Services Logs in the Event Viewer as shown below.
These events show much less information than the system event, but do show your commit limits and charges to the system too. Sample below.
Log Name: Microsoft-Windows-Resource-Exhaustion-Detector/Operational Source: Microsoft-Windows-Resource-Exhaustion-Detector Event ID: 1003 Task Category: Resource Exhaustion Detection Events Level: Warning Keywords: Events related to exhaustion of system commit limit (virtual memory). User: SYSTEM Computer: ExServer01.Domain.Com Description: The Windows Resource Exhaustion Detector received a notification that the computer is low on virtual memory. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" /> <EventID>1003</EventID> <Version>0</Version> <Level>3</Level> <Task>2</Task> <Opcode>22</Opcode> <Keywords>0x4000000020000000</Keywords> <TimeCreated SystemTime="2010-09-03T10:52:01.431065200Z" /> <EventRecordID>180</EventRecordID> <Correlation ActivityID="{0B95CAB5-E004-4C92-BF5D-3BFA39FDF7EE}" /> <Execution ProcessID="1160" ThreadID="8312" /> <Channel>Microsoft-Windows-Resource-Exhaustion-Detector/Operational</Channel> <Computer>ExServer01.domain.com</Computer> <Security UserID="S-1-5-18" /> </System> <UserData> <CommitLimitExhaustion xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events"xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events"> <SystemCommitLimit>21261021184</SystemCommitLimit> <SystemCommitCharge>21258543104</SystemCommitCharge> </CommitLimitExhaustion> </UserData> </Event>
A couple of potential events that can be seen when memory resources are low are shown below.
Depending on the component used to instantiate a specific function will determine what component will log the event in the system log. Finding root cause for memory issues has become significantly easier with this new Reliability Infrastructure and I hope this blog helps show you some of the methods for troubleshooting these type issues.
Until next time!!!
You work a lot for this. Thanks from all people that consider it is a step forward.