Previously I had wrote a set of steps for fixing/repairing broken Exchange 2007 counters at http://blogs.technet.com/b/mikelag/archive/2009/10/21/how-to-fix-repair-broken-exchange-2007-counters.aspx. These set of steps have been replaced with an easier way of loading the counters using Powershell for Exchange 2010. This new method applies mainly for managed code services in Exchange 2010 as some of the previous lodctr/unlodctr commands still work for services such as MSExchangeIS and ESE.

In this example, I am going to use the same ESE missing counters as an example.

The important values that we are interested are the following. These values must exist prior to loading the ESE performance counters.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance]
"Close"="ClosePerformanceData"
"Collect"="CollectPerformanceData"
"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\V14\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"
"Open"="OpenPerformanceData"
"PerfIniFile"="eseperf.ini"

Here is a screenshot of what it will look like:

Again, verify if the name Disable Performance Counters is set in the registry and if so, is the value set to a value other than 0? If this is set to a value of 4, those counters had a problem at one point and were disabled. If this is set to 4, then change it to 0 and then reopen Performance Monitor to see if the counters can be seen now. If that works, life is good and you can either stop reading here or continue on to reloading the counters if they are still not available.

If the values First Counter, First Help, Last Counter and Last Help are listed, then it is recommended to unload the performance counters first before attempting to reload them again.

To unload performance counters for ESE, perform the following steps:

• Close down any running versions of Performance Monitor and stop any other monitoring services that might be attempting to use those counters
• Open the Exchange Management Shell
• Type add-pssnapin Microsoft.Exchange.Management.PowerShell.Setup and hit enter
• Type remove-perfcounters –definitionfilename “C:\Program Files\Microsoft\Exchange Server\V14\Bin\perf\AMD64\eseperf.xml” and hit enter
  Note: Ensure that the path is changed to the correct location of eseperf.xml on your server

To reload performance counters for ESE, perform the following steps

• Close down any running versions of Performance Monitor and stop any other monitoring services that might be attempting to use those counters
• Open the Exchange Management Shell
• Type add-pssnapin Microsoft.Exchange.Management.PowerShell.Setup and hit enter
• Type new-perfcounters –definitionfilename “C:\Program Files\Microsoft\Exchange Server\V14\Bin\perf\AMD64\eseperf.xml” and hit enter
  Note: Ensure that the path is changed to the correct location of eseperf.xml on your server

This will reload the appropriate performance counters for ESE. Once that is done, check the application event log to ensure that they were loaded OK and no Perflib errors exist. Open Performance monitor again to see if the counters are available.

If you are looking to reload any of the Exchange counters, you can substitute the appropriate XML in the command with any of the available XMLs from the \Program Files\Microsoft\Exchange Server\V14\Setup\Perf directory.

Here is a screenshot of some of the available performance counters

I hope this helps in your performance counter monitoring in Exchange 2010.

On Windows 2008 servers with Exchange 2007/2010 installed, there are times when you may run out of virtual memory for various reasons. One could be a memory leak in some application or simply not configuring the paging file correctly.

Once you run out of virtual memory on any given server, various applications may start failing/crashing on the server due to the inability to obtain memory to complete a specific function that is being called. In some cases, this could lead to a possible blue screen of death (BSOD).

For server based systems, the new Reliability Infrastructure helps automatically diagnose various operating system components. Of that infrastructure, Resource Exhaustion Detection and Resolution (RADAR) helps notify you when you are resources are reaching critical levels. RADAR is part of the Diagnostic Policy service that is installed on each server.

When RADAR detects that memory has reached a critical state, a 2004 event will be logged to the system log. An example of one of these events is shown below. As you can see, it has various information that provides overall memory consumption for various system resources, the top processes for memory consumption, file version information and paged/nonpaged pool memory that includes the top tags that could attribute to the memory problem. The bolded parts are the area of interest.

Log Name:      System
Source:        Microsoft-Windows-Resource-Exhaustion-Detector
Event ID:      2004
Task Category: Resource Exhaustion Diagnosis Events
Level:         Warning
Keywords:      Events related to exhaustion of system commit limit (virtual memory).
User:          SYSTEM
Description:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: store.exe (7580) consumed 11282399232 bytes, MSExchangeMailboxAssistants.exe (21200) consumed 590950400 bytes, and w3wp.exe (21092) consumed 562757632 bytes.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
    <EventID>2004</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>3</Task>
    <Opcode>33</Opcode>
    <Keywords>0x8000000020000000</Keywords>
    <TimeCreated SystemTime="2010-09-03T10:47:01.431311400Z" />
    <EventRecordID>169289</EventRecordID>
    <Correlation ActivityID="{AC93AF3C-02AE-433D-8C22-FA32493FAD8C}" />
    <Execution ProcessID="1160" ThreadID="8312" />
    <Channel>System</Channel>
    <Computer>Exserver01.domain.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <UserData>
    <MemoryExhaustionInfo xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
      <SystemInfo>
        <SystemCommitLimit>21261021184</SystemCommitLimit>
        <SystemCommitCharge>20993597440</SystemCommitCharge>
        <ProcessCommitCharge>19448094720</ProcessCommitCharge>
        <PagedPoolUsage>453672960</PagedPoolUsage>
        <PhysicalMemorySize>17176764416</PhysicalMemorySize>
        <PhysicalMemoryUsage>17025470464</PhysicalMemoryUsage>
        <NonPagedPoolUsage>422363136</NonPagedPoolUsage>
        <Processes>133</Processes>
      </SystemInfo>
     <ProcessInfo>
        <Process_1>
          <Name>store.exe</Name>
          <ID>7580</ID>
          <CreationTime>2010-09-02T11:21:32.755807700Z</CreationTime>
          <CommitCharge>11282399232</CommitCharge>
          <HandleCount>5619</HandleCount>
          <Version>14.1.218.10</Version>
          <TypeInfo>1089</TypeInfo>
        </Process_1>
        <Process_2>
          <Name>MSExchangeMailboxAssistants.exe</Name>
          <ID>21200</ID>
          <CreationTime>2010-08-28T06:50:53.878440200Z</CreationTime>
          <CommitCharge>590950400</CommitCharge>
          <HandleCount>2664</HandleCount>
          <Version>14.1.218.10</Version>
          <TypeInfo>1090</TypeInfo>
        </Process_2>
        <Process_3>
          <Name>w3wp.exe</Name>
          <ID>21092</ID>
          <CreationTime>2010-08-31T08:25:12.245594900Z</CreationTime>
          <CommitCharge>562757632</CommitCharge>
          <HandleCount>2817</HandleCount>
          <Version>7.0.6002.18005</Version>
          <TypeInfo>67</TypeInfo>
        </Process_3>
        <Process_4>
          <Name>powershell.exe</Name>
          <ID>19692</ID>
          <CreationTime>2010-09-03T09:12:48.188589800Z</CreationTime>
          <CommitCharge>152682496</CommitCharge>
          <HandleCount>629</HandleCount>
          <Version>6.0.6002.18111</Version>
          <TypeInfo>136</TypeInfo>
        </Process_4>
        <Process_5>
          <Name>mmc.exe</Name>
          <ID>18768</ID>
          <CreationTime>2010-09-03T09:12:42.167067000Z</CreationTime>
          <CommitCharge>107646976</CommitCharge>
          <HandleCount>464</HandleCount>
          <Version>6.0.6002.18005</Version>
          <TypeInfo>144</TypeInfo>
        </Process_5>
        <Process_6>
          <Name>explorer.exe</Name>
          <ID>13396</ID>
          <CreationTime>2010-09-03T09:12:24.929288000Z</CreationTime>
          <CommitCharge>22032384</CommitCharge>
          <HandleCount>451</HandleCount>
          <Version>6.0.6002.18005</Version>
          <TypeInfo>152</TypeInfo>
        </Process_6>
      </ProcessInfo>
      <PagedPoolInfo>
        <Tag_1>
          <Name>MmSt</Name>
          <PoolUsed>216638928</PoolUsed>
        </Tag_1>
        <Tag_2>
          <Name>CM31</Name>
          <PoolUsed>103596032</PoolUsed>
        </Tag_2>
        <Tag_3>
          <Name>MmRe</Name>
          <PoolUsed>15907504</PoolUsed>
        </Tag_3>
      </PagedPoolInfo>
      <NonPagedPoolInfo>
        <Tag_1>
          <Name>SmMs</Name>
          <PoolUsed>161243168</PoolUsed>
        </Tag_1>
        <Tag_2>
          <Name>BCM0</Name>
          <PoolUsed>40694064</PoolUsed>
        </Tag_2>
        <Tag_3>
          <Name>Cont</Name>
          <PoolUsed>35498720</PoolUsed>
        </Tag_3>
      </NonPagedPoolInfo>
      <ExhaustionEventInfo>
        <Time>2010-09-03T10:47:18.540433800Z</Time>
      </ExhaustionEventInfo>
    </MemoryExhaustionInfo>
  </UserData>
</Event>

This helps you determine what resource was the possible offender without having to install any additional tools on the server to troubleshoot this. The best part is that you don’t have to wait for an additional event to occur as the information has already been collected and logged.

There is another place where events are logged which is under the Windows Resource Exhaustion Detector (Resource-Exhaustion-Detector) under Applications and Services Logs in the Event Viewer as shown below.

These events show much less information than the system event, but do show your commit limits and charges to the system too. Sample below.

Log Name:      Microsoft-Windows-Resource-Exhaustion-Detector/Operational
Source:        Microsoft-Windows-Resource-Exhaustion-Detector
Event ID:      1003
Task Category: Resource Exhaustion Detection Events
Level:         Warning
Keywords:      Events related to exhaustion of system commit limit (virtual memory).
User:          SYSTEM
Computer:      ExServer01.Domain.Com
Description:
The Windows Resource Exhaustion Detector received a notification that the computer is low on virtual memory.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
    <EventID>1003</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>2</Task>
    <Opcode>22</Opcode>
    <Keywords>0x4000000020000000</Keywords>
    <TimeCreated SystemTime="2010-09-03T10:52:01.431065200Z" />
    <EventRecordID>180</EventRecordID>
    <Correlation ActivityID="{0B95CAB5-E004-4C92-BF5D-3BFA39FDF7EE}" />
    <Execution ProcessID="1160" ThreadID="8312" />
    <Channel>Microsoft-Windows-Resource-Exhaustion-Detector/Operational</Channel>
    <Computer>ExServer01.domain.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <UserData>
    <CommitLimitExhaustion xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events"xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
      <SystemCommitLimit>21261021184</SystemCommitLimit>
      <SystemCommitCharge>21258543104</SystemCommitCharge>
    </CommitLimitExhaustion>
  </UserData>
</Event>

A couple of potential events that can be seen when memory resources are low are shown below.

• MSExchangeRepl Service failing to read a log file for database copy due to an out of memory error condition.
  
  Log Name:      Application
  Source:        MSExchangeRepl
  Event ID:      2168
  Task Category: Service
  Level:         Error
  Keywords:      Classic
  User:          N/A
  Computer:      ExServer01.Domain.Com
  Description:
  Log file F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log' for database copy EXServer MBX Store 001\ExServer01' couldn't be read. Error: Out of Memory (-1011)
  Event Xml:
  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
      <Provider Name="MSExchangeRepl" />
      <EventID Qualifiers="49156">2168</EventID>
      <Level>2</Level>
      <Task>1</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2010-09-02T11:21:13.000000000Z" />
      <EventRecordID>3190563</EventRecordID>
      <Channel>Application</Channel>
      <Computer>Exserver01.domain.com</Computer>
      <Security />
    </System>
    <EventData>
      <Data>F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log</Data>
      <Data>EXServer MBX Store 001\ExServer01' </Data>
      <Data>Out of Memory (-1011)</Data>
    </EventData>
  </Event>
  
• A Registry flush operation failing to write the SOFTWARE hive to disk
  
  Log Name:      System
  Source:        Microsoft-Windows-Kernel-General
  Event ID:      6
  Task Category: None
  Level:         Error
  Keywords:     
  User:          SYSTEM
  Computer:      ExServer01.domain.com
  Description:
  An I/O operation initiated by the Registry failed unrecoverably.The Registry could not flush hive (file): '\SystemRoot\System32\Config\SOFTWARE'.
  Event Xml:
  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
      <Provider Name="Microsoft-Windows-Kernel-General" Guid="{a68ca8b7-004f-d7b6-a698-07e2de0f1f5d}" />
      <EventID>6</EventID>
      <Version>0</Version>
      <Level>2</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x8000000000000000</Keywords>
      <TimeCreated SystemTime="2010-09-03T10:48:17.714333400Z" />
      <EventRecordID>169290</EventRecordID>
      <Correlation />
      <Execution ProcessID="4" ThreadID="92" />
      <Channel>System</Channel>
      <Computer>ExServer01.domain.com</Computer>
      <Security UserID="S-1-5-18" />
    </System>
    <EventData>
      <Data Name="FinalStatus">0xc000014d</Data>
      <Data Name="ExtraStringLength">36</Data>
      <Data Name="ExtraString">\SystemRoot\System32\Config\SOFTWARE</Data>
    </EventData>
  </Event>

Depending on the component used to instantiate a specific function will determine what component will log the event in the system log. Finding root cause for memory issues has become significantly easier with this new Reliability Infrastructure and I hope this blog helps show you some of the methods for troubleshooting these type issues.

Until next time!!!