PerfGuide

Low Available RAM

Clint Huffman — Tue, 28 Sep 2010 08:08:12 GMT

If you have arrived here, then you have identified a low available RAM condition on your Windows computer. If this is not correct, then return to the Start of the Performance Guide.

Available RAM Concepts

When the counter “\Memory\Available MBytes" has a value less than 64, then the computer is low on available RAM. This is certainly a relative threshold because years ago when computers had only 100MBs of RAM, 64MBs of available RAM would have been a good thing. In this case, we assume at least 1GB of RAM is installed on the computer, therefore 64MBs would be considered too low.

The Available MBytes counter is the sum of all free, zero (a page of memory with all zeroes written to it), and standby pages of memory. All of this memory is page of RAM that can be reused for processes. Generally, about 10% of your RAM being available is considered good because that allows up to 10% of it to be used for disk cache – meaning if a file is being read from the disk, but it already has it in disk cache, then it reads it from RAM versus going to the disk. Disks are dramatically slower than RAM, so having a lot of RAM being used for disk cache is a good this. This is why it is a good thing to have more RAM.

If the “\Memory\Free & Zero Page List Bytes” counter constantly has over 100MBs or more of memory, then this means that you likely have too much RAM which isn’t a bad thing unless you are concerned about the electricity drain from the RAM modules.

Troubleshooting

If Available MBytes is less than 10% of RAM or less than 100 [MBs], then it is important to determine which processes are consuming the most RAM, the processes on an increasing trend or RAM usage, the System Cache memory usage, and if the system is reaching its commit limit.

Is the System Low on Committed Memory: Lack of RAM may be a symptom of the system reaching its commit limit. If “\Memory\Committed Bytes In Use” is greater than 75%, then the computer may be running low on physical resources (RAM and/or page file). If you suspect this problem, then follow the link below. Otherwise, continue to step 3.
[Link to future blog post]
Identify which Processes are Consuming the Most RAM: The “Working Set” of a process is the amount of RAM in use by a process. Look at “\Process(*)\Working Set” to see which processes are consuming the most RAM and which are on an increasing trend. Keep in mind that Working Set is only the amount of RAM usage of a process and does not show memory that has been paged out.
Is System Cache Consuming RAM: If the values from the “\Memory\Cache Bytes” and/or “\Memory\System Cache Resident Bytes” performance counters are more than 50% of RAM, then go here. [Link to future blog post]
Is the Computer a Virtual Computer: Some virtual computer hosts might remove RAM from the virtual computer when the host or other virtual computers are low on RAM. If you suspect this condition, then consult with your virtual server software vendor and/or add RAM to to the host and virtual computer.

Treating the Symptoms

If all of the above troubleshooting steps have been exhausted, then the symptoms of low available RAM can be treated with one or more of the following:

Add RAM: More RAM will increase the amount available to the system.
Increase Page File Access Speeds: When the computer is low on available RAM, it will aggressively page out the working sets of processes. If read/write speeds to the page file are delayed due to other disk activity, then consider placing the page file on faster disks. See Disk performance [Link to future blog post].
Increase the Page File or Add more Page Files: When the computer is low on available RAM, it will aggressively page out the working sets of processes. Adding more page files, increasing the maximum range, or allowing the system to manage the page file(s) will allow it to page out more memory. See troubleshooting committed memory here [Link to future blog post].
Avoid “Memory Optimizers”: Windows is very efficient at managing memory. So called, “memory optimizers” simply force the kernel to delete its disk cache and trim the working sets (more paging) of processes. All this does is cause a lot of unnecessary disk I/O.

User Mode Versus Privileged Mode Processor Usage

Clint Huffman — Tue, 28 Sep 2010 07:34:26 GMT

You have arrived at this point in the adventure because you have identified high processor usage on your Windows computer. If this is not correct, then return to the Start of the Performance Guide.

Threads (the worker bees of a process) can execute in one of two modes: User Mode or Privileged Mode

This article will help you determine what kind of processor mode your computer or server is spending most of it’s time in. This is important because this is a major decision path in the adventure and change change the troubleshooting steps and the outcome dramatically.

“\Processor(_Total)\% Processor Time” is the sum of “\Processor(*)\% User Time” and “\Processor(*)\% Privileged Time”.

Privileged (Kernel) Mode

Privileged Time is the amount of time being spent in the Windows kernel executing system calls such as drivers, IRPs (I/O Request Packets), context switching, etc. If the operating system is spending more than 30% of it’s time in privileged mode, then it means that it is likely doing a high amount of I/O and one or more of the drivers are executing to manage that I/O.

You can identify this by using Task Manager, clicking on the Performance tab, then go to View, Show Kernel Times or by using the “\Processor(*)\% Privileged Time” performance counter in Performance Monitor.

The following screenshot shows a high amount of privileged time processor usage.

If your computer or server is spending more than 30% of it’s time in privileged mode, then look at a % DPC Time, % Interrupt Time, and Context Switches/sec in performance monitor.

If “\Processor(*)\% DPC Time” is greater than 20%, or if “\Processor(*)\% Interrupt Time” is greater then 20%, then go here.

If both of the counters above show very low values (less than 20%), then it could also be caused by high amounts of context switches per second.

Here is a case study on how to solve high context switching due to high disk I/O.

The Case of the 2 Million Context Switches

With that said, high context switching can also be caused by a high number of ready threads on the system, so reducing the number of threads can help as well.

Here is a good article by Mark Russinovich on how to use the driver SDK tool, KernRate, to profile the kernel. KernRate only works on Windows XP/2003 computers.
Mark Russinovich’s The Case of the System Process CPU Spikes

User Mode

User Time is the amount of time the processor spends executing application code therefore we need to determine what processes are consuming the most time and the function calls they are executing the most.

You can identify this by using Task Manager, clicking on the Performance tab, then go to View, Show Kernel Times or by using the “\Processor(*)\% User Time” performance counter in Performance Monitor.

The following screenshot shows a high amount of user time processor usage.

If your computer or server is spending most of it’s time in user mode, then follow this link on how to troubleshoot this issue:
How To: Identify Functions causing a High User-mode CPU Bottleneck for Server Applications in a Production Environment

Choose Your Own Adventure: High Deferred Procedure Calls (DPCs) or High Interrupts

Clint Huffman — Tue, 28 Sep 2010 07:31:57 GMT

You have arrived here because you have identified a high amount (greater than 50%) of processor time (\Processor(*)\% Processor Time), a high amount (greater than 30%) of privileged time (kernel time) (\Processor(*)\% Privileged Time) and a high amount of deferred procedure calls (greater than 20%) (\Processor(*)\% DPC Time) or a high number of interrupts (greater than 20%) (\Processor(*)\% Interrupt Time) on your Windows computer. If this is not correct, then return to the Start of the Performance Guide.

% DPC Time is the percentage of time that the processor spent receiving and servicing deferred procedure calls (DPCs) during the sample interval. DPCs are interrupts that run at a lower priority than standard interrupts. % DPC Time is a component of % Privileged Time because DPCs are executed in privileged mode. They are counted separately and are not a component of the interrupt counters.

% Interrupt Time is the time the processor spends receiving and servicing hardware interrupts during sample intervals. This value is an indirect indicator of the activity of devices that generate interrupts, such as the system clock, the mouse, disk drivers, data communication lines, network interface cards and other peripheral devices. These devices normally interrupt the processor when they have completed a task or require attention. Normal thread execution is suspended during interrupts. Most system clocks interrupt the processor every 10 milliseconds, creating a background of interrupt activity. suspends normal thread execution during interrupts. This counter displays the average busy time as a percentage of the sample time.

High Deferred Procedure Calls (DPCs) and high interrupts are typically caused by very busy or poorly written device drivers. Consider using a tool such as Microsoft xPerf to assist with diagnosis of this problem.

Here is an article by Windows IT Pro Magazine that goes into diagnosing DPC issues using Microsoft xPerf:

Windows IT Pro Magazine: Examining xPerf
http://www.windowsitpro.com/print/performance/examining-xperf.aspx

Tip: If the computer hangs often, then while it is hung, press the Cap Lock key on the keyboard to see if the light on the keyboard toggles on and off when you press it. If the Cap Lock key light does not respond, then the computer is hung due to a hardware problem. Otherwise, it is most likely a software problem.

The Microsoft PFE Performance Guide: Start Here

Clint Huffman — Tue, 28 Sep 2010 07:13:23 GMT

Welcome to the Microsoft Premier Field Engineering (PFE) Windows performance analysis guide. This is a web of interconnected documents intended to assist Microsoft Premier Field Engineers (PFE), Microsoft Support Professionals, and IT Professionals with prescriptive guidance on how to diagnose Windows performance issues in production environments and provide steps toward resolution.

This document assumes the reader is familiar with how to use Microsoft Windows Performance Monitor (perfmon) to gather and view live performance counters.

You have arrived here because you suspect a performance problem with your Microsoft Windows computer or server.

Initial Indicators of Windows Performance Problems

Start with the four primary resources of a computer: Memory, Processor, Disk, and Network, then branch out from there. The following is the performance counters (Microsoft Performance Monitor) that indicate a potential issue with each respective resource. Microsoft Windows Performance Monitor performance counters are used to initially identify performance problems, then various other tools are used to as next steps towards resolution.

Memory

Microsoft Windows uses many kinds of memory. Windows memory can be categorized into physical RAM, committed memory, application virtual memory, and kernel virtual memory.

Physical RAM

The easiest and best initial indicator of a lack of physical RAM condition is the “\Memory\Available MBytes” performance counter. Available MBytes is the amount of physical memory, in Megabytes, immediately available for allocation to a process or for system use. It is equal to the sum of memory assigned to the standby (cached), free and zero page lists.

If “\Memory\Available MBytes” is less than 100MBs or less than 5% of total physical RAM, then the computer may be running critically low on physical RAM. If you suspect this condition, then go here.

Committed Memory

Committed Bytes is the amount of committed virtual memory, in bytes. Committed memory is the physical memory which has space reserved on the disk paging file(s). In other words, it is memory that is “in use” by processes. The Commit Limit is the sum of all of the physical resources in which the operating system can use to store data – it is the sum of RAM and all of the page files. Once all of RAM and all of the page files are full and unable to expand, then the system has reached its commit limit. The “\Memory\% Committed Bytes In Use” performance counter is the percentage of the Commit Charge measured by “\Memory\Committed Bytes” compared to the commit limit measured by “\Memory\Commit Limit”.

If “\Memory\Committed Bytes In Use” is greater than 75%, then the computer may be running low on physical resources (RAM and/or page file). If you suspect this problem, then follow this link for next steps:
[Link to future blog post]

Application Virtual Memory

Each process in Windows has its own, private virtual address space. Ideally, virtual memory should be unimaginably large, but the fact remains that it is a finite resource. If an application (user mode) runs out of virtual memory, then it will likely crash with an out of memory exception.

Identify the Maximum Virtual Address Space for Applications: Use the following command (usable on Windows XP/2003 and greater) to determine the maximum size of application virtual memory:
wmic PATH Win32_OperatingSystem GET MaxProcessMemorySize

Sample output of this command for a Windows 7 x64 computer is:
MaxProcessMemorySize
8589934464

The output is in bytes, therefore this Windows 7 x64 computer has 8TBs of virtual memory per process. If an application (user mode) runs out of virtual memory (meaning if it gets close to the MaxProcessMemorySize), then it will likely crash with an out of memory exception which can typically be found in the Application Event log unless the application is designed for this condition.
Is Virtual Memory Low: If “\Process(*)\Virtual Memory” exceeds 80% of the MaxProcessMemorySize, then the application is likely running out of virtual memory and might crash soon if unable to find contiguous memory for its next memory allocation. If you suspect this problem, then follow this link for next steps:
[Link to future blog post]

Kernel Virtual Memory

The kernel also resides in virtual memory. If it runs out of virtual memory, then the operating system can hang. The kernel has three important resources that resources in virtual memory: System Page Table Entries (PTEs), pool paged, and pool non-paged memory pools.

System Page Table Entries

System Page Table Entries provide the mapping between virtual and physical memory. If the system is out of PTEs, then it is unable to allocate memory to processes.

If “\Memory\Free System Page Table Entries” is less than 10,000, then the operating system will likely suffer long performance delays. If you suspect this problem, then follow this link for next steps:
[Link to future blog post]

While the future blog post is under construction, here is other documents on troubleshooting low PTE problems:

Pool Paged and Pool Non-Paged Memory Pools

Pool Paged and Pool Non-Paged pools serve as the memory resources that the operating system and device drivers use to store their data structures. When they are unable to allocate memory, then the operating system will likely suffer long performance delays.

Identify the Maximum Pool Sizes: Performance counters do not provide the maximum sizes of the memory pools, therefore use Process Monitor and the Microsoft Debugging Tools for Windows to identify them. The following document shows how to install the Microsoft Debugging Tools for Windows and how to use Process Monitor to show the maximum pool sizes:
Pushing the Limits of Windows: Paged and Nonpaged Pool by Mark Russinovich
Is Virtual Memory Pools Low: If “\Memory\Pool Paged Bytes” or “\Memory\Pool NonPaged Bytes” are close to or exceed 80% of their respective maximum sizes, then the operating system will likely suffer long performance delays. If you suspect this problem, then follow the link for next steps:
[Link to future blog post]

While the future blog post is under construction, here is a document on troubleshooting low pool memory problems:
- Symptoms: Lack of Free System Page Table Entries (PTEs), system-wide delays (I/O request failures), and low on Paged Pool Memory and/or Non-paged Pool Memory on 32-bit Windows
- Pushing the Limits of Windows: Paged and Nonpaged Pool by Mark Russinovich

Processor

If “\Processor(_Total)\% Processor Time” is greater than 80% on average, then the computer may be busy with the processor resource. If your computer meets this criteria or you suspect it, then follow this link:
User Mode Versus Privileged Mode Processor Usage

Disk

If “\LogicalDisk(*)\Avg Disk Sec/Read” or “\LogicalDisk(*)\Avg Disk Sec/Write” is greater than 15ms (0.015 seconds), then the computer may have a disk performance issue. If your computer meets this criteria or you suspect it, then follow this link:
[Link to future blog post]

Network

If “\Network Interface(*)\Output Queue Length” is greater than 2 on average, then the network adapter is not able to put network packets on the network fast enough. This could be due to network latency, chattiness, packet loss, etc. If your computer meets this criteria or you suspect it, then follow this link:
[Link to future blog post]