Counter of the Week

Analyzing Storage Performance

2009-03-19T02:54:00Z

Introduction

The purpose of this article is to provide prescriptive guidance on how to troubleshoot logical and physical disk response times in regards to Windows performance analysis.

Start with the following performance counters to analyze disk response times:

\LogicalDisk\Avg. Disk Sec/Read
\LogicalDisk\Avg. Disk Sec/Write
\LogicalDisk\Disk Bytes/Sec
\LogicalDisk\Disk Reads/Sec
\LogicalDisk\Disk Writes/Sec
\LogicalDisk\Split IO/sec
\LogicalDisk\Disk Transfers/sec

These counters are generally the first ones to look at because we are looking for the following attributes of the Input/Output profile:

Average disk seconds/read and average disk seconds/write (response times): Are the users having to wait for application responses? Are we exceeding established thresholds for disk drive performance degradation (generally > 15 ms)?
Throughput: Are we saturating any of the pipes, such as mainboard bus, SCSI connection, SAN connection, or other link between servers and storage. Are we reaching the throughput limit of the disk subsystem?
Transfers/second: Is the server and its applications generating more I/O than the disk subsystem can keep up with? For example, suppose you have allocated 4 disk drives to a single logical disk group that is configured as RAID 1+0. Assuming 200 Input/Output Operations per second (IOPs) capability of a given disk drive, that RAID group will be capable of around 400 IOPs (cache and read-ahead may increase that number somewhat). Even being very generous and saying that with cache and optimizations a disk drive can perform 400 IOPs, the most that could be hoped for in write operating on a 4 disk RAID 1+0 is 800 IOPs. If transfers/second exceeds that number at the same time as response times are deteriorating, chances are there just are not enough disk drives to back that logical disk and the assigned workload.
Reads/second and writes/second: Gives you an indication as to the mix of workload that you are dealing with. Certain disk subsystem types handle certain workloads better than others. For example, some RAID-5 controllers can handle large I/O writes and sequential reads relatively well.
Split I/O: Does the operating system have to perform more than one command for each I/O? Split I/O is a good indicator of fragmentation, which can reduce performance by causing excessive seek time.

This article is grouped by symptoms, then by possible causes.

Symptoms: Long disk response times and High I/O

Applies to:

Windows Server 2003 (all editions) unless otherwise specified
Windows XP (all editions) unless otherwise specified
Windows Server 2000 (all editions) unless otherwise specified

Symptom Details:

Long disk response times: A “LogicalDisk\Avg. Disk Sec/Read” or “LogicalDisk\Avg. Disk Sec/Write” value greater than 15ms though occasional spikes are not necessarily cause for immediate concern.
High I/O: “\LogicalDisk\Disk Transfers/sec” is at or near the number of I/O operations per second that each physical spindle is designed to handle which is typically between 80 to 180 per disk.

Possible Causes	How to Diagnose	Possible Solutions and/or Recommendations
Storage response time reduced because of misaligned partitions.	A misaligned partition is the result of creating a partition that is cylinder aligned, versus sector aligned. Windows, up until Vista, aligned partitions using Cylinder, Head, and Sector addressing. Most storage controllers will still report some value for C/H/S, even though that scheme is likely not used for disk addressing. Windows will therefore create a partition on sector 63. You can see this in MSINFO32, or various other methods (look for hidden sectors, partition offset, etc). If the starting offset for a partition is 63 sectors (32,256 bytes), you can guarantee misalignment. Though the alignment issue predominately affects RAID disks, where a volume cluster might cross a RAID chunk boundary, there are other boundaries that could be misaligned such as cache lines.	DiskPart: To resolve this issue, use the Diskpart.exe tool to create the disk partition and to specify a starting offset of 2,048 sectors (1 megabyte). A starting offset of 2,048 sectors covers most stripe unit size scenarios. For more information, go to “Disk performance may be slower than expected when you use multiple disks in Windows Server 2003, in Windows XP, and in Windows 2000” http://support.microsoft.com/kb/929491

Symptoms: General poor response from storage subsystem

Applies to:

Windows Server 2003 (all editions) unless otherwise specified
Windows XP (all editions) unless otherwise specified
Windows Server 2000 (all editions) unless otherwise specified

Symptom Details:

Long disk response times: A “LogicalDisk\Avg. Disk Sec/Read” or “LogicalDisk\Avg. Disk Sec/Write” value greater than 15ms though occasional spikes are not necessarily cause for immediate concern.
Low I/O: “\LogicalDisk\Disk Transfers/sec” is well below the number of I/O operations per second that each physical spindle is designed to handle which is typically between 80 to 180 per disk.
Split I/O: “Physical Disk\Split IO/sec can be an indicator of volume fragmentation.
High Queue Lengths, poor response times. LogicalDisk\Average Disk Queue Length is averaging higher than 2-3 plus the number of spindles. At the same time, “LogicalDisk\Avg. Disk Sec/Read” or “LogicalDisk\Avg. Disk Sec/Write” value greater than 15ms are observed.
Low throughput, high number of transfers. “Transfers/second” counter is relatively high, but the overall “Disk Bytes/sec” is low.

Possible Causes	How to Diagnose	Possible Solutions and/or Recommendations
High disk fragmentation	Use the Windows Disk Defragmentation tool to analyze the disk fragmentation. If the disk is more than 20% fragmented, then consider running the defragmentation tool during your next maintenance period. In Performance Monitor, look at the “Split I/O” counter. This indicates that a single request was split into multiple requests, likely as the result of fragmentation.	Defragment the Disk: Use the Windows Disk Defragmentation tool to defragment the disk during your next maintenance period. Note: Some products like Microsoft SQL Server have clustered indexes that are mapped to hard disk clusters, therefore consult your database administrator before defragmenting.
Lack of free space	Use the “\LogicalDisk\% Free Space” counter to determine if the disk has less than 30% free space. % Free Space is the percentage of total usable space on the selected logical disk drive that was free. Performance is not really affected until the available disk drive space is less than 30 percent. When 70 percent of the disk drive is used, the remaining free space is located closer to the disk's spindle at the center of the disk drive, which operates at a lower performance level. Lack of disk free space can cause severe disk performance. Note: Low disk free space disk performance can vary on hardware RAID solutions depending on how the hardware spreads the data on the spindles.	Remove Files: Move or delete unnecessary files from the disk drive. Add Physical Spindles: Add additional disks to the LUN or disk volume. Increase the Partition Size: If contiguous disk space is available the partition can be expanded using Diskpart. This is often an option when disks are added to disk groups in the underlying storage device.
Insufficient number of disks	In many cases the workload presented to a storage device is greater than was originally designed for. Look at the disk queue length counters in Performance Monitor. The rule is that you want: 2 outstanding requests for low-performance disks 3 outstanding requests for high-performance disks (such as Fibre Channel storage devices)	Add Physical Spindles: Most of the time it is a matter of adding physical disks to the disk group that is suffering. The Windows side of the story will vary depending on maintenance windows and so forth. The volume can be extended in Windows using Diskpart, because the new disk capacity will show up as free disk space at the end of the current disk device. The only other option is to backup or move the data, delete and then recreate the partition, making sure to align the partition using Diskpart unless using Windows Vista or later.
Flooding the I/O channel and causing retries or “busy” from the storage device	Windows may be sending too many I/Os at one time to the storage device, resulting in “BUSY” being returned by the storage device until buffers are freed. There is no easy way to determine if this is the case, with the possible exception of iSCSI and using Network Monitor. With direct attached storage you would need a bus analyzer or advanced tracing in order to find out if busy is being returned to the host. With Fibre Channel the only way to determine if a port is being overrun is with a Fibre Channel trace.	Adjust the HBA Queue Length: Fibre Channel: Fibre Channel Host Bus Adapters (HBAs) have settings that control the number of outstanding I/O sent to the storage device. A Storport miniport HBA driver architecture provides a “per LUN” queue of 255 outstanding requests. Most HBAs have a default setting of 32, though OEMs can change this and often do, usually to 16. What we do here is check the current setting, find out how many disk (LUNs) we are sending I/O to, and calculate the overall I/O load. We can try reducing the queue length setting at the HBA and see if this helps improve response time. If lowering the queue depth helps, we know that we were sending too many I/O through the HBA. If this setting makes no change, then there is likely some other issue we need to look at.
Low throughput, high number of transfers	Examine the “Disk Bytes/Transfer”, “Disk Bytes/Read”, and “Disk Bytes/Write” counters. You may find that the overall request size is small, say 9 KB, or that writes are small (8 KB) and reads are as expected. The one thing lacking in Performance Monitor is to tell us where on the disk the bytes are being written. If the disks are constantly performing random reads or write to wildly different locations on the disk, the benefits of cache are lessened.	Adjust Read/Write Ratio: It may be possible to adjust the read/write ratio settings on the storage subsystem cache. This is something the storage vendor would handle, not Microsoft. Adjust Short-Stroking: It may be possible to lessen the spatial effects of random data by a technique known as “short-stroking”. What this means is that in Windows you create a partition that only fills up ½ of the available disk space. The physicals of the disks mean that average seek time will be lessened when there is less distance for the armitures & disk heads to cover. This does mean that some disk space will be sacrificed in order to gain performance. Adjust Element Size: If using RAID, the stripe-unit size, or element size, may not be sized for a smaller workload. The result is that not enough physical disks are involved in the workload. The storage vendors will normally have a handle on how to size the stripe units in the storage groups. If a storage group needs to have its stripe unit size resized, the group usually will need to be recreated. This means that the data will be lost, so be sure it is backed up first. Also, and this point cannot be stressed enough, be sure that the partition is created/recreated to a stripe unit boundary using Diskpart.

More Information

Perfmon Log Capture Interval: Generally speaking, if capturing performance data on a live system using the Windows Performance Monitor, the sampling interval should be kept fairly non-intrusive, such as every 10 seconds. The problem with sampling at 10 seconds or longer is that we tend to miss a lot of data. If in a testing environment we should set the capture interval to 1 second and capture both Physical Disk and Logical Disk counters. If we are capturing at short intervals like 10 seconds or less, we may not want to capture other counters at the same time so as to not impose too much overhead on the system for performance monitoring.

Capturing Logical Disk versus Physical Disk Counters: The other thing to keep in mind with Performance Monitor is that if we are gathering performance data, there is a cost in performance associated with gathering a specific performance counter. At the same time there is little additional host performance cost to go ahead and capture the entire performance object. The point being that when measuring storage performance, go ahead and capture Physical Disk and Logical Disk objects and not just individual performance counters. If the physical disks only have 1 partition per disk, then there is really no need to capture the Logical Disk counters. The exception being of course if you are making use of Mount Points within Windows and you need to measure performance of individual physical disks.

When capturing performance data, there is sometimes a concern about the size of the capture file. If capturing only Physical Disk and Logical Disk counters, even at a 1 second interval, the resulting file will not get to be excessively large. For a cost of 100 MB or so, and depending on the number of disk devices. If capturing only Physical Disk and Logical Disk counters, even at a 1 second interval, the resulting the counter log file will typically not grow, excessively large, perhaps 100 MB or so, depending on the number of disk devices.

References

Ruling Out Disk-Bound Problems
http://technet.microsoft.com/en-us/library/5bcdd349-dcc6-43eb-9dc3-54175f7061ad.aspx
How to Identify a Disk Performance Bottleneck Using the Microsoft Server Performance Advisor (SPA) Tool
http://www.codeplex.com/PerfTesting/Wiki/View.aspx?title=How%20To%3a%20Identify%20a%20Disk%20Performance%20Bottleneck%20Using%20SPA1&referringTitle=How%20Tos
Download Details for Microsoft Service Performance Advisor (SPA)
http://www.microsoft.com/downloads/details.aspx?FamilyID=09115420-8c9d-46b9-a9a5-9bffcd237da2&DisplayLang=en

Written By: Robert Smith

Contributors: Clint Huffman, Jimmy May, and Ken Brumfield

Symptoms: Lack of Free System Page Table Entries (PTEs) and Error Message: ERROR_NO_SYSTEM_RESOURCES (1450)

2008-04-08T01:16:15Z

Introduction

The purpose of this article is to provide prescriptive guidance on how to troubleshoot free system page table entries (PTEs) in regards to Windows performance analysis.

Start with the following performance counters to analyze free system PTE’s:

· \Memory\Free System Page Table Entries

A page table is the data structure used by the Windows Virtual Memory Manager (VMM) to store the mapping between virtual addresses and physical addresses in memory. The performance counter Free System Page Table Entries is the number of page table entries not currently used by the system.

From the process perspective, each element of virtual address conceptually refers to a byte of physical memory. It is the responsibility of the Virtual Memory Manager (VMM) in conjunction with processor memory management unit (MMU) to translate or map each virtual address into a corresponding physical address.

The VMM performs the mapping by dividing the RAM into fixed-size page frames, creating system PTEs to store information about these page frames, and mapping them. System PTEs are small kernel-mode buffers of memory that are used to communicate with the disk I/O subsystem and the network. Each PTE represents a page frame and contains information necessary for the VMM to locate a page.

Note: Troubleshooting System PTE’s is explained in more detail at in the “Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues” article mentioned in the References section below.

Applies to:

Microsoft Windows 2000 Service Pack 1
Microsoft Windows 2000 Service Pack 2
Microsoft Windows 2000 Advanced Server

Symptom Details:

Lack of Free System Page Table Entries (PTEs): Use the “Memory\Free System Page Table Entries” performance counter for values under 5000. Alternatively, the !pte command in the kernel debugger can be used to examine PTEs.
Error Message: ERROR_NO_SYSTEM_RESOURCES (1450): The error message: ERROR_NO_SYSTEM_RESOURCES (1450) appears in a bug check (blue screen) in Windows.

Possible Cause: SCSI miniport driver instructs the Scsiport.sys file to map buffers, the Scsiport.sys file tries to map the buffer that is described by the I/O request packet's MDL

How to Diagnose

· Run “Winver” or Add/Remove Programs to determine the latest service pack applied to windows.

Possible Solutions and/or Recommendations

· Obtain the latest Service Pack for Windows 2000: Apply the latest service pack for Windows 2000. See the following article for more information: http://support.microsoft.com/?id=291228

References:

Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
http://www.microsoft.com/downloads/details.aspx?FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a&DisplayLang=en
“Windows Internals Microsoft Windows Server 2003, Windows XP, and Windows 2000” by Mark E. Russinovich and David Soloman.
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/kb/294418
System Pages
http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx?mfr=true
How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/kb/316739

Contributors:

Clint Huffman, Shane Creamer, Rick Anderson, Maximilian Silva, Matthew Walker, Pavel Lebedynskiy, John Rodriguez, Mike Lagase, Yong Rhee

Symptoms: Lack of Free System Page Table Entries (PTEs) and no system-wide delays

2008-04-08T01:10:06Z

Introduction

The purpose of this article is to provide prescriptive guidance on how to troubleshoot free system page table entries (PTEs) in regards to Windows performance analysis.

Start with the following performance counters to analyze free system PTE’s:

· \Memory\Free System Page Table Entries

Applies to:

32-bit Windows Server 2003 (all editions) unless otherwise specified

Symptom Details:

Lack of Free System Page Table Entries (PTEs): Use the “Memory\Free System Page Table Entries” performance counter for values under 5000. Alternatively, the !pte command in the kernel debugger can be used to examine PTEs.
No periodic system-wide delays: No system-wide delays or hangs that occur regularly or occur during elevated load on the system. This is measured by the user experience and I/O response times of the system.

Possible Cause: Inaccurate data

How to Diagnose

· Run “Winver” or Add/Remove Programs to determine the latest service pack applied to windows.

Possible Solutions and/or Recommendations

· Install Windows Server 2003 Service Pack 1: The Performance Monitor “Memory\Free System Page Table Entries” counter is inaccurate on installations of Windows Server 2003 without Service Pack 1. For more information about this topic, see Microsoft Knowledge Base article 894067 “The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003” (http://go.microsoft.com/fwlink/?linkid=3052&kbid=894067)

References

Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
http://www.microsoft.com/downloads/details.aspx?FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a&DisplayLang=en
“Windows Internals Microsoft Windows Server 2003, Windows XP, and Windows 2000” by Mark E. Russinovich and David Soloman.
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/kb/294418
System Pages
http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx?mfr=true
How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/kb/316739

Contributors:

Clint Huffman, Shane Creamer, Rick Anderson, Maximilian Silva, Matthew Walker, Pavel Lebedynskiy, John Rodriguez, Mike Lagase, Yong Rhee

Symptoms: Lack of Free System Page Table Entries (PTEs) and system-wide delays (I/O request failures)

2008-04-08T00:27:00Z

Introduction

The purpose of this article is to provide prescriptive guidance on how to troubleshoot free system page table entries (PTEs) in regards to Windows performance analysis.

Start with the following performance counters to analyze free system PTE’s:

· \Memory\Free System Page Table Entries

This article is grouped by symptoms, then by possible causes.

Symptoms: Lack of Free System Page Table Entries (PTEs) and system-wide delays (I/O request failures)

Applies to:

32-bit Windows Server 2003 (all editions) unless otherwise specified
32-bit Windows XP (all editions) unless otherwise specified
32-bit Windows Server 2000 (all editions) unless otherwise specified

Symptom Details:

· Lack of Free System Page Table Entries (PTEs): Use the “Memory\Free System Page Table Entries” performance counter for values under 5000. Alternatively, the !pte command in the kernel debugger can be used to examine PTEs.

Periodic system-wide delays: System-wide delays or hangs that occur regularly or occur during elevated load on the system. This is measured by the user experience and I/O response times of the system.

Possible Cause: Use of Physical Address Extension (PAE) Kernel

How to Diagnose

· To determine if the Windows 2003 server is booted with the PAE kernel, by checking for a value of 1 in the registry key, “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PhysicalAddressExtension”. Alternatively, the boot.ini file can be searched for the PAE boot switch.
Note: when execution protection is enabled on 32-bit Windows, the system automatically boots in PAE mode (automatically selecting the PAE kernel).

· The PAE switch allows the Windows operating system to address more than 4GB’s of physical memory and in certain environments the PAE switch is automatically enabled. On x86 systems, the page table index is 10 bits wide (9 on PAE), allowing you to reference up to 1024 4-byte PTEs. Therefore, the /PAE switch causes PTE entries to use twice the normal allotted virtual address space. For more information, see “Microsoft Windows Internals” by Mark E. Russinovich and David Soloman.

Possible Solutions and/or Recommendations

· Migrate to a 64-bit Windows: 64-bit Windows has much higher amount of memory available System PTE’s. Specifically 64-bit has a maximum memory size of 128GBs for PTE’s while 32-bit Windows has a 660MB maximum for PTE’s. For more information, see http://support.microsoft.com/kb/294418.

· Migrate to Windows Server 2008 or Windows Vista: Windows Vista and Windows Server 2008 use dynamically memory pools to better manage system PTE memory. For more information, see http://support.microsoft.com/kb/294418.

· Remove the /PAE switch: If not needed, then remove the /PAE switch from the boot.ini file.

Possible Cause: Use of the /3GB switch

How to Diagnose

· The /3GB switch significantly reduces the number of free system PTE’s. To check if the /3GB switch is enabled use one of the following methods:

o Open the boot.ini file and look for the /3GB switch on the most recently used boot option.

o Check the value of the registry key: “HKLM\SYSTEM\CurrentControlSet\Control\SystemStartOptions”

o Check the value of the WMI property root\cimv2\Win32_OperatingSystem.MaxProcessMemorySize for a value close to 3,145,536 (3GBs) and greater than 2,097,024 (2GBs).

· Refer to the chart below for the starting amount of free PTE’s in different operating system configurations. Note the different in the amount of free PTE’s when the /3GB switch is enabled.

Possible Solutions and/or Recommendations

· Use the USERVA switch: Use the /USERVA switch on Windows Server 2003 to give back some of the memory to the kernel specifically for system PTE’s. For more information on the USERVA switch, go to: “How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB” http://support.microsoft.com/kb/316739.
Note: For a Windows 2000 server in a /3GB scenario the SystemPages registry setting is used to accomplish the same effect as the /userva switch. For more information, go to http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx.

· Remove the /3GB switch: If not needed, then remove the /3GB switch from the boot.ini file.

· Reduce the amount of physical memory: See “To much physical memory on 32-bit Windows” below.

Possible Cause: Too much physical memory

How to Diagnose

· Refer to the “Estimated Kernel Resources Chart” in the More Information section for the starting amount of free PTE’s in different operating system configurations. Note the how increasing the amount of physical memory on the server reduces the number of free PTE’s on the server.

Possible Solutions and/or Recommendations

· Reduce the amount of physical memory: Reduce the amount of physical memory to free up more virtual memory for system PTE’s.

Possible Cause: High resource consumption and/or poorly written device drivers

How to Diagnose

Possible Solutions and/or Recommendations

· Remove any unnecessary third-party drivers and drivers not listed on the Hardware Compatibility List (HCL): Drivers that are not on the Hardware Compatibility List (HCL) have a higher likelihood of causing system problems and/or misuse of system resources. For more information about the Microsoft Hardware Compatibility List (HCL) go to, http://winqual.microsoft.com/hcl/.

· Use /BASEVIDEO: Use the /BASEVIDEO boot switch in the boot.ini file or a generic video driver to free up system page table entries. Video boards use the system page table entries to map their buffers in kernel space. This usage competes with the need for system page table entries by Microsoft Exchange.

· Use the USERVA switch: Use the /USERVA switch on Windows Server 2003 to give back some of the memory to the kernel specifically for system PTE’s. For more information on the USERVA switch, go to: “How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB” http://support.microsoft.com/kb/316739. For a Windows 2000 server in a /3GB scenario the SystemPages registry setting is used to accomplish the same effect as the /userva switch.

· Remove the /3GB switch: If not needed, then remove the /3GB switch from the boot.ini file.

· Reduce the amount of physical memory: See “To much physical memory on 32-bit Windows” below.

Possible Cause: High number of mailboxes on Microsoft Exchange Server

How to Diagnose

Possible Solutions and/or Recommendations

· Reduce the number of mailboxes on the server and/or remove the public folder role from the mailbox server.

High number of Microsoft Exchange mailbox stores that have the public folder server set as their default public store.

· Reduce the number of mailbox stores that have the public folder server set as their default public store. This action will reduce the number of clients (and, as a result, the number of user sessions) on the public folder server.

Estimated Kernel Resources Chart

This chart shows the estimated, initial maximum size of kernel resources on 32-bit Windows Server 2003 computers when using PAE, /3GB, with booted different physical memory sizes. Use the “!vm” command when doing a kernel debug to find the actual kernel resources.

Memory

Default ( /PAE for 6-16GB )

/3GB

1GB

Free System PTE: 51k

Paged Pool: 282MB

Non Paged Pool: 212MB

Free System PTE: 32k

Paged Pool: 163MB

Non Paged Pool: 131MB

2GB

Free System PTE: 196k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 16k

Paged Pool: 262MB

Non Paged Pool: 131MB

3GB

Free System PTE: 195k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 14k

Paged Pool: 262MB

Non Paged Pool: 131MB

4GB

Free System PTE: 106k

Paged Pool: 336MB

Non Paged Pool: 285MB

Free System PTE: 15k

Paged Pool: 258MB

Non Paged Pool: 154MB

6GB

Free System PTE: 186k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 239MB

Non Paged Pool: 131MB

8GB

Free System PTE: 182k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 225MB

Non Paged Pool: 131MB

12GB

Free System PTE: 175k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 196MB

Non Paged Pool: 131MB

16GB

Free System PTE: 167k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 169MB

Non Paged Pool: 131MB

References

Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
http://www.microsoft.com/downloads/details.aspx?FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a&DisplayLang=en
“Windows Internals Microsoft Windows Server 2003, Windows XP, and Windows 2000” by Mark E. Russinovich and David Soloman.
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/kb/294418
System Pages
http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx?mfr=true
How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/kb/316739

Contributors:

Clint Huffman, Shane Creamer, Rick Anderson, Maximilian Silva, Matthew Walker, Pavel Lebedynskiy, John Rodriguez, Mike Lagase, Yong Rhee

Welcome to the Counter of the Week blog

2008-04-08T00:01:00Z

This is an initiative to document common performance issues through Microsoft Windows performance counters. Think of this as the "troubleshooting" section in the back of the user manual. While the content in these documents are nothing new the format of these documents are to help you quickly find symptoms of an issue, how to diagnose the issue, and how to solve issue (or next steps in troubleshooting). I hope you find these useful!

Symptoms: Lack of Free System Page Table Entries (PTEs), system-wide delays (I/O request failures), and low on Paged Pool Memory and/or Non-paged Pool Memory on 32-bit Windows

2008-04-07T22:03:00Z

Introduction

The purpose of this article is to provide prescriptive guidance on how to troubleshoot free system page table entries (PTEs) in regards to Windows performance analysis.

Start with the following performance counters to analyze free system PTE’s:

\Memory\Free System Page Table Entries

Applies to:

32-bit Windows Server 2003 (all editions) unless otherwise specified
32-bit Windows XP (all editions) unless otherwise specified
32-bit Windows Server 2000 (all editions) unless otherwise specified

Symptom Details:

Lack of Free System Page Table Entries (PTEs): Use the “Memory\Free System Page Table Entries” performance counter for values under 5000. Alternatively, the !pte command in the kernel debugger can be used to examine PTEs.
Periodic system-wide delays: System-wide delays or hangs that occur regularly or occur during elevated load on the system. This is measured by the user experience and I/O response times of the system.
Low on Paged Pool Memory: Usage of more than 80% of the maximum size of kernel paged pool kernel memory. Use “!vm” command while in a kernel debug to determine the size of the kernel paged pool memory, then compare it to the “\Memory\Pool Paged Bytes” performance counter. A chart of estimated sizes is provided below in the More Information section of this article.
and/or
Low on Non-Paged Pool Memory: Usage of more than 80% of the maximum size of kernel non-paged pool kernel memory. Use “!vm” command while in a kernel debug to determine the size of the kernel non-paged pool memory, then compare it to the “\Memory\Pool Nonpaged Bytes” performance counter. A chart of estimated sizes is provided below in the More Information section of this article.

Possible Cause: Use of Physical Address Extension (PAE) Kernel

How to Diagnose

To determine if the Windows 2003 server is booted with the PAE kernel, by checking for a value of 1 in the registry key, “HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PhysicalAddressExtension”. Alternatively, the boot.ini file can be searched for the PAE boot switch.
Note: when execution protection is enabled on 32-bit Windows, the system automatically boots in PAE mode (automatically selecting the PAE kernel).
The PAE switch allows the Windows operating system to address more than 4GB’s of physical memory and in certain environments the PAE switch is automatically enabled. On x86 systems, the page table index is 10 bits wide (9 on PAE), allowing you to reference up to 1024 4-byte PTEs. Therefore, the /PAE switch causes PTE entries to use twice the normal allotted virtual address space. For more information, see “Microsoft Windows Internals” by Mark E. Russinovich and David Soloman.

Possible Solutions and/or Recommendations

Migrate to a 64-bit Windows: 64-bit Windows has much higher amount of memory available System PTE’s. Specifically 64-bit has a maximum memory size of 128GBs for PTE’s while 32-bit Windows has a 660MB maximum for PTE’s. For more information, see http://support.microsoft.com/kb/294418.
Migrate to Windows Server 2008 or Windows Vista: Windows Vista and Windows Server 2008 use dynamically memory pools to better manage system PTE memory. For more information, see http://support.microsoft.com/kb/294418.
Remove the /PAE switch: If not needed, then remove the /PAE switch from the boot.ini file.

Possible Cause: Use of the /3GB switch

How to Diagnose

The /3GB switch significantly reduces the number of free system PTE’s. To check if the /3GB switch is enabled use one of the following methods:
Open the boot.ini file and look for the /3GB switch on the most recently used boot option.
Check the value of the registry key: “HKLM\SYSTEM\CurrentControlSet\Control\SystemStartOptions”
Check the value of the WMI property root\cimv2\Win32_OperatingSystem.MaxProcessMemorySize for a value close to 3,145,536 (3GBs) and greater than 2,097,024 (2GBs).
Refer to the “Estimated Kernel Resources Chart” in the More Information section for the starting amount of free PTE’s in different operating system configurations. Note the different in the amount of free PTE’s when the /3GB switch is enabled.

Possible Solutions and/or Recommendations

Use the USERVA switch: Use the /USERVA switch on Windows Server 2003 to give back some of the memory to the kernel specifically for system PTE’s. For more information on the USERVA switch, go to: “How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB” http://support.microsoft.com/kb/316739.
Note: For a Windows 2000 server in a /3GB scenario the SystemPages registry setting is used to accomplish the same effect as the /userva switch. For more information, go to http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx.
Remove the /3GB switch: If not needed, then remove the /3GB switch from the boot.ini file.
Reduce the amount of physical memory: See “To much physical memory on 32-bit Windows” below.
Migrate to a 64-bit Windows: 64-bit Windows has much higher amount of memory available System PTE’s. Specifically 64-bit has a maximum memory size of 128GBs for PTE’s while 32-bit Windows has a 660MB maximum for PTE’s. For more information, see http://support.microsoft.com/kb/294418.
Migrate to Windows Server 2008 or Windows Vista: Windows Vista and Windows Server 2008 use dynamically memory pools to bettermanage system PTE memory. For more information, see http://support.microsoft.com/kb/294418.

Possible Cause: Too much physical memory

How to Diagnose

Refer to the “Estimated Kernel Resources Chart” in the More Information section for the starting amount of free PTE’s in different operating system configurations. Note the how increasing the amount of physical memory on the server reduces the number of free PTE’s on the server.

Possible Solutions and/or Recommendations

Reduce the amount of physical memory: Reduce the amount of physical memory to free up more virtual memory for system PTE’s.
Migrate to a 64-bit Windows: 64-bit Windows has much higher amount of memory available System PTE’s. Specifically 64-bit has a maximum memory size of 128GBs for PTE’s while 32-bit Windows has a 660MB maximum for PTE’s. For more information, see http://support.microsoft.com/kb/294418.
Migrate to Windows Server 2008 or Windows Vista: Windows Vista and Windows Server 2008 use dynamically memory pools to better manage system PTE memory. For more information, see http://support.microsoft.com/kb/294418.

Possible Cause: High resource consumption and/or poorly written device drivers

How to Diagnose

Possible Solutions and/or Recommendations

Remove any unnecessary third-party drivers and drivers not listed on the Hardware Compatibility List (HCL): Drivers that are not on the Hardware Compatibility List (HCL) have a higher likelihood of causing system problems and/or misuse of system resources. For more information about the Microsoft Hardware Compatibility List (HCL) go to, http://winqual.microsoft.com/hcl/.
Use /BASEVIDEO: Use the /BASEVIDEO boot switch in the boot.ini file or a generic video driver to free up system page table entries. Video boards use the system page table entries to map their buffers in kernel space. This usage competes with the need for system page table entries by Microsoft Exchange.
Use the USERVA switch: Use the /USERVA switch on Windows Server 2003 to give back some of the memory to the kernel specifically for system PTE’s. For more information on the USERVA switch, go to: “How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB” http://support.microsoft.com/kb/316739. For a Windows 2000 server in a /3GB scenario the SystemPages registry setting is used to accomplish the same effect as the /userva switch.
Remove the /3GB switch: If not needed, then remove the /3GB switch from the boot.ini file.
Reduce the amount of physical memory: See “To much physical memory on 32-bit Windows” below.
Migrate to a 64-bit Windows: 64-bit Windows has much higher amount of memory available System PTE’s. Specifically 64-bit has a maximum memory size of 128GBs for PTE’s while 32-bit Windows has a 660MB maximum for PTE’s. For more information, see http://support.microsoft.com/kb/294418.
Migrate to Windows Server 2008 or Windows Vista: Windows Vista and Windows Server 2008 use dynamically memory pools to better manage system PTE memory. For more information, go to http://msdn2.microsoft.com/en-us/library/bb870880(VS.85).aspx.

Estimated Kernel Resources Chart

Memory

Default ( /PAE for 6-16GB )

/3GB

1GB

Free System PTE: 51k

Paged Pool: 282MB

Non Paged Pool: 212MB

Free System PTE: 32k

Paged Pool: 163MB

Non Paged Pool: 131MB

2GB

Free System PTE: 196k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 16k

Paged Pool: 262MB

Non Paged Pool: 131MB

3GB

Free System PTE: 195k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 14k

Paged Pool: 262MB

Non Paged Pool: 131MB

4GB

Free System PTE: 186k

Paged Pool: 336MB

Non Paged Pool: 285MB

Free System PTE: 15k

Paged Pool: 258MB

Non Paged Pool: 154MB

6GB

Free System PTE: 186k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 239MB

Non Paged Pool: 131MB

8GB

Free System PTE: 182k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 225MB

Non Paged Pool: 131MB

12GB

Free System PTE: 175k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 196MB

Non Paged Pool: 131MB

16GB

Free System PTE: 167k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 169MB

Non Paged Pool: 131MB

References

Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
http://www.microsoft.com/downloads/details.aspx?FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a&DisplayLang=en
“Windows Internals Microsoft Windows Server 2003, Windows XP, and Windows 2000” by Mark E. Russinovich and David Soloman.
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/kb/294418
System Pages
http://technet2.microsoft.com/windowsserver/en/library/c5ccbaec-f552-4f61-a488-8ee3330d1eeb1033.mspx?mfr=true
How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/kb/316739
Active Directory Integration Tuning
http://blogs.technet.com/cotw/archive/2008/04/07/symptoms-lack-of-free-system-page-table-entries-ptes-system-wide-delays-i-o-request-failures-and-low-on-paged-pool-memory-and-or-non-paged-pool-memory-on-32-bit-windows.aspx

Contributors:

Clint Huffman, Shane Creamer, Rick Anderson, Maximilian Silva, Matthew Walker, Pavel Lebedynskiy, John Rodriguez, Mike Lagase, Yong Rhee.