This is the second flash about Windows 2003 kernel memory issues and Exchange 2003. The first flash provided technical background. This flash discusses important hardware issues that may affect Windows kernel memory and cause Exchange instability. The next flash (available in about 10 days) will explain the effect of large user security tokens on kernel memory usage.
Servers that implement hot-add memory or that are configured with more than 4 gigabytes of physical RAM can require large amounts of additional kernel memory to support these resources. This reduces the kernel memory available for applications. Even though disk, network and processor bandwidth may be more than adequate, lack of kernel resources can be the bottleneck that limits application scalability.
The impact of this issue for Exchange is primarily on large Exchange 2003 mailbox or public folder servers that host or accept connections from thousands of clients. In some cases, sub-optimal large memory configuration on a large Exchange 2003 server can reduce the server's carrying capacity by half and cause serious server instability.
If kernel memory issues are not adequately addressed, it may be necessary to reduce the server load by removing large numbers of mailboxes or to reduce the number of concurrent client connections to the server.
Symptoms of kernel memory exhaustion include:
This flash provides recommendations for optimizing the use of kernel memory on large memory hardware running Exchange Server 2003. Other applications may have different requirements and recommendations.
CALL TO ACTION
The Exchange Server Best Practices Analyzer Tool (ExBPA) has been updated in configuration file version 220.127.116.11 to identify and make recommendations for optimizing the majority of hardware platforms affected by these issues. All customers should run ExBPA and implement its memory optimization recommendations for Exchange Server 2003 running on Windows 2003.
Customers may also manually implement the optimization recommendations described below in this flash.
NOTE: Windows 2000 and Exchange 2000 are not greatly impacted by these issues because (1) Windows 2000 does not support these hardware features as fully as does Windows 2003, and (2) Exchange 2000 does not run on Windows 2003.
FREQUENTLY ASKED QUESTIONS (FAQ)
Which hardware configurations make additional demands on kernel memory?
Microsoft has identified two increasingly common hardware features that have significant impact on Windows kernel memory:
Large memory support is available in Windows Server 2003 and in Windows 2000
Windows kernel memory must be allocated to support addressability for these new memory features. In the case of hot-add memory, the kernel impact is proportional not just to the actual amount of RAM installed, but to the theoretical maximum that could be installed.
To give you an idea of how significant a problem this can be, a server that enables support for 64 gigabytes of hot-add RAM can reduce a server's initial available kernel paged pool memory by more than 30%.
What should I do to increase the amount of kernel memory available for Exchange?
IMPORTANT NOTE: The following recommendations are in addition to the general memory tuning recommendations described in this Microsoft Knowledge Base article:
How to optimize memory usage in Exchange Server 2003
Recommendations 1 and 2 below should be followed for all Exchange servers.
Recommendations 3 and 4 apply only to servers that support hot-add memory. If you are running Windows 2003 without a service pack, you should evaluate both recommendation 3 and recommendation 4 and implement them both if they are applicable. If you are running Windows 2003 with Service Pack 1 or later you do not need to implement recommendation 3. You should evaluate recommendation 4 and implement it if it is applicable.
ExBPA detects Exchange server configuration problems and recommends solutions and optimizations. ExBPA checks for memory configuration optimizations beyond those described in this flash, and includes best practices and optimizations beyond those recommended in the Microsoft Knowledge Base. It is a good idea to run ExBPA after making configuration changes to ensure that the changes were correctly applied.
If the server already has more than 4GB of memory installed, adjust the server settings according to the following documentation: http://technet.microsoft.com/en-us/library/aa997419.aspx
Contact the manufacturer of the server for specific instructions for disabling hot-add memory. If hot-add memory cannot be disabled, then ensure that the /NOPAE switch is set in the server's boot.ini file.
This recommendation applies regardless of whether you have installed any service pack.
Setting the following registry key will disable Windows memory reservations for RAM not already installed in the server. You must reboot the server after setting this key before it will take effect. If you install additional RAM in the server, another reboot will be required before the additional RAM is recognized by Windows.
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
Value Data: 1
For Windows 2003 with no service pack installed, the DynamicMemory setting will be ignored if the /3GB switch is set. The DynamicMemory setting works if the /3GB switch is absent. This problem has been corrected in Service Pack 1 for Windows 2003.
Because Exchange back-end servers typically run with the /3GB switch, the DynamicMemory setting will be ineffective for many Exchange servers running on Windows 2003 with no service pack installed. In these cases, you must follow recommendation 3 above to control hot-add RAM on Windows 2003 with no service pack installed.
NOTE: We still recommend that you configure the DynamicMemory setting on Windows 2003 with no service pack. Although the setting may have no immediate impact, it will be immediately effective after upgrading to a Windows service pack. You may also set this key if you are unsure whether your server supports hot-add memory. The presence of the key will do no harm on servers that do not support hot-add memory.
How can I monitor kernel memory on my server?
For more information about determining how much initial kernel memory is allocated on your server, and how it is allocated between various kernel memory pools, please see this article in the Microsoft Knowledge Base:
The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003
For general information about monitoring and troubleshooting memory usage on an Exchange server, please refer to this white paper:
Troubleshooting Exchange Server 2003 Performance
What if I want to leave hot-add memory enabled? Or what if I host other applications on the same server that could make use of additional RAM?
Exchange 2003 is compatible with both hot-add and PAE memory. There is no hard requirement that either of these features be disabled on an Exchange server. The problem is that these features contend with Exchange and other applications and services for kernel resources. Rather than disabling these features, you could also choose to reduce the application load on the server until kernel memory issues are no longer encountered.
If Exchange does not use PAE, then why shouldn't I disable PAE on every Exchange server?
If an Exchange 2003 server is running on Windows 2003 Service Pack 1, you should not disable PAE.
PAE must be enabled to take advantage of the support for hardware-based Data Execution Prevention (DEP) that was introduced in Windows 2003 Service Pack 1. DEP is a security feature that prevents several common buffer overrun attacks from executing, regardless of whether particular software is vulnerable to the attack.
For more information about DEP, please refer to this Microsoft Knowledge Base article:
A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003
Windows 2003 Service Pack 1 will detect AMD or Intel processors that support DEP and enable the PAE kernel even if the /NOPAE switch is set in the server's boot.ini file. (To disable PAE on servers that support DEP, you must also set the /EXECUTE switch in the server's boot.ini file.)
For versions of Windows prior to Windows 2003 Service Pack 1, Microsoft's advice is the opposite: You should not enable PAE on Exchange servers running on previous versions of Windows.
There is negligible benefit for Exchange performance and memory management from enabling PAE on previous versions of Windows, and there may be significant drawbacks in terms of kernel memory management.
If other applications on an Exchange server installed on a previous version of Windows require PAE, then you must install this update before enabling PAE:
If your server supports PCI Express chipsets, and your server has 4 GB or more of RAM installed, then PAE must be enabled to prevent 256 megabytes or more of server RAM from being "hidden." This is because PCI Express reserves a large block of memory for its own use. Enabling PAE will allow the operating system to replace this lost memory with RAM mapped above the 4 gigabyte boundary. Enabling this memory will not provide a significant improvement in Exchange performance.
Can I increase kernel memory by removing the /3GB switch from the server's boot.ini file?
Microsoft does not recommend removing the /3GB switch if your server meets the criteria for which this switch is recommended. These criteria are described here:
Removing the /3GB switch will increase the initial amounts of memory available for kernel memory pools. For example, it will increase the allocation for paged pool memory from approximately 250 megabytes to 350 megabytes. However, this benefit will come at the expense of the loss of 1 gigabyte of user mode virtual memory.
If you remove the /3GB switch for a large Exchange mailbox server, you must also reduce the number of mailboxes hosted on the server below the point where virtual memory fragmentation errors are reported. Removing the /3GB switch reduces the size of the Exchange cache buffers by half and this will impact performance.