Thoughts from the EPS Windows Server Performance Team
As promised - here's the long awaited post on the infamous /3GB switch. At least once a week we have this discussion with a Systems Administrator somewhere who has this set in the boot.ini file on all of the servers but doesn't know why. Maybe someone added it to the server build process at some point or perhaps someone read about the /3GB switch somewhere and thought that it would improve performance or enable them to see the full 4GB or 8GB of physical memory installed on the system. So let's start by dispelling a few /3GB myths:
OK - so what does the /3GB switch really do? If you recall, from the Memory Management 101 post, Windows 32-bit Operating Systems implement a virtual memory system based on a flat 32-bit address space. This address space translates into 4GB of virtual memory - no more, and no less. By default, the address space is divided into two equal chunks. 2GB for the Kernel and 2GB for User-mode processes. The Kernel space is common for all applications and the User-mode processes each get their own 2GB address space to work with.
So where does the /3GB switch come in? Windows 2000 Advanced Server, Windows 2000 Datacenter Server, Windows XP SP2 and later and all versions of Windows Server 2003 support the /3GB boot-time option to allow the user mode address space to grow to 3GB. The /3GB option was intended as a short term solution to allow applications such as database servers to maintain more data in memory than a 2GB address space allowed. However, using the /3GB method to increase the user-mode memory space comes at a cost.
Remember that we only have a 4GB total address space to work with. If we have to allocate an additional 1GB of this address space to the user-mode space, then the System space is cut in half. Drivers, Heap, Paged & NonPaged Memory all have only half the resources to work with now. However, because of the way memory mapping works, cutting the kernel space in half does a lot more than just reducing the address space. Many of the structures within the kernel virtual memory space are cut back by far more than 50%. For example, we took a Windows Server 2003 Enterprise R2 machine with 1GB of RAM installed and compared some values with and without the /3GB switch enabled.
As you can see, the Free System PTE's drops by over 200,000. Keep in mind that this is only a test server that isn't under any sort of load. A machine under medium to heavy load could quite easily run out of free PTE's - meaning that the system can no longer map system pages such as I/O space, kernel stacks and memory descriptor lists. In addition, look at NonPaged Pool after the /3GB parameter is enabled. The NonPaged Pool maximum is only 130MB. Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any IRQL. Once we run into NonPaged pool depletion, we're looking at our old friend, the Event ID 2019.
OK - so let's quickly recap what we've discussed so far. The /3GB switch is not related to the amount of physical memory installed in a system. It is useful if you have an application that can take advantage of a larger address space. For a process to access the full 3GB address space, the image file must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header.
If the flag is not set in the image header, then the OS reserves the third gigabyte so that the application won't see virtual addresses greater than 0x7FFFFFFF. You set this flag by specifying the linker flag /LARGEADDRESSAWARE when building the executable. This flag has no effect when running the application on a system with a 2-GB user address space. Therefore if you enable the /3GB switch, then applications that do not have this flag set can only use the standard 2GB of User mode memory, and the Kernel is still limited to the 1GB space - which means that 1GB of virtual memory is basically wasted!
Earlier, we mentioned that there were some applications that benefit from the use of the /3GB switch. The predominant scenario where the /3GB switch is not only recommended, but actually required is with Microsoft Exchange servers that house public folders and / or mailboxes. Due to the way that Exchange handles memory management, the additional 1GB of user mode memory is required to ensure that the Store.exe process does not run out of virtual address space. However, in order to guard against System PTE depletion, the system can be tuned using the /USERVA switch in conjunction with the /3GB switch. This tunes the actual amount of memory for the Address space. For example, setting USERVA=3030 means that the process space is actually only 3,030MB and not 3,072MB (which would be the process space with only the /3GB switch present). The additional 42MB is used for System PTE usage. The USERVA value can safely be tweaked as low as 2800 - however, if it is necessary to set USERVA this low, then you probably want to start thinking about scaling your Exchange environment to spread the load!
Ideally, there should always be at least 24,000 Free System PTE's at boot time. Depending on server workload there may be wide variances in the amount of Free System PTE's during the course of a normal duty cycle, so it may be necessary to implement some long-term monitoring to ensure that the server does not fall below 10,000 Free PTE's.
So there you have it - the /3GB switch demystified. Hopefully this post, as well as the others in our Memory Management series will help you understand a bit more about how and why the Operating System behaves the way it does. Remember that the /3GB switch is intended to be used in very specific instances - and now you know why!
- Tim Newton
Is there some way to (programatically) find out when the OS is booted in the /3GB mode? The 32-bit version of my application sets a hard limit of 1700 MB after which it does not try to allocate any more memory. I'd like to switch this to 2700 MB when the /3GB switch is available. The best I can come up with till now is to read the boot.ini file, but I am wondering if there is anything clever out there, that I can do.
Thank you for the elaborate discussions on Windows Memory.
helped me to clear a lot of doubt about the /3GB switch
Keep up the good work, Tim
Sashank: You can use powershell/wmi to get the value from boot.ini.
I Have a Windows 2003 SP2 Terminal Server with 2 GB in RAM and i haven't found a good advice in order to use (or not) the /3GB /Userva:3030 in a Terminal Server.
Following on from our post on troubleshooting a basic application crash , it's time to start troubleshooting
In previous posts, we've discussed the Basics of Memory Management , Pool Resources and of course the
In response to Gonzalo's question, the answer is NO. You do not want to enable /3GB or USERVA unless you have a specific application need to do so, such as in an Exchange environment. On a terminal server, the last thing you want to do is cripple the kernel by enabling /3GB.
"I Have a Windows 2003 SP2 Terminal Server with 2 GB in RAM and i haven't found a good advice in order to use (or not) the /3GB /Userva:3030 in a Terminal Server."
My servers have 17 GB of RAM and the /3GB /PAE switches enabled (vendors recommendation).
The BIOS shows 17 GB, but the OS only displays 15 GB. I thought the /3GB would limit to 16, but why 15 GB? Any ideas?
You wrote about Microsoft Exchange servers. Using Catia, a high end auto / aero CAD package, the recomendation has been to run 3 GB by default. Is this a problem? We commonly work with data files in the hundreds of MBs
In previous posts we've discussed the basics of memory management including an overview of kernel and
Is the /3GB Switch required in the 32-Bit Version of Windows 2003 R2????
Thanks for that info.
Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either
After running the EXBPA, it recomends adding the /3GB to the Win2003 domain contollers in addition to the Exchange 2003 mailbox servers. What are the benifits to adding the /3GB to the DCs?
Hi guy's, we get asked this a lot in the Terminal Services/Citrix world, where misguides admins add /3GB switch and cripple their servers.
Can you guy's confirm that in a 32bit Windows 2003 terminal services environment that this is a totally inappropriate modification?