Thoughts from the EPS Windows Server Performance Team
As promised - here's the long awaited post on the infamous /3GB switch. At least once a week we have this discussion with a Systems Administrator somewhere who has this set in the boot.ini file on all of the servers but doesn't know why. Maybe someone added it to the server build process at some point or perhaps someone read about the /3GB switch somewhere and thought that it would improve performance or enable them to see the full 4GB or 8GB of physical memory installed on the system. So let's start by dispelling a few /3GB myths:
OK - so what does the /3GB switch really do? If you recall, from the Memory Management 101 post, Windows 32-bit Operating Systems implement a virtual memory system based on a flat 32-bit address space. This address space translates into 4GB of virtual memory - no more, and no less. By default, the address space is divided into two equal chunks. 2GB for the Kernel and 2GB for User-mode processes. The Kernel space is common for all applications and the User-mode processes each get their own 2GB address space to work with.
So where does the /3GB switch come in? Windows 2000 Advanced Server, Windows 2000 Datacenter Server, Windows XP SP2 and later and all versions of Windows Server 2003 support the /3GB boot-time option to allow the user mode address space to grow to 3GB. The /3GB option was intended as a short term solution to allow applications such as database servers to maintain more data in memory than a 2GB address space allowed. However, using the /3GB method to increase the user-mode memory space comes at a cost.
Remember that we only have a 4GB total address space to work with. If we have to allocate an additional 1GB of this address space to the user-mode space, then the System space is cut in half. Drivers, Heap, Paged & NonPaged Memory all have only half the resources to work with now. However, because of the way memory mapping works, cutting the kernel space in half does a lot more than just reducing the address space. Many of the structures within the kernel virtual memory space are cut back by far more than 50%. For example, we took a Windows Server 2003 Enterprise R2 machine with 1GB of RAM installed and compared some values with and without the /3GB switch enabled.
As you can see, the Free System PTE's drops by over 200,000. Keep in mind that this is only a test server that isn't under any sort of load. A machine under medium to heavy load could quite easily run out of free PTE's - meaning that the system can no longer map system pages such as I/O space, kernel stacks and memory descriptor lists. In addition, look at NonPaged Pool after the /3GB parameter is enabled. The NonPaged Pool maximum is only 130MB. Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any IRQL. Once we run into NonPaged pool depletion, we're looking at our old friend, the Event ID 2019.
OK - so let's quickly recap what we've discussed so far. The /3GB switch is not related to the amount of physical memory installed in a system. It is useful if you have an application that can take advantage of a larger address space. For a process to access the full 3GB address space, the image file must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header.
If the flag is not set in the image header, then the OS reserves the third gigabyte so that the application won't see virtual addresses greater than 0x7FFFFFFF. You set this flag by specifying the linker flag /LARGEADDRESSAWARE when building the executable. This flag has no effect when running the application on a system with a 2-GB user address space. Therefore if you enable the /3GB switch, then applications that do not have this flag set can only use the standard 2GB of User mode memory, and the Kernel is still limited to the 1GB space - which means that 1GB of virtual memory is basically wasted!
Earlier, we mentioned that there were some applications that benefit from the use of the /3GB switch. The predominant scenario where the /3GB switch is not only recommended, but actually required is with Microsoft Exchange servers that house public folders and / or mailboxes. Due to the way that Exchange handles memory management, the additional 1GB of user mode memory is required to ensure that the Store.exe process does not run out of virtual address space. However, in order to guard against System PTE depletion, the system can be tuned using the /USERVA switch in conjunction with the /3GB switch. This tunes the actual amount of memory for the Address space. For example, setting USERVA=3030 means that the process space is actually only 3,030MB and not 3,072MB (which would be the process space with only the /3GB switch present). The additional 42MB is used for System PTE usage. The USERVA value can safely be tweaked as low as 2800 - however, if it is necessary to set USERVA this low, then you probably want to start thinking about scaling your Exchange environment to spread the load!
Ideally, there should always be at least 24,000 Free System PTE's at boot time. Depending on server workload there may be wide variances in the amount of Free System PTE's during the course of a normal duty cycle, so it may be necessary to implement some long-term monitoring to ensure that the server does not fall below 10,000 Free PTE's.
So there you have it - the /3GB switch demystified. Hopefully this post, as well as the others in our Memory Management series will help you understand a bit more about how and why the Operating System behaves the way it does. Remember that the /3GB switch is intended to be used in very specific instances - and now you know why!
- Tim Newton
...but not on a SBS box.
Please do not use the /3 switch there.
...I Hope! Well done to the performance team for their recent Windows Server articles, in particular
I see a lot of customers unnecessarily using the boot.ini /3GB switch. Explaining when and when not to
Keep in mind that the /Userva switch was introduced in Windows 2003 and does not work with Windows 2000.
You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together.
"You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together."
That has not been my experience, but we only used the /PAE and /3GB tuning switch on one particular set of SQL cluster servers.
"Brad Rutkowski said:
You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together."
This is incorrect. Our Exchange 2003 servers use the following and are quite happy.
/PAE /3GB /USERVA=3030* /BASEVIDEO
*2970 in some cases after some PTE tuning
Without /PAE we never see over 3.75GB of physical memory installed on this hardware set, so it helps out a lot.
>/3GB won't enable you to see the additional 4GB or 8GB of RAM you added to your new server
>/3GB doesn't necessarily make your application 50% more efficient
>/3GB should not be a standard for your environment (there are exceptions, and we'll get to those)
From the writing, I can't immediately tell: are these the myths? Or are these the "dispellations" ?
The two sentences preceding the ones you cited should provide the context:
"At least once a week we have this discussion with a Systems Administrator somewhere who has this set in the boot.ini file on all of the servers but doesn't know why. Maybe someone added it to the server build process at some point or perhaps someone read about the /3GB switch somewhere and thought that it would improve performance or enable them to see the full 4GB or 8GB of physical memory installed on the system"
So, if you think of the bullet points as factual statements:
- /3GB should not be a standard for your environment (except in certain circumstances)
- /3GB won't enable you to see the additional 4GB or 8GB of RAM
- /3GB does not necessarily make your application 50% more efficient
Hope this clears up the misunderstanding!
Thanks for the feedback
No sure how I found this blog but its damm good. Check out these that I have been reading today! IE7
""Brad Rutkowski said:
Without /PAE we never see over 3.75GB of physical memory installed on this hardware set, so it helps out a lot."
So I think Brad is going back to a issue back from Windows 2000 Server. Their use to be a issue when you addeded /3GB & /PAE together on DataCenter edition with more then 16 GB's of ram, it would cap you down to 16GB, regardless if you had More then that for memory..
Another issue with having both enabled is that /PAE increases the size of your PTE Size from 4 bytes to 8 Bytes...So when you have /3GB & /PAE together, in certain situations you can hurt yourself...
For Example..../3GB cuts down your nonpaged and pool paged memory and PAE will double your PTE Size. So if you do a Large File copy using the "Copy" command...You can incur a performance problem because you will be consuming Pool Memory at a faster rate because your PTE size is increased while your pool memory has decreased in size which might result in failed copies and depelted pool memory.
But on another note, in Windows 2003 SP1 (and servers with Hot-Add Memory), Windows will load the ntrknlpa (thats PAE kernel) by default. So essentialy you have PAE running on any newly built 2K3 Server with SP1+
In a Windows 2003 server with sp1 and 16GB of RAM, I still need to use /PAE to allow the OS to address more than 3.75GB of RAM.
In this case is there any benefit to using /3GB? As there is more than enough RAM anyway, is there any point to limiting the Kernel to 1GB of RAM?
I would anticipate performance problems in this case.
What are everyone's thoughts?
"In a Windows 2003 server with sp1 and 16GB of RAM, I still need to use /PAE to allow the OS to address more than 3.75GB of RAM.
What are everyone's thoughts?"
I've seen that Issue on DEP enabled Proc's where you need to add the /PAE switch even if SP1 is installed (is this a HP Server by chance?).
As for adding the /3GB, what is the role of this server? You have to remember that /3GB is virtual Memory which is different the Physical memory. The Memory managment model works only in a 2GB/2GB (or 3GB/1GB) form. If their is nothing on your server that would take advantage of /3GB (any app that utilizes /largeaddressaware), then it would not benfit at all, it would likely degrade the performance of your box.
If you don't have a DEP enabled processor, you will still need to add /PAE even if your have upgraded to SP1+
As for adding the /3GB, what is the role of this server?
You have to remember that /3GB is virtual Memory which is different the Physical memory. The Memory managment model works only in a 2GB/2GB (or 3GB/1GB) form.
If their is nothing on your server that would take advantage of /3GB (any app that utilizes /largeaddressaware), then it would not benfit at all, it would likely degrade the performance of your box.
This is a follow-up to our Processes consuming high amounts of virtual memory post from January. Pagefile