Memory Management - Demystifying /3GB

Published 23 March 07 06:10 AM

As promised - here's the long awaited post on the infamous /3GB switch.  At least once a week we have this discussion with a Systems Administrator somewhere who has this set in the boot.ini file on all of the servers but doesn't know why.  Maybe someone added it to the server build process at some point or perhaps someone read about the /3GB switch somewhere and thought that it would improve performance or enable them to see the full 4GB or 8GB of physical memory installed on the system.  So let's start by dispelling a few /3GB myths:

  • /3GB won't enable you to see the additional 4GB or 8GB of RAM you added to your new server
  • /3GB doesn't necessarily make your application 50% more efficient
  • /3GB should not be a standard for your environment (there are exceptions, and we'll get to those)

 OK - so what does the /3GB switch really do?  If you recall, from the Memory Management 101 post, Windows 32-bit Operating Systems implement a virtual memory system based on a flat 32-bit address space.  This address space translates into 4GB of virtual memory - no more, and no less.  By default, the address space is divided into two equal chunks.  2GB for the Kernel and 2GB for User-mode processes.  The Kernel space is common for all applications and the User-mode processes each get their own 2GB address space to work with.

So where does the /3GB switch come in?  Windows 2000 Advanced Server, Windows 2000 Datacenter Server, Windows XP SP2 and later and all versions of Windows Server 2003 support the /3GB boot-time option to allow the user mode address space to grow to 3GB.  The /3GB option was intended as a short term solution to allow applications such as database servers to maintain more data in memory than a 2GB address space allowed.  However, using the /3GB method to increase the user-mode memory space comes at a cost. 

Remember that we only have a 4GB total address space to work with.  If we have to allocate an additional 1GB of this address space to the user-mode space, then the System space is cut in half.  Drivers, Heap, Paged & NonPaged Memory all have only half the resources to work with now.  However, because of the way memory mapping works, cutting the kernel space in half does a lot more than just reducing the address space.  Many of the structures within the kernel virtual memory space are cut back by far more than 50%.  For example, we took a Windows Server 2003 Enterprise R2 machine with 1GB of RAM installed and compared some values with and without the /3GB switch enabled.

Default OS Build:
Free System PTE's 251,980 (1,007,920 kb)
NonPaged Pool Max 206,848 kb

With /3GB enabled:
Free System PTE's 34,884 (139,536 kb)
NonPaged Pool Max 129,312 kb

As you can see, the Free System PTE's drops by over 200,000.  Keep in mind that this is only a test server that isn't under any sort of load.  A machine under medium to heavy load could quite easily run out of free PTE's - meaning that the system can no longer map system pages such as I/O space, kernel stacks and memory descriptor lists.  In addition, look at NonPaged Pool after the /3GB parameter is enabled.  The NonPaged Pool maximum is only 130MB.  Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any IRQL.  Once we run into NonPaged pool depletion, we're looking at our old friend, the Event ID 2019.

OK - so let's quickly recap what we've discussed so far.  The /3GB switch is not related to the amount of physical memory installed in a system.  It is useful if you have an application that can take advantage of a larger address space.  For a process to access the full 3GB address space, the image file must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header.

If the flag is not set in the image header, then the OS reserves the third gigabyte so that the application won't see virtual addresses greater than 0x7FFFFFFF. You set this flag by specifying the linker flag /LARGEADDRESSAWARE when building the executable.  This flag has no effect when running the application on a system with a 2-GB user address space. Therefore if you enable the /3GB switch, then applications that do not have this flag set can only use the standard 2GB of User mode memory, and the Kernel is still limited to the 1GB space - which means that 1GB of virtual memory is basically wasted!

Earlier, we mentioned that there were some applications that benefit from the use of the /3GB switch.  The predominant scenario where the /3GB switch is not only recommended, but actually required is with Microsoft Exchange servers that house public folders and / or mailboxes.  Due to the way that Exchange handles memory management, the additional 1GB of user mode memory is required to ensure that the Store.exe process does not run out of virtual address space.  However, in order to guard against System PTE depletion, the system can be tuned using the /USERVA switch in conjunction with the /3GB switch.  This tunes the actual amount of memory for the Address space.  For example, setting USERVA=3030 means that the process space is actually only 3,030MB and not 3,072MB (which would be the process space with only the /3GB switch present).  The additional 42MB is used for System PTE usage.  The USERVA value can safely be tweaked as low as 2800 - however, if it is necessary to set USERVA this low, then you probably want to start thinking about scaling your Exchange environment to spread the load! 

Ideally, there should always be at least 24,000 Free System PTE's at boot time.  Depending on server workload there may be wide variances in the amount of Free System PTE's during the course of a normal duty cycle, so it may be necessary to implement some long-term monitoring to ensure that the server does not fall below 10,000 Free PTE's. 

So there you have it - the /3GB switch demystified.  Hopefully this post, as well as the others in our Memory Management series will help you understand a bit more about how and why the Operating System behaves the way it does.  Remember that the /3GB switch is intended to be used in very specific instances - and now you know why!

Additional Resources:

 - Tim Newton

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

# bitzie said on March 26, 2007 3:52 PM:

...but not on a SBS box.

Please do not use the /3 switch there.

# Clive Watson's Weblog said on March 27, 2007 7:53 AM:

...I Hope! Well done to the performance team for their recent Windows Server articles, in particular

# Active Directory, Cluster and other fun stuff... said on March 28, 2007 8:13 AM:

I see a lot of customers unnecessarily using the boot.ini /3GB switch. Explaining when and when not to

# Jeff Louque said on March 30, 2007 11:19 AM:

Keep in mind that the /Userva switch was introduced in Windows 2003 and does not work with Windows 2000.

# Brad Rutkowski said on March 30, 2007 5:05 PM:

You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together.

# Jason Boche said on April 3, 2007 2:38 PM:

"You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together."

That has not been my experience, but we only used the /PAE and /3GB tuning switch on one particular set of SQL cluster servers.

# bday said on April 3, 2007 3:32 PM:

"Brad Rutkowski said:  

You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together."

This is incorrect. Our Exchange 2003 servers use the following and are quite happy.

/PAE /3GB /USERVA=3030* /BASEVIDEO

*2970 in some cases after some PTE tuning

Without /PAE we never see over 3.75GB of physical memory installed on this hardware set, so it helps out a lot.

# Confused said on April 3, 2007 4:17 PM:

>/3GB won't enable you to see the additional 4GB or 8GB of RAM you added to your new server

>/3GB doesn't necessarily make your application 50% more efficient

>/3GB should not be a standard for your environment (there are exceptions, and we'll get to those)

From the writing, I can't immediately tell: are these the myths? Or are these the "dispellations" ?

# CC Hameed said on April 3, 2007 5:06 PM:

The two sentences preceding the ones you cited should provide the context:

"At least once a week we have this discussion with a Systems Administrator somewhere who has this set in the boot.ini file on all of the servers but doesn't know why.  Maybe someone added it to the server build process at some point or perhaps someone read about the /3GB switch somewhere and thought that it would improve performance or enable them to see the full 4GB or 8GB of physical memory installed on the system"

So, if you think of the bullet points as factual statements:

- /3GB should not be a standard for your environment (except in certain circumstances)

- /3GB won't enable you to see the additional 4GB or 8GB of RAM

- /3GB does not necessarily make your application 50% more efficient

Hope this clears up the misunderstanding!

Thanks for the feedback

# Carpe Diem: Flaphead.com @ Home said on April 9, 2007 1:59 PM:

No sure how I found this blog but its damm good. Check out these that I have been reading today! IE7

# Billy Dhillon said on April 11, 2007 10:37 AM:

""Brad Rutkowski said:  

You should aslo mention that 3GB and PAE dont like to play nice and should not be enabled together."

This is incorrect. Our Exchange 2003 servers use the following and are quite happy.

/PAE /3GB /USERVA=3030* /BASEVIDEO

*2970 in some cases after some PTE tuning

Without /PAE we never see over 3.75GB of physical memory installed on this hardware set, so it helps out a lot."

So I think Brad is going back to a issue back from Windows 2000 Server. Their use to be a issue when you addeded /3GB & /PAE together on DataCenter edition with more then 16 GB's of ram, it would cap you down to 16GB, regardless if you had More then that for memory..

Another issue with having both enabled is that /PAE increases the size of your PTE Size from 4 bytes to 8 Bytes...So when you have /3GB & /PAE together, in certain situations you can hurt yourself...

For Example..../3GB cuts down your nonpaged and pool paged memory and PAE will double your PTE Size. So if you do a Large File copy using the "Copy" command...You can incur a performance problem because you will be consuming Pool Memory at a faster rate because your PTE size is increased while your pool memory has decreased in size which might result in failed copies and depelted pool memory.

But on another note, in Windows 2003 SP1 (and servers with Hot-Add Memory), Windows will load the ntrknlpa (thats PAE kernel) by default. So essentialy you have PAE running on any newly built 2K3 Server with SP1+

# Nudge said on April 11, 2007 6:07 PM:

In a Windows 2003 server with sp1 and 16GB of RAM, I still need to use /PAE to allow the OS to address more than 3.75GB of RAM.

In this case is there any benefit to using /3GB?  As there is more than enough RAM anyway, is there any point to limiting the Kernel to 1GB of RAM?

I would anticipate performance problems in this case.

What are everyone's thoughts?

# Billy Dhillon said on April 12, 2007 10:49 AM:

"In a Windows 2003 server with sp1 and 16GB of RAM, I still need to use /PAE to allow the OS to address more than 3.75GB of RAM.

In this case is there any benefit to using /3GB?  As there is more than enough RAM anyway, is there any point to limiting the Kernel to 1GB of RAM?

I would anticipate performance problems in this case.

What are everyone's thoughts?"

I've seen that Issue on DEP enabled Proc's where you need to add the /PAE switch even if SP1 is installed (is this a HP Server by chance?).

As for adding the /3GB, what is the role of this server? You have to remember that /3GB is virtual Memory which is different the Physical memory. The Memory managment model works only in a 2GB/2GB (or 3GB/1GB) form. If their is nothing on your server that would take advantage of /3GB (any app that utilizes /largeaddressaware), then it would not benfit at all, it would likely degrade the performance of your box.

# xentelworker said on April 12, 2007 11:43 AM:

"In a Windows 2003 server with sp1 and 16GB of RAM, I still need to use /PAE to allow the OS to address more than 3.75GB of RAM.

In this case is there any benefit to using /3GB?  As there is more than enough RAM anyway, is there any point to limiting the Kernel to 1GB of RAM?

I would anticipate performance problems in this case.

What are everyone's thoughts?"

If you don't have a DEP enabled processor, you will still need to add /PAE even if your have upgraded to SP1+

(Ref: http://support.microsoft.com/default.aspx?scid=kb;en-us;875352)

As for adding the /3GB, what is the role of this server?

You have to remember that /3GB is virtual Memory which is different the Physical memory. The Memory managment model works only in a 2GB/2GB (or 3GB/1GB) form.

If their is nothing on your server that would take advantage of /3GB (any app that utilizes /largeaddressaware), then it would not benfit at all, it would likely degrade the performance of your box.

# Ask the Performance Team said on May 15, 2007 6:02 AM:

This is a follow-up to our Processes consuming high amounts of virtual memory post from January. Pagefile

# Sashank said on May 21, 2007 4:59 PM:

Is there some way to (programatically) find out when the OS is booted in the /3GB mode? The 32-bit version of my application sets a hard limit of 1700 MB after which it does not try to allocate any more memory. I'd like to switch this to 2700 MB when the /3GB switch is available. The best I can come up with till now is to read the boot.ini file, but I am wondering if there is anything clever out there, that I can do.

Thank you for the elaborate discussions on Windows Memory.

Sashank.

# Digvijay said on June 21, 2007 11:04 AM:

Nice Post..

helped me to clear a lot of doubt about the /3GB switch

Keep up the good work, Tim

# xelox said on August 29, 2007 8:21 PM:

Sashank: You can use powershell/wmi to get the value from boot.ini.

# Gonzalo said on September 22, 2007 9:31 PM:

Hi.

I Have a Windows 2003 SP2 Terminal Server with 2 GB in RAM and i haven't found a good advice in order to use (or not) the /3GB /Userva:3030 in a Terminal Server.

# Ask the Performance Team said on September 25, 2007 10:04 AM:

Following on from our post on troubleshooting a basic application crash , it's time to start troubleshooting

# Ask the Performance Team said on September 28, 2007 10:35 AM:

In previous posts, we've discussed the Basics of Memory Management , Pool Resources and of course the

# Tim Newton said on October 5, 2007 12:37 PM:

In response to Gonzalo's question, the answer is NO. You do not want to enable /3GB or USERVA unless you have a specific application need to do so, such as in an Exchange environment. On a terminal server, the last thing you want to do is cripple the kernel by enabling /3GB.

Gonzalo posted:

"I Have a Windows 2003 SP2 Terminal Server with 2 GB in RAM and i haven't found a good advice in order to use (or not) the /3GB /Userva:3030 in a Terminal Server."

# Richard said on October 9, 2007 10:10 AM:

Hi,

My servers have 17 GB of RAM and the /3GB /PAE switches enabled (vendors recommendation).

The BIOS shows 17 GB, but the OS only displays 15 GB.  I thought the /3GB would limit to 16, but why 15 GB? Any ideas?

# John said on October 23, 2007 5:43 PM:

You wrote about Microsoft Exchange servers. Using Catia, a high end auto / aero CAD package, the recomendation has been to run 3 GB by default. Is this a problem? We commonly work with data files in the hundreds of MBs

# Ask the Performance Team said on October 30, 2007 9:14 AM:

In previous posts we've discussed the basics of memory management including an overview of kernel and

# Louis said on January 15, 2008 6:09 PM:

Is the /3GB Switch required in the 32-Bit Version of Windows 2003 R2????

# wizualizacje architektoniczne said on February 4, 2008 9:27 AM:

Thanks for that info.

Regards

# feechka-ca said on February 8, 2008 7:39 PM:

<a href= http://1.mastr6.com >portland zip codes</a>

# lol said on February 8, 2008 9:02 PM:

<a href= http://index1.bestmaiden.com >postmasters assigned in fl</a>

# kybernetegisms said on February 20, 2008 8:07 PM:

Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either

# Noticias externas said on February 20, 2008 8:51 PM:

Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either

# MarkR said on February 22, 2008 2:15 PM:

After running the EXBPA, it recomends adding the /3GB to the Win2003 domain contollers in addition to the Exchange 2003 mailbox servers.  What are the benifits to adding the /3GB to the DCs?

# Cmonurz said on May 10, 2008 6:37 AM:

Hi guy's, we get asked this a lot in the Terminal Services/Citrix world, where misguides admins add /3GB switch and cripple their servers.

Can you guy's confirm that in a 32bit Windows 2003 terminal services environment that this is a totally inappropriate modification?

Thanks!

# Krishna said on May 14, 2008 11:57 PM:

If my server has 32GB of RAM, is it possible to make IIS use all 32GB of RAM ?

# Werner said on May 30, 2008 3:08 PM:

So what is the maxinum amount of RAM that will work on a Intel Core2Duo with Win XP SP2 without any going to waste?

# Manvendra Kumar said on June 30, 2008 4:19 AM:

I have 4GB of ram installed and 3gb switch enabled. I am running SQL 2000. Few days back my server crashed. Is there any possibilities that it could have been caused due to 3gb switch limit settings. I couldn't find any errors through logs, event and dump analysis. If yes, what kind of error will indicate that it was caused due 3gb switch settings.

# Anil said on July 13, 2008 7:07 AM:

When configuring a new Exchange 2003 SP2 server on a Windows 2003 server with 4GB of RAM, what are the three settings that need to be changed in Windows to optimize memory?

# Paulo Oliveira said on August 28, 2008 10:05 AM:

So, this means that I don´t need the /3GB switch to make windows reconize up to 4GB? The OS will reconize automaticly?

Regards,

Paulo Oliveira.

# timothyn said on September 2, 2008 5:34 PM:

Paulo,

 Yes, that is correct. The /3GB switch has absolutely nothing to do with how much physical RAM you have installed in the machine, simply how much VIRTUAL memory is allocated to user mode. The confusion over this is why I wrote this post in the first place.

Leave a Comment

(required) 
(optional)
(required) 

About CC Hameed

I joined Microsoft as a Support Engineer on the Performance team in September 2005. Prior to that I spent a couple of years working the late night shift on our Platforms 24x7 team. Working for Microsoft was always a dream job - so I am living the dream! I was on the Windows Vista Beta team in 2006, which was one of the coolest projects I have ever worked on, until I took on the task of driving the AskPerf Blog. As you can tell by my logo, I am a huge Manchester United fan and I have successfully managed to brainwash my two daughters into sharing my passion for the Red Devils much to the dismay of their mother! I also coach both my daughters' soccer teams. In addition I am an avid MMO gamer, and have an extensive DVD movie collection.

This Blog

Syndication

Page view tracker