Blogs

Using the /3GB Switch with SBS 2003

  • Comments 6
  • Likes

I’d like to address a question that was asked during last week's MVP Summit, and periodically comes up on the various SBS mailing lists and newsgroups:  What is the definitive statement regarding the use of the /3GB switch on SBS servers?  I’m addressing SBS specifically, but these recommendations generalize to any DC running Exchange.   

Our official answer is contained in KB 815372 (emphasis mine):

For Exchange Server computers which are at the same time Active Directory Domain Controllers or Global Catalog servers we also do not recommend setting the /3GB switch in boot.ini. We recommend having dedicated Active Directory Domain Controllers or Global Catalog servers.

Because Microsoft Small Business Server installs as an integrated domain controller and Exchange server, we also do not recommend setting the /3GB switch when you run Exchange Server 2003 in a Small Business Server environment. However, the other concepts and settings that are described in this article apply equally to Exchange Server 2003 in a Microsoft Small Business Server environment

So, if you're not experiencing any problems with your SBS server, and performance is otherwise acceptable, most people can stop reading here.  The more intrepid, fearless admins and consultants will want to know why we don't recommend using the /3GB switch except for an extremely rare case detailed below.

More detailed explanations:

Mike Lee of the Exchange team has an excellent article describing the inner workings of Exchange memory management available at http://msexchangeteam.com/archive/2005/12/07/415733.aspx.

If you really want to get in to the nth level of detail on memory management, Mark Russinovich's Windows Internals book covers the topic in excrutiating detail in chapter 7.

Why this is important

The reasoning behind the KB article above is that you can actually starve your Active Directory for memory if Exchange (or potentially, other apps) experience heavy stress.  The underlying architecture of Exchange relies heavily on the GC for lookups, which can be quite intensive when, for instance, the server is processing a very large amount of mail (think mailing lists, mailmerges, mailbox-level backups).  In rare instances, you can get in to a scenario where AD is out of resources, even though Exchange still happily has enough user mode memory.  So, the long and the short of it is that, by using the /3GB switch on a DC, you are unlikely on a production SBS server to ever make full use of the additional gig of RAM.

The rare exception

So, let me address that extremely rare scenario where an SBS server may actually benefit from the /3GB switch.  I hesitate to do this because I know that human nature dictates that, even if I say that this will affect one in 10,000 SBS installs, there is an inclination to think "Well, maybe one of my SBS servers could be that one unlucky one".  Here's how to definetively determine that.

First, read http://msexchangeteam.com/archive/2005/12/28/416551.aspx.  The article is titled "Large Security Tokens and Kernel Memory Exhaustion", but it also applies more generally to PTE exhaustion in general.

It's then time to monitor memory and PTE usage on the server.  The reference for this is here: 

http://www.microsoft.com/technet/prodtechnol/exchange/guides/TrblshtE2k3Perf/7a44b064-8872-4edf-aac7-36b2a17f662a.mspx?mfr=true

Specifically, you'll want to monitor these performance counters:

Kernel memory space Performance Monitor counter Performance Monitor triggers with default boot.ini Performance Monitor triggers with boot.ini options /3GB and /USERVA = 3030

Paged Pool

Memory\Pool Paged Bytes

“Warning” when the Pool Paged Bytes counter exceeds 300 MB

“Critical” when the Pool Paged Bytes counter exceeds 320MB

“Warning” when the Pool Paged Bytes counter exceeds 200 MB

“Critical” when the Pool Paged Bytes counter exceeds 220 MB

Nonpaged Pool

Memory\Pool Nonpaged Bytes

“Warning” when the Pool Nonpaged Bytes counter exceeds 200 MB

“Critical” when the Pool Nonpaged Bytes counter exceeds 220 MB

“Warning” when the Pool Nonpaged Bytes counter exceeds 100 MB

“Critical” when the Pool Nonpaged Bytes counter exceeds 110 MB

System PTEs

Memory\Free System Page Table Entries *

“Warning” when Free System Page Table Entries is less than 8,000

“Critical” when Free System Page Table Entries is less than 5,000

“Warning” when the Free System Page Table Entries is less than 8000

“Critical” when the Free System Page Table Entries is less than 5,000

In most cases, the problem you are most likely to see first will be PTE exhaustion.  The error you will see logged in Event Viewer is:

Event Type: Error
Event Source: MSExchangeIS
Event Category: Content Engine
Event ID:
12800
Date: Date
Time: Time
User: N/A
Computer: Computer_Name
Description:
Message processing failed because there is not enough available memory (8007000E-82000387).

If you see this error, the next step is to verify that you are truly running out of memory and/or PTEs.  To do this, configure Perfmon to capture data on the counters above.  The link to the white paper above describes in great detail how to configure PerfMon and the various symptoms and resolutions to these problems.

In a nutshell

The vast majority of SBS servers will not benefit from use of the /3GB switch.  In some instances, performance may actually degrade.  In my straw poll of support escalation engineers on site here, we have never seen an SBS go over 2 GB of Exchange memory usage due to legitimate use (i.e. users actually sending large amounts of email to stress the store).  In virtually every instance we have seen here of the Store using massive amounts of memory, the root cause turned out to be:

  • by far the most common cause of memory problems is an improperly sized pagefile.  Administrators often think that adding an 8gb pagefile will help them in terms of available memory.  What actually happens is that this puts an increased strain on the server due to the number of page file handles we must track.  http://support.microsoft.com/default.aspx?scid=kb;EN-US;237740 covers how to correctly set pagefile size based on your RAM.
  • a corrupt database
  • Exchange add-ins
  • client misconfiguration (e.g. a client-side Outlook add-in that was auto-forwarding itself massive CAD files)
  • administrators who implement the various Exchange KBs without taking into account that Exchange is on a DC (for example, modifying heapdecommitfreeblockthreshold, systempages, etc.) 

In each of these cases, using the /3GB switch would actually make the problem worse, since Exchange would continue to exhaust system resources.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Comments
  • Nice stuff!
    Leave us not forget the ISA recommendations as well from the ISA performance best practices:
    http://www.microsoft.com/technet/security/prodtech/isa/isaprfbp.mspx

  • Hello Mark,

    Thank you for doing more to help clarify issues than anyone has before now to my knowledge.

    In particular, your description of how to definitively monitor memory and PTE exhaustion now provides Users a concrete way of establishing whether they might be that "one in 10,000" machines (although I suspect that your estimation may be skewed by the long history of SBS where until very recently most SBS had no more than 2GB physical RAM installed. Today is another era though, and I suspect your estimations might change as people might believe that with more physical RAM the /3GB switch might be implementable without causing excessive paging to the pagefile, although a very theoretical model I developed still estimates less than 10% would indisputably benefit <if> all SBS were evenly distributed from "minimal RAM" to 4GB max).

    But,
    Some questions...

    - You only mention Exchange, and although Exchange ordinarily would be a primary beneficiary of the /3GB switch, it's not the only major Server application on a typical SBS box. We also have SQL which is being used more and more all the time. At least a brief comment would be appreciated on whether SQL can benefit. Also, any other 3rd party applications that are installed on SBServer almost certainly will <not> be using substantial kernel mode addresses and will require almost exclusively User mode addresses.

    - When you mention Exchange, I assume it's because of its "special" way of grabbing "real" rather than virtual addresses, but maybe some statement would be useful in provisioning Exchange's actual memory requirements? From practice, I've found that in a <small business> (not large enterprise) network, 500mb of real RAM can be adequate despite Exchange's behavior to grab everything it can.

    - You mention the /3GB switch, but you don't mention that there are sub-switches that can restrict memory address acquisition, maybe they are important or maybe not because I wonder if this focus on Exchange is all that relevant and we should instead be focusing on the kernel memory space

    - Although running the monitoring you describe might turn up interesting information, can Microsoft (yourself) suggest what types and patterns of activities would impact the use of kernel mode memory? In general, we all know that DC functionality falls into this category. I assume that such things as "kernel mode pumping" in VPNs are included but haven't a clue exactly how much and in what pattern. Something like "Such and such activity (like file access, IIS access, VPN support) requires on average [something] and will be persisted [time] before those resources will be available again.

    - Related to the previous point, I suspect that properly provisioning for sufficient kernel memory addresses depends largely on numbers of Users and the types of activities this network mainly supports. Are there some studies that suggest some kind of profile(s) of networks that would need more than 1GB of RAM? When you advocate 2GB of kernel memory space, that is the same as any other DC in a large enterprise which ordinarily supports thousands if not tens of thousands of Users. I then wonder why the big difference of halving kernel mode memory addresses to 1GB that all of a sudden can only support less than 1%, maybe even less than 1% of 1% of a DC in an Enterprise.

    - I strongly suspect if 3rd party applications aren't using Windows Security, then again there would be less pressure on kernel mode memory and greater use of User Mode memory.

    Thx,
    Great Stuff!



  • The Official SBS Blog : Using the /3GB Switch with SBS 2003: http://blogs.technet.com/sbs/archive/2006/06/30/439628.aspxMark...

  • Thanks, Mark. As you noted, this question has been asked for many years. Glad it only took 3 years to get the "definitive" answer!

  • Hi Mark,

    i´m very happy that in the 64bit world this memory mangement problems are totally solved.

    BTW: Is there a preview whitepaper where the answer is given how much RAM "SBS Server Longhorn" can handle?

    Regards P. Kohn
    www.governmentforum.de
    Mod @ www.mcseboard.de

  • PingBack from http://www.hilpers.com/1045343-warnung-zugewiesener-speicher/2