There is a lot of confusion as to the purpose and impact of these BOOT.INI switches for Windows Server - this leads to a number of cases where "insufficient system resources", "Srv event 2019/2020" and even "hung server" are the symptoms presented.
The brief version is this:
PAE (Physical Address Extension) allows the OS to address more than 4GB of physical RAM and is required for hardware DEP (Data Execution Prevention) for CPUs that have NX/XD support
3GB is used to increase the size of the user-mode virtual address space from 2GB at the cost of the kernel virtual address space
The above switches are only relevant for x86 systems, as PAE is implicit on x64 and the virtual address space is 128GB for user and kernel modes.
The principal change that occurs by using PAE is the memory required to address each page of memory increases from 32 bits to 64 bits (not all the bits are used for the address translation, some are used for defining the state of the page).
This means that for the same amount of physical RAM in a system, you will have more kernel virtual address space used if PAE is used compared with when it is not.
The 4GB virtual address space is split by default into 2GB for user mode (0x00000000-0x7FFFFFFF) and 2GB for kernel mode (0x80000000-0xFFFFFFFF) - the kernel being shared by all processes, and each process having its own user mode space.
When using the 3GB switch this split is shifted to favour the user mode processes: 3GB for user mode (0x00000000-0xBFFFFFFF) 1GB for kernel mode (0xC0000000-0xFFFFFFFF)
In an attempt to illustrate the impact of either or both of these switches, consider the following diagram where user mode is the yellow section and kernel mode is the green section, with the memory used for addressing is represented by the white boxes in the kernel space:
It is very simplified, but hopefully helps demonstrate how kernel space can be exhausted by the use of both switches, and it does not help to have more physical RAM as this gives nothing to the limited virtual space available to the kernel (it can make it worse, in fact!).
PAE is required to make use of >4GB of physical RAM, or if DEP is required - so the choice of whether to use this switch is a no-brainer.
So when to use 3GB?
Very few applications can make use of having a larger user mode address space, and they have to be compiled with "/LARGEADDRESSAWARE" to take advantage of it.
Typically database and mail servers can make use of the extra space and it is often more of a performance gain than a "required to make it work".
If your application requires more than 2GB of user mode address space but using the 3GB switch exhausts kernel space, what can be done?
USERVA is a switch to use along with 3GB which can be used to give some of the stolen address space back to the kernel by setting it between 2048 and 3072 (MB) to define the total size of the user mode address space.
How to tell if you've run out of system resources, and which?
A local kernel debug (or manual crash dump) and the output from the command "!vm" should let you know if you exhausted one of the pools (paged and nonpaged) or ran out of PTEs (Page Table Entries, used to address memory).
(It is possible to have empty memory but have run out of PTEs to be able to address it - there is a finite amount of space allocated for PTEs.)
There may be clear indicators such as warnings about system PTEs or failed pool allocations, but if not then the values to check/compare are:
NonPagedPool Usage vs NonPagedPool Max PagedPool Usage vs PagedPool Maximum Free System PTEs (should be >10,000 under load)
Reference for further information on PAE: Operating Systems and PAE Support