I've been asked a lot of great questions lately and thought I'd post some of them.

As you might know, I am one of the instructors of the popular workshop "Vital Signs" which teaches students Windows architecture and how to identify performance bottlenecks. One of the instructors recently had a student who asked if a process (application) can be limited on the amount of physical memory (RAM) that it can use. The answer is it largely depends on the overall physical memory usage, but it can be limited.

The amount of physical memory that a process uses is called Working Set. The operating system's memory manager (referring to Microsoft Windows and Microsoft Windows Server) controls that amount of physical memory that a process uses by expanding and trimming the working set size of the process. The process itself has little control over this, but this model allows the operating system to manage physical resources more efficiently by only allowing memory that is most actively touched (read or written to) in physical memory.

If you use a tool like Process Explorer (http://live.sysinternals.com/procexp.exe), you can have it show the field Max Working Set. This is the amount of physical memory that the process believes it will need and it provides a suggestion to the operating system as to how large or small the working set of the process should be. For the most part though, maximum working set is largely ignored. The reason for this is that if the operating system has plenty of available physical memory, then it will allows frequently accessed pages of a working set to stay in physical memory simply because it is more efficient to keep it there.

According to the Sysinternals Administrators Reference book, here are the definitions of the working set fields:

• Minimum Working Set. The amount of physical memory reserved for the process; the operating system guarantees that the process’ working set can always be assigned at least this amount. The process can also lock pages in the working set up to that amount minus eight pages. This minimum does not guarantee that the process’ working set will always be at least that large, unless a hard limit has been set by a resource management application.
• Maximum Working Set. Indicates the maximum amount of working set assigned to the process. However, this number is ignored by Windows unless a hard limit has been configured for the process by a resource management application.
• Working Set Size. The amount of physical memory assigned to the process by the memory manager.

Ref: Russinovich, Mark E.; Aaron Margosis (2011-06-29). Windows® Sysinternals Administrator’s Reference (p. 59).

Also, per David Solomon (one of the authors of the Windows Internals series of book), Process Explorer does not show if the process has a hard or soft max or min [working set] set. Also, he suggests using the Windows API, SetProcessWorkingSetSizeEx, to set a hard working set size.

With that said, there is a tool called the Windows System Resource Manager which can limit the amount of working set that a process uses. This tool is installable (not installed by default) through the Add Features console on Windows Server 2008 R2.

For more information on this subject, I recommend watching Mark Russinovich's, "Mysteries of Memory Management Revealed".