I've been asked a lot of great questions lately and thought I'd post some of them.
As you might know, I am one of the instructors of the popular workshop "Vital Signs" which teaches students Windows architecture and how to identify performance bottlenecks. One of the instructors recently had a student who asked if a process (application) can be limited on the amount of physical memory (RAM) that it can use. The answer is it largely depends on the overall physical memory usage, but it can be limited.
The amount of physical memory that a process uses is called Working Set. The operating system's memory manager (referring to Microsoft Windows and Microsoft Windows Server) controls that amount of physical memory that a process uses by expanding and trimming the working set size of the process. The process itself has little control over this, but this model allows the operating system to manage physical resources more efficiently by only allowing memory that is most actively touched (read or written to) in physical memory.
If you use a tool like Process Explorer (http://live.sysinternals.com/procexp.exe), you can have it show the field Max Working Set. This is the amount of physical memory that the process believes it will need and it provides a suggestion to the operating system as to how large or small the working set of the process should be. For the most part though, maximum working set is largely ignored. The reason for this is that if the operating system has plenty of available physical memory, then it will allows frequently accessed pages of a working set to stay in physical memory simply because it is more efficient to keep it there.
According to the Sysinternals Administrators Reference book, here are the definitions of the working set fields:
Ref: Russinovich, Mark E.; Aaron Margosis (2011-06-29). Windows® Sysinternals Administrator’s Reference (p. 59).
Also, per David Solomon (one of the authors of the Windows Internals series of book), Process Explorer does not show if the process has a hard or soft max or min [working set] set. Also, he suggests using the Windows API, SetProcessWorkingSetSizeEx, to set a hard working set size.
With that said, there is a tool called the Windows System Resource Manager which can limit the amount of working set that a process uses. This tool is installable (not installed by default) through the Add Features console on Windows Server 2008 R2.
For more information on this subject, I recommend watching Mark Russinovich's, "Mysteries of Memory Management Revealed".