We recently had a customer visit the Enterprise Engineering Center to gain a better understanding of the scalability and limitations of thier current Internet Security and Acceleration (ISA) Server / Exchange 2003 Server front end server design. They had just implemented a new centralized Exchange 2003 infrastructure and were in the process of migrating 70,000 mailboxes to the new environment. Due to security policies, all of thier outlook clients connect to exchange using RPC over HTTPS. Because each Outlook client requires on average about 8 connections (for directory service, mailbox, public folders, etc.), their environment needed to support over 560,000 connections. As they began to migrate mailboxes to the new environment they began to experience 32-bit architecture kernel non-paged pool memory limitations on the Exchange front end servers. After approximately, 20,000 connections, http.sys (a kernal mode driver than consumes npp memory for every RPC over HTTPS connection) had exhausted all available non-paged pool memory and the server became unresponsive. In order to support 560,000 connections, the customer would have to implement about 30 Exchange front end servers. Since thier original deployment only had 6 Exchange front end servers, so this would represent a significant and unacceptable increase in hardware and support costs.

The ultimate solution to this problem was to wait until Exchange 2007 is released and implement 64-bit Exchange Client Access Servers. The available non-paged pool memory goes from 250MB in 32-bit OS's to 128GB in 64-bit OS's thus eliminating the issue. In this case the customer couldn't delay thier migration until Exchange 2007 and wasn't comfortable running pre-release software in thier environment.

The Exchange team came up with an interim solution - replace the 32-bit Exchange 2003 front end servers with 64-bit Windows 2003 Servers running the RPC over HTTP Proxy service. Then when Exchange 2007 is released, add the CAS server role. This turned out to be a great solution for the customer so I thought it was worth blogging.

I'm in the process of writing 2 technet articles. One will describe the non-paged pool memory limitation and options available and one will describe some ways to tweak a 64-bit RPC Proxy server to support a large number of connections. I'll be posting draft versions here shortly.