The purpose of this document is to provide guidance into Forefront Protection 2010 for Exchange Server (FPE) capacity planning regarding hardware, such as the number of processing cores, memory requirements, and when to scale up based on CPU utilization and throughput. Performance guidelines using the capacity planning tool are provided for Exchange Server 2010 deployments. A new interactive capacity planning tool will be released in the future for FPE 2010.
Performance guidelines are also provided here on how to leverage the existing Forefront Security for Exchange Server (FSE) Version 10 with Service Pack 1 capacity planning tool for Exchange Server 2007.
Note: The performance measurement baseline information provided is based on the existing product documentation for Exchange Server.
All organizations are unique and driven by requirement specifications, policies, behaviors, and cultures that guide and inform hardware purchase decisions. This document can help clarify the additional load created by FPE for Exchange Server 2010 on sample server environments and provides guidance to help with capacity planning decisions. This information should be combined with your experience, the Exchange server capacity planning guidelines, and your general knowledge of your own organization and IT landscape.
There are two main reference architectures used for FPE deployments.
The first is the Enterprise Reference Architecture shown in Figure 1. This is a scalable unit intended to be deployed within a larger organization and is composed of an Edge server, Hub server, multiple Mailbox servers, an Active Directory server, and a Client Access server (CAS).
Figure 1. Enterprise Reference Architecture
Figure 2 shows the Standard Reference Architecture, which is intended to be a scalable unit, targeted at small to medium sized organizations. This architecture is composed of a dedicated Edge server, one or more multi-role servers that encompass the Hub server, Mailbox server, and CAS roles, and then a dedicated Active Directory server.
Figure 2. Standard Reference Architecture
Given the two reference architectures, the unique server roles are identified as follows:
In this server role, the SMTP data stream flows into an organization and message hygiene is performed. Message hygiene includes antispam, antivirus, antispyware, and custom filtering. The FPE protection technology software resides on this server along with the Microsoft Exchange Transport service. The messages that pass an enterprise’s message hygiene are routed to the appropriate Hub server for additional routing. Messages that have undergone scanning as part of the message hygiene at the Edge are not scanned again at the Hub.
The Exchange Hub server accepts routing SMTP information from the Edge server, can perform additional message hygiene based on configuration, and then routes the messages to the appropriate Mailbox server or additional Hub servers. The FPE protection technology software resides on this server, along with the Microsoft Exchange Transport services.
The Exchange Mailbox server accepts the incoming messages from the Hub server and can perform additional message hygiene based on configuration. In addition, the Mailbox server can perform scheduled and on-demand scanning requests. The FPE protection services reside on this server, along with the Microsoft Exchange Information Store service.
In an Exchange multi-role server role, this server contains the capabilities of the FPE Hub server, FPE Mailbox server, and Exchange CAS all in one server. So, it provides aggregate functionality.
The following sections provide descriptions of each of the Exchange Server roles running FPE along with applicable guidance around performance.
On the Edge server it is critical to understand the load requirements for performing message hygiene. This can be broken down into two main components: SMTP traffic that results in messages delivered to a user in your organization, and SMTP traffic detected as spam or malware. The Edge server running FPE can also be configured to send outgoing e-mail from your internal users, but the load from this activity is fairly minimal. You can assess your current message load by monitoring the following performance counters using Exchange Listener:
Over a 24 hour period, you can obtain the average and peak message load.
Another critical component for evaluating proper sizing on the Edge server is the configuration. There are several ways the Edge server can be configured, but only a few select ones are addressed for sizing.
Note: Additional configuration information will be provided in the future in an updated capacity planning tool for FPE.
The three main areas of configuration, in order of importance, are:
Engine management is comprised of two different components.
The first is the Intelligent Engine Management (IEM) configuration, which enables the user to have FPE automatically update and select the most appropriate engines to use based on the default Engines and Performance setting of scanning with all available engines. This configuration results in all engines scanning incoming mail when available. (Availability is based on non side-by-side engine updating.) This setting can be modified to manually select a certain number of engines.
The second component is the Engines and Performance setting. Again, by default, the subset of available engines are selected to perform message hygiene. The user can select an Engines and Performance setting that ranges from always scanning with the enabled engines to only scanning with one engine.
By default, the number of transport scanning processes is set to 4. That means there are 4 processes used to perform malware message hygiene.
By default, the premium antispam functionality is enabled on all FPE Edge servers. This can be disabled on the FPE Edge server.
Figure 3 shows the CPU performance of a recommended Edge server configuration. This graph shows the relative performance of the previous version of the product, FSE, compared against the current version, FPE. Samples were taken, and then a linear interpolation was created to see the projected CPU utilization compared to the incoming message rate.
Processors: 2x2 => 4 cores; Intel Core 2 Duo; 2.66 MHz
Memory: 4G RAM
Message Mix: Sample of Microsoft IT
Engines and Performance: Scans with all engines (5 engines) and no antispam.
Scanning Processes: 4
Figure 3. FPE CPU Utilization
The next graph, Figure 4, shows the CPU performance relative to the number of scanning processes on a higher performance machine – 8 cores. This graph represents the scale up capacity using the default configuration verses a configuration that is tuned for a scanning process per core. The measurements represent the areas for normal throughput without the submission queue backing up.
Figure 4. Throughput Scalability on 8 Core Server
Note: The submission queue starts to back up even though the CPU utilization never reaches more than an aggregate 23% of the server.
Processors: 2x4 => 8 cores; Dell R610; 2.93 GHz Xeon
Memory: 16G RAM
Scanning Processes: 4 and 8
During these various tests, the malware scanning detection was disabled and only the premium antispam capabilities were enabled. The CPU utilization increased 3.3% on an incoming message rate of 100 messages per second. This was relative to the baseline numbers with FPE enabled on the Edge server without any message hygiene.
Memory measurements were also conducted on a variety of configurations and the recommendations are provided at http://go.microsoft.com/fwlink/?LinkID=163921. The overall memory footprint of the product is larger than that of its predecessor due to a new service. This increases the memory footprint approximately 650 MB.
The recommendation is to use a 4 core server (not hyper-threaded) for the Edge. Although the product supports 1 and 2 core server configurations, this impacts your overall throughput. You can also use more than 4 cores; however, in doing so, you should also change the configuration of the number of scanning processes to align with the number of cores on your server. The default configuration is 4 scanning processes; this should be increased to 8 for an 8 core server.
The memory recommendation is provided at http://go.microsoft.com/fwlink/?LinkID=163921. Overall, you should size your server configuration for at least 2 GB over the recommended Exchange Server 2010 Edge server guidance.
Scaling Up / Out
The default configuration, at about 75% utilization, allows for approximately 45 incoming SMTP messages per second. This is on top of whatever incoming rate you experience for spam. The 45 incoming SMTP messages per second are those messages that are eventually routed to the Hub servers. This is with 5 scanning engines enabled.
If you want to support a larger incoming rate, you can:
· Increase the number of cores on your server.
· Decrease the number of engines used by each of the scanning processes.
· Add another Edge server.
The data on how this impacts scale will be available in a future Microsoft TechNet article. However, there can be significant scale gain by changing from 5 scanning engines to 4, or 3.
The Hub server uses a very similar configuration as the Edge server, however, there are two significant differences.
First, the premium antispam functionality is not enabled by default on this role. Secondly, if an Edge server is forwarding mail to a Hub server, then message hygiene has already been accomplished except in cases where the configuration differs from the default. So, the message hygiene provided by the Hub server is for internal originating mail.
The basic paradigm that aligns with Exchange is a user that sends/receives a total of 50 messages per day that average 75 KB per message. Out of the 50 messages per day, an estimated 20% are sent, an estimated 25% of the remaining 80% are received from the Internet, and the remaining 75% of the 80% are received from internal recipients.
Using these calculations and the numbers from the Edge server, at peak times a message rate of 8 messages per second are scanned.
The guidance from Exchange Server 2010 for a user with 50 messages per day is 1000 mailboxes per core. Using the default configuration, this is 4000 mailboxes that send/receive 50 messages per day on the recommended processor configuration. This results in a peak realization of approximately 8 messages per second that require message hygiene from the Hub as it routes the messages.
The recommendation is to use a 4 core server (not hyper-threaded) for the Hub server. Although the product supports 1 and 2 core server configurations, this impacts your overall throughput and Hub to mailbox fan-out. You can also use more than 4 cores; however, in doing so, you should also change the configuration of the number of scanning processes to align with the number of cores on your server. The default configuration is 4 scanning processes and this should be increased to 8 for an 8 core server.
The memory recommendation is provided at http://go.microsoft.com/fwlink/?LinkID=163921. Overall, you should size your server configuration for at least 2 GB over the recommended Exchange 2010 Edge server guidance.
Hub to Mailbox Fan-Out
Based on the recommended hardware, with 5 scanning engines and fully loaded Mailbox servers being serviced by the Hub, there is a 1:5 Hub to mailbox relationship. In terms of cores, 1 core can support a downstream of 5000 users that send/receive 50 messages per day. You can scale this linearly, so if the representative users send/receive 100 messages per day, the Hub can support 2500 per core; 200 sent/received per day yields 1250 downstream users per core.
If you want to support a larger Hub to mailbox ratio, you can either:
The data on how this can impact scale will be available in a future Microsoft TechNet article. However, there can be significant scale gain by changing from 5 scanning engines to 4 or 3.
There are three factors that impact performance on the Mailbox server: scanning of mailboxes using scheduled and on-demand scans, the configuration of the server, and the load of the users that generate traffic that require message scanning.
The basic paradigm that aligns with Exchange represents a user that sends/receives a total of 50 messages per day averaging 75 KBs/message. Out of the 50 messages per day, an estimated 20% are sent, an estimated 25% of the remaining 80% are received from the Internet, and the remaining 75% of the 80% are received from internal recipients. Most of the activities that generate realtime scanning will be on-access scanning of the messages. This traffic can be modeled conservatively by assuming 20% of all access mails are scanned out of two times the number of mails received on a daily basis. This is constant given the correlation between the Exchange server recommendations and the mail being received.
Two areas in configuration need special attention. First, the number of engines used in scanning on the Mailbox server role. As with all other roles, the number of engines used impacts performance. Secondly, the user can enable the rescanning of the messages if they enable the Scan after engine update option.
Scheduled scanning is intended to run on a schedule basis and scans all the mailboxes serviced by the Exchange Mailbox server role. This should not be done during the day as it significantly impacts the overall performance of the Mailbox server. On-demand scanning is used to scan a small set of mailboxes at any given time. This introduces a temporary performance impact on the mailbox server but is recoverable and dependent upon the number of mailboxes scanned and their size. Given the variety of the inputs for on-demand scanning it cannot be adequately modeled to determine the overall impact on the Mailbox server.
The guidance from Exchange Server 2010 for a user with 50 messages per second is 1000 mailboxes per core. Using the default configuration, this is 4000 mailboxes that send/receive 50 messages per second on the recommended processor configuration. Using the conservative approach of 20% of messages rescanned per user on twice the set of incoming messages, the impact of the scanning and overhead of the FPE protection software should not exceed 20%.
The recommendation is to use at least a 4 core server (not hyper-threaded) for the Mailbox server. Consult the Exchange 2010 processor guidelines and make sure that the minimum requirements are met. If using more than 4 cores, you should change the configuration of the number of scanning processes to align with the number of cores on your server. The default configuration is 4 scanning processes and this should be increased to 8 for an 8 core server.
The memory recommendation is provided at http://go.microsoft.com/fwlink/?LinkID=163921. Overall, you should size your server configuration for at least 2 GB over the recommended Exchange 2010 Edge Server guidance.
In terms of cores, 1 core can support 20% less than the Exchange Server 2010 recommended users per core. For example, for a profile of users receiving and sending 50 messages per day, 1000 users can be supported per core for an Exchange 2010 Mailbox server. With FPE installed on the Mailbox server, the value is 20% lower, meaning 800 users per core. So, you can scale up by sizing your Mailbox server appropriately with more cores, or scale out by adding additional mailbox servers to support your profiled number of users. Refer to Microsoft Exchange Server Profile Analyzer for help in identifying your user requirements on the Mailbox server.
The 20% factor is based on the default IEM of 5 engines. You can reduce this overall impact by changing the number of engines used to scan on the Mailbox server.
The Multi-Role server supports Hub server functionality as well as Mailbox server functionality. In addition, the CAS may also be hosted on this computer. Understanding the performance implications is a bit more complicated, especially when the Hub server is really used as a message hygiene server without an associated Edge role in the organization.
The main factor impacting performance is the overall responsibility of the server that directly relates to the expected load it must handle. In general, the Hub server role is not meant to be the main message hygiene role in an organization and it is assumed that this is not in this case.
In addition, the server has 8 total scanning processes associated with FPE protection technologies – 4 dedicated to the Hub server role and 4 dedicated to the Mailbox server role. This impacts the overall scanning throughput capabilities.
In lab measurements, the overall impact of the message scanning throughput is roughly a 25% impact. Exchange recommends no more than 500 users per core on a multi-server role configuration based on a user sending/receiving 50 messages per day. This leads to an aggregate impact of 6.2 scanning messages per second on the server, which correlates to 10% CPU impact. When factoring in the 25% scanning impact, this comes out to a 12.5% reduction in the number of users that can be supported on a multi-server role with FPE protection technologies installed.
· Messages per second per 1000 users (Mailbox): ((1000*40*2)/5) / (80*60*60/2) = 1.11
· Messages per second per 1000 users (Hub): (1000*(.75*.8*50)/(8*60*60/2)) = 2.083
· Total impact => 3.19 messages per second per 1000 users
The recommendation is to use at least a 4 core server (not hyper-threaded) for the Multi-Role server. Consult the Exchange 2010 processor guidelines and make sure that the minimum requirements are met. If you are using more than 4 cores, you should change the configuration of the number of scanning processes to align with the number of cores on your server. The default configuration is 4 scanning processes for Hub and 4 for Mailbox. So you should increase the numbers proportionately if using a server with more than 8 cores.
The memory recommendation is provided at http://go.microsoft.com/fwlink/?LinkID=163921. Overall, you should size your server configuration for at least 3.5 GB over the recommended Exchange 2010 Edge server guidance.
The Multi-Server role can support 437 users per core that send/receive 50 messages per day. You can scale up by:
· Increasing the number of cores on your multi-server role.
· Decreasing the numbers of engines used on the Hub server role and Mailbox server roles.
· Migrating users to additional hardware.
In addition to capacity planning guidelines discussed previously, you should consider the following best practice attributes for a healthy deployment that affects overall performance:
Maintain the Submission Queue Length at zero.
· Apply the following script: \MSExchangeTransport Queues(_total)\Submission Queue Length
Maintain CPU utilization at less than 75%.
· Plan and conserve amount of resources required to maintain acceptable level of service.
Light-emitting diode (LED) light needs to be monitored for blinking.
· The hard drive LED should maintain a blinking state. Any other state (light always on) indicates thrashing. It is important to maintain adequate disk input/output and utilize memory swapping to prevent thrashing.
FPE (or FSE) databases should be monitored and routinely maintained.
· Clean/purge database to reduce size.
· Enable retention policies.
Note: If you are expecting the incident database to grow soon, it is a good idea to allocate more free space for the database.
Apply service packs as required.
· Perform update for all required service packs on a regular basis.
Note: Exchange Server fixes are distributed in service packs. Service packs keep the product current. Service packs include updates, system administration tools, drivers, and additional components. If you do not stay up to date on service packs, performance and product functionality are seriously compromised.
Reference the previous tool for FSE Version 10 with SP1: http://go.microsoft.com/fwlink/?LinkId=179850
Use this tool with the following guidance:
· If using the recommended hardware of a 4 core processor, you should be able to reduce CPU utilization by 20% and increase the number of supported users by 25%.
· Increase the amount of expected server memory by 500 MB.
If you have comments or questions, please contact us at: firstname.lastname@example.org
Senior Program Manager
We are proposing the following solution:
2 x HUB\CAS Role (NLB Config)
2 x MBX Roles
We are planning to deploy Exchange Ent to the MBX Servers. We were hoping to deploy Std to the HUB\CAS Servers, but it appears that if Forefront is being installed for Scanning on the Hub Servers....that Enterprise Exchange is required....is this true?
Forefront does not place any requirements on which version of Exchange is installed, enterprise or standard. As long as the Exchange requirements are met, Forefront will scan fine.
Did you read this somewhere in the Forefront documentation?