NOTE: For more comprehensive coverage of Exchange 2007 storage as well as final RTM recommendations, please see the following blog post:
This is the first of a four part Web log where I will focus on the different server roles in Exchange 12 and their disk I/O characteristics. Over the next few months I'll be addressing other Exchange 12 storage concerns in three upcoming Web logs:
As with all server applications, Exchange 12 servers need to be properly deployed with sufficient, storage capacity and performance. The server roles in Exchange 12; Hub Transport Servers and Edge Transport Servers (collectively referred to in this blog entry as Transport servers), Client Access Servers, Unified Messaging Servers, and Mailbox Servers have different storage requirements and different backup and restore requirements, in part because they carry out different functions:
The Exchange 12 Mailbox Server role is the core server role upon which all other server roles build. Once you determine your mailbox profile, which includes user I/O per second and capacity, you can begin to plan your deployment. How many users you place on an Exchange Server is usually a mixture of preventing a hardware bottleneck and the ability to backup and restore that data within your Service Level Agreement (SLA).
There are three storage requirements that must be balanced to achieve a successful Exchange 12 deployment. The first is the transactional I/O, or the performance measured in latency for each I/O to be satisfied by the storage. The second is the backup and restore throughput, or how quickly your data can be moved to and from your backup medium. The third is capacity, or ensuring you have enough space in the chosen RAID configuration for your production LUNs, and on the target backup medium.
In Optimizing Storage for Exchange Server 2003, we outline how to size your disk I/O requirements using mailbox profiling. For example, say you want to place 3000 users on a server with a .4 IOPS profile with 2GB mailboxes. Your performance requirement would be 1200 I/Os per second (IOPS). You would have to ensure that you could backup and restore 6 terabytes of information. If your backup SLA is 4 hours, you would have to backup 1.5TB of data in an hour or 417MB a second. If your backup solution could only backup 300MB a second, you'd have to reduce the mailbox size or the number of users by 28%.
In Exchange 2000 Server, the best practice, which was influenced by virtual memory limitations, was to fill out a storage group with 5 databases before creating another storage group. In Exchange Server 2003, those limitations were considerably reduced, and the best practice is to add an additional storage group for each new database until the maximum number of storage groups has been created. With Exchange 12, the I/O footprint is reduced due to enhancements to JET, the underlying database engine used by Exchange.
Core JET Enhancements
Exchange 12 reduces the overall I/O footprint for Exchange because of several key design changes in JET:
- A 64-bit operating system and a 64-bit Exchange Server enable a much larger database cache, increasing from 900 MB to potentially dozens of gigabytes, depending on the total system memory.
- Database read operations also benefit from many new cache optimizations. I/O coalescing increase from 64 KB to 1 MB further reduces disk I/O by increasing the opportunity to read and write larger I/O.
- There is no streaming databases (STM) file and the ExIFS has been removed.
- In Enterprise edition up to 50 databases in up to 50 storage groups can be utilized on a single server.
As a 64-bit application, Exchange 12 does not have the virtual memory limitations of its 32-bit predecessors. Exchange 12 supports up to 50 databases and up to 50 storage groups, but you cannot place more than 5 databases in a storage group. Each storage group creates its own separate transaction log and is the basic unit for backup and restore. The maximum amount of data that JET can write to the transaction log before it writes it to the database is a cache called the checkpoint depth. Using 1 database in a storage group allocates the entire checkpoint depth to that database, increasing the likelihood that multiple updates to a database page will be done in cache, and only the last update will be written to the database, thereby reducing I/O.
Exchange Mailbox Data Components
Mailbox activities that impact disk I/O
How it impacts disk I/O
JET database (.edb file)
The Exchange 12 Mailbox Server stores all mail in a JET database. The JET database is randomly accessed and uses an 8 KB page size, though I/O coalescing can result in larger I/Os. For reliability, and in some cases performance reasons, the database should be on disks that do not contain transaction logs.
Transaction log files (.log files)
All changes made to the database are first committed to the transaction log, which is a sequential write to the disk. The writes vary in size from 512 bytes to the log buffer size.
There has been much development work to improve the search feature in Exchange 12. Content indexing is a random workload that should be placed on the same LUN as your database and will typically be about 5% of the database size. Since content indexing runs in the background indexing messages as they arrive, the disk I/O impact is minimal.
If a process requests a page in memory and the system cannot find the page at the requested location, a page fault occurs. If the page is elsewhere in memory, the fault is a soft page fault. If the page must be retrieved from disk, the fault is a hard page fault. Most processors can handle large numbers of soft page faults without consequence. However, hard page faults can cause significant delays. Continuous high rates of disk paging indicate a memory shortage.
Most content conversion is done on the CAS and Hub Transport servers. Legacy WebDAV content conversion, for legacy Outlook Web Access clients, occurs on the Exchange 2003 Mailbox Server. When a client requests data that must be converted on a CAS server, the data is accessed from the Exchange 2003 Mailbox server, converted in the Mailbox server's TMP folder, and sent to the CAS server. To improve performance the TMP folder should not be on the same LUN as the page file and operating system.
The Exchange 12 information store requires periodic online maintenance to be run against each database. The two tasks that impact disk I/O are the hard deletion of messages and mailboxes that are older than the configured retention policy, and online defragmentation of the database. Since backup will halt online defragmentation, care must be taken to give both backup and database maintenance exclusive windows of time to complete their tasks.
Backup & Restore
The process of backing up data requires that data be read from your database and transaction log file volumes. This additional I/O can impact user response times and should be avoided during business hours.
The process of soft recovery requires that JET play back all of the transaction log files. This causes the I/O profile to be a sequential read stream. As a result, the recovery performance improves if the transaction log files are on a disk with fast sequential disk access.
Detailed Backup & Restore strategies will be outlined in the forthcoming blog, Exchange 12 Backup and Restore.
Backing up mailboxes requires careful planning and below I outline some considerations for Volume Shadow Copy Service (VSS) and streaming online backups. There are trade-offs in every solution that impact variables such as cost, time, and reliability. Most administrators define a window for online database maintenance and defragmentation, and scheduled operating system maintenance. This competes with the backup window and special care is needed to balance backup, maintenance, and production load. Larger mailboxes may make a daily full backup strategy impossible to complete within your SLA. A common strategy to lessen the pain of a full nightly backup is to perform weekly full and daily differentials. With this strategy, you need to recover the full backup, and then recover the last differential backup.
Volume Shadow Copy Service (VSS)
The article entitled Best Practices for Using Volume Shadow Copy Service with Exchange Server 2003 details VSS fundamentals and Exchange 2003 best practices, and I recommend that you read it (and not just because I wrote it). In addition to everything covered in that article, there are two major VSS-related considerations in Exchange 12 that I want to focus on; larger mailboxes, and the ability to backup a CCR & LCR replica. While VSS backups can be made on either the production or replica data, we recommended that you backup the replica and avoid an impact to the production physical disks.
With LCR, Exchange 12 is replicating Exchange data to a separate disk on the same server. When using VSS clones on the replica, the replica storage should be configured on different physical disks so that the clone operation and checksum integrity do not impact the production physical disks. With VSS snapshots on the replica, the replica storage should be configured on different physical disks so that the checksum integrity and subsequent streaming to tape won't impact your production physical disks.
With CCR, Exchange 12 is replicating Exchange data to a separate server. This server is a node in the cluster, but the target replica is not shared storage. When using VSS clones, you are able to run the checksum integrity on the replicated copy on the passive node using different physical disks thereby isolating the backup process. With VSS snapshots, the checksum integrity and subsequent streaming to tape won't impact your production server or physical disks.
Streaming Online Backup
Unlike VSS, where the data is typically moved within the storage appliance, when using streaming backups, the server is responsible for moving your data. You don't have to worry about the performance impact of the checksum integrity process because every page is checked during the backup. Whereas VSS can only backup one storage group at a time, a streaming backup can backup multiple storage groups concurrently. These multiple streams can stress the limits of your backup media, whether it is gigabit Ethernet or fiber channel attached tape. For many customers, the backup SLA window, divided by the throughput-per-minute of their streaming backup medium, limit the permissible size of the storage group. For example; if you had a 1-hour SLA on your storage group, and can backup 100MB/sec, your maximum storage group size would be 360GB.
Client Access Server (CAS)
The Client Access Server offloads many stateless tasks from the Mailbox server (assuming the roles are installed on different physical servers), and provides a unified namespace where your users need only point to a single name regardless of which Mailbox server they are on. Internet protocols such as IMAP4, POP3, and HTTP (OWA) are serviced by the Client Access Server. Outlook Anywhere (formerly known as RPC over HTTP), ActiveSync, Autodiscovery and Web Services are other examples of features serviced by the Client Access Server.
The Client Access Server can be bottlenecked by CPU, memory, and network, yet it has a very small disk I/O footprint. SMTP traffic, a potential disk I/O consideration in an Exchange 200x front-end server, is now owned exclusively by the Hub Transport and Edge Transport servers.
CAS Server activities that impact disk I/O
Why it impacts disk I/O
Protocol logging is a sequential write that, if enabled, will be a performance hit and consume disk space to store the logs. When you keep a history of the protocol you choose to log, you can verify whether the protocol is performing its work as expected or is experiencing communications problems, and identify attacks from the Internet.
Content conversion for all Exchange 12 protocols occurs on the Client Access server. When a client requests data that must be converted on a Client Access server, the data is accessed from the Mailbox server, converted in the Client Access server's TMP folder, and sent to the client. Legacy Outlook Web Access WebDAV clients have their content converted on the Exchange 2003 Mailbox server. To improve performance the TMP folder should not be on the same LUN as the paging file and operating system.
See Paging under Exchange Mailbox Data Components
The Hub Transport and Edge Transport servers are the bridgehead and gateway for Exchange 12. Their primary mission is to send and receive mail. Many businesses will deploy a transport server into two groups; message hygiene (Edge Transport Server) and routing (Hub Transport Server). The Edge Transport Server's primary responsibility is to protect the Exchange infrastructure from incoming mail containing spam or viruses. The Hub Transport server will then categorize the clean mail, and deliver it to the proper Mailbox server. The storage impact to these servers will fluctuate depending on the number of messages handled per second, and the average size of those messages.
Edge & Hub Transport Server activities that impact disk I/O
JET database (mail.que file)
Both the Exchange 12 Edge Transport and & Hub Transport Server store all mail in a JET database. The JET database is randomly accessed and uses an 8 KB page size. For reliability, and in some cases performance reasons, the database should be on separate disks from the transaction logs.
Protocol Logging & Message Tracking Logs
Message tracking and the protocol logging is a sequential write that if enabled, will be a disk performance hit and consume disk space to store the logs. When you keep a history of the protocol you choose to log, you can verify whether the protocol is performing its work as expected or is experiencing communications problems, and identify attacks from the Internet.
On the Hub Transport server, incoming mail from the Internet is converted to MAPI prior to being delivered. This content conversion process occurs in the TMP folder. To improve performance, the TMP folder should not be on the same LUN as the paging file and the operating system.
Customization of the Transport Server is done via code, known as agents, running in the CLR environment and triggered by an event. Some agents log data which will be a disk performance hit and consume disk space to store the logs.
I hope this helped you identify how disk I/O impacts Exchange 12 based on the server roles. I am looking forward to adding more content with my next blog, where I will focus on Exchange 12 high availability storage considerations.
VSS Supportability KB: 822896
- Robert Quimbey