Infrastructure snapshots : Exchange

Demystifying Exchange 2010 database availability group (DAG)

mbaher@microsoft.com — Mon, 27 Jul 2009 18:40:00 GMT

What is a DAG?

A database availability group (DAG) is the base component of the high availability and site resilience framework that is built into Exchange 2010. A DAG is a group of up to 16 Mailbox servers that host a set of databases and provide automatic database-level recovery from failures that affect individual servers or databases. Exchange 2010 uses the same continuous replication technology found in Exchange 2007. Exchange 2010 combines on-site data replication (CCR) and off-site data replication (SCR) into a single framework which is the DAG. Once servers have been added to a DAG, administrators can add replicated database copies incrementally, and Exchange 2010 switches between these copies automatically, as needed, to maintain availability.

When can I create a DAG?

After you've deployed Exchange 2010, you can create a DAG, add Mailbox servers to the DAG, and then replicate mailbox databases between the DAG members. A DAG can be created using the New Database Availability Group wizard in the Exchange Management Console, or by running the New-DatabaseAvailabilityGroup cmdlet in the Exchange Management Shell. When creating a DAG, you provide a name for the DAG, and optional witness server and witness directory settings. In addition, one or more IP addresses are assigned to the DAG, either by using static IP addresses or by allowing the DAG to be automatically assigned the necessary IP addresses using Dynamic Host Configuration Protocol (DHCP).

Do I need to setup windows cluster for the DAG to work?

No, there is nothing called a standalone or clustered Exchange 2010 installation. After you install a normal Exchange 2010 mailbox server, you need to run the New-DatabaseAvailabilityGroup cmdlet to create a DAG, once the DAG has been created, mailbox servers can be added to the DAG. When the first server is added to the DAG, a cluster is formed for use by the DAG. DAGs make limited use of Windows Failover Clustering technology, namely the cluster heartbeat, cluster networks, and the cluster database (for storing data that changes or can change quickly, such as database state changes from active to passive or vice versa, or from mounted to dismounted and vice versa). As each subsequent server is added to the DAG, it is joined to the underlying cluster (and the cluster's quorum model is automatically adjusted by the system, as needed), and the server is added to the DAG object in Active Directory. And because DAGs rely on Windows Failover Clustering, they can only be created on Exchange 2010 Mailbox servers that are running Windows Server 2008 Enterprise Edition or Windows Server 2008 R2 Enterprise Edition. In addition, each Mailbox server in the DAG must have at least two network interface cards in order to be supported

What's happening when I create a DAG or join a server to an existing DAG?

When the first Mailbox server is added to a DAG, the following occurs:

- The Windows Failover Clustering component is installed, if it is not already installed.

- A failover cluster is created using the name of the DAG.

- A cluster network object (CNO) is created in default computers container.

- The name and IP address of the DAG is registered as a Host (A) record in DNS.

- The server is added to DAG object in Active Directory.

- The cluster database is updated with information on the databases that are mounted on the added server.

When the second and subsequent servers are added to the DAG, the following occurs:

- The server is joined to Windows Failover Cluster for the DAG.

- The quorum model is automatically adjusted:

- A Node Majority quorum model is used for DAGs with an odd number of members.

- A Node and File Share Majority quorum is used for DAGs with an even number of members.

- The witness directory and share are automatically created by Exchange when needed.

- The server is added to DAG object in Active Directory.

- The cluster database is updated with info on mounted databases

Can I have DAG members from different subnets?

Yes, during the cluster creation, the Add-DatabaseAvailabilityGroupServer task retrieves the IP address(es) configured while you are creating the DAG, takes whatever appropriate IP and ignores the ones don't match any of the subnets found on the server. This gives you the flexibility to have a DAG with members on the same or different subnets "in case you will have a DAG node in another datacenter".

Can I use a 3rd party replication tool to replicate the databases in the DAG?

By default, a DAG is designed to use the built-in continuous replication feature to replicate mailbox databases between servers in the DAG. If you are using third-party data replication that supports the Third Party Replication API in Exchange 2010, you must create the DAG in third party replication mode by using the New-DatabaseAvailabilityGroup cmdlet with the ThirdPartyReplication parameter, but note that Once this mode is enabled, it cannot be disabled.

Can I encrypt the DAG network traffic?

DAGs support the use of encryption by leveraging the encryption capabilities of the Windows Server operating system. DAGs use Kerberos authentication between Exchange servers. Microsoft Kerberos SSP’s EncryptMessage/DecryptMessage APIs handle encryption of DAG network traffic. Microsoft Kerberos SSP supports multiple encryption algorithms. The Kerberos authentication handshake picks the strongest encryption protocol supported in the list: typically AES 256-bit, potentially with a SHA Hash Message Authentication Code (HMAC) to maintain integrity of the data

Can I compress the DAG network communication?

DAGs also support built-in compression. When compression is enabled, DAG network communication uses XPRESS, which is Microsoft’s implementation of the LZ77 algorithm. This is the same type of compression used in many Microsoft protocols, in particular, MAPI RPC compression between Outlook and Exchange

What is the minimum network interfaces required for a DAG?

As indicated above, each Mailbox server in the DAG must have at least two network interface cards in order to be supported. In Exchange 2010, DAGs primarily have two types of networks: MAPI networks, which are used by other Exchange 2010 servers to communicate with the Mailbox server in the DAG, and Replication networks, which are used for log shipping and seeding within the DAG. So the reason to have at least two NICs is that this configuration enables you to configure one MAPI network and one Replication network.

What will happen if one of my DAG networks encountered a failure?

In the event of a failure affecting the MAPI network, a server failover will occur (assuming there are healthy mailbox database copies that can be activated). In the event of a failure affecting the Replication network, if the MAPI network is unaffected by the failure, log shipping and seeding operations will revert to use the MAPI network. When the failed Replication network is restored, log shipping and seeding operations will revert back to the Replication network. To increase the high availability on your DAG, additional MAPI and/or Replication networks can be added, as needed. Also you can prevent an individual network from being a single point of failure by using network adapter teaming or similar technology.

Can I host other roles on a mailbox server that is member of a DAG?

Unlike Exchange 2007, where clustered mailbox servers required dedicated hardware, Mailbox servers in a DAG can host other Exchange roles (Client Access, Hub Transport, Unified Messaging), providing full redundancy of Exchange services and data with just two servers. This can be an excellent option for small and medium organizations where the number of mailboxes and email traffic doesn't require a dedicated hardware for each role.

Facts about Exchange 2007 SP2…

mbaher@microsoft.com — Sun, 26 Jul 2009 18:26:00 GMT

Exchange 2007 SP2 is coming very soon and I thought of highlighting some important facts about it:

Installations of Exchange 2007 SP2 will require Windows Installer 4.5
Exchange 2007 SP2 will Include Exchange 2010 schema. This will not eliminate the need however to run PrepareSchema when you deploy Exchange 2010 as we still have some checks we want to run and preps we need to do for 2010. However you need to plan for a schema preparation before deploying SP2.
New blocking setup prerequisites to address: http://msexchangeteam.com/archive/2008/09/05/449764.aspx
Exchange Volume Snapshot Backup Functionality - A new backup plug-in has been added to the product that will enable customers to create Exchange backups when a backup is invoked through the Windows Server 2008 Backup tool. Exchange Server 2007 didn't have this capability on Windows Server 2008 and additional solutions were required to perform this task.

Update Rollup 9 for Exchange Server 2007 Service Pack 1 has been released

mbaher@microsoft.com — Sun, 26 Jul 2009 16:45:00 GMT

We have released Update Rollup 9 for Exchange Server 2007 Service Pack 1 (KB 970162) to the download center. The release of the rollup via Microsoft Update will happen on July 28. Read the details at http://msexchangeteam.com/archive/2009/07/17/451835.aspx also KB 970162 has more details about this release and a complete list of all fixes included in this rollup.

Note this important point in the article “Support for Windows Server 2008 R2 Domain Controllers in the environment (Note: Exchange Server 2007 itself is not supported to be installed on a Windows Server 2008 R2 system”

Large Message Processing in Exchange, Part 1: Prevention and Planning

mbaher@microsoft.com — Sun, 26 Jul 2009 16:38:00 GMT

An Excellent article that you should read at http://msexchangeteam.com/archive/2009/07/07/451737.aspx

Thanks Scott Landry, Mohammad Nadeem and Bill Long for this great blog,

Exchange 2010 Archiving... Why to Archive???

mbaher@microsoft.com — Sat, 25 Jul 2009 00:25:00 GMT

From few weeks I was reading about Exchange 2010 archiving and I thought of blogging about it to summarize what we can do with Exchange 2010 Archiving and what is the advantage from using it.

Why Archive E-Mail:

As data volume grows inside the mailboxes, outlook performance starts to decrease
Archiving helps in eliminating PSTs. Let’s not forget that PSTs also add to further performance/management issues and adds lot of challenges to compliance and discovery
Manual retrieval cost for old data can be huge (backup tapes, PSTs, etc...)
PSTs are accessible on local machine only
PSTs can be easily corrupted “can be a big issue without a backup”
Lost laptop results in exposure of PSTs content

Exchange 2010 Archiving features:

Secondary Personal Archive. It acts as a secondary mailbox node. The archiving mailbox must be in the same database hosting the primary mailbox
Controlled Retention Policies. It provides integrated retention policies on the folder/item level
Multi Mailbox Search. Delegated non-IT compliance officers, HR, legal, etc... Can do a legal multi-mailbox search using a role-based GUI. Search can be filtered using multi criteria such as sender, receiver, expiry policy, message size, IRM protected items, etc...
Legal Hold. Suspected mailbox can be put on legal hold to be included in a multi-mailbox search. Users can be notified in their outlook that their mailbox is under legal hold.
Ease of Access. Appears alongside a user's primary mailbox in outlook or outlook web access
Easy PST Migration. Supported drag & drop for PSTs to the personal archive for easy migration of existing PSTs
Controlled Quota. Archive quota can be set separately from primary mailbox
Retention Policies Integration. Mail in primary mailbox can be moved to the archive through integrated retention policies.

Advantage of Archive:

Compliance & Discovery. PSTs are difficult to be discovered by IT administrators. With Archiving it's possible for data discovery and legal hold for all mailbox content
Data Protection. With PSTs it is very complex to protect the data inside the PSTs by the company backup system. With Archiving all mailbox content can be protected by the backup system.
Quota Management. Previously users were forced to keep track on their quota and move data to their PSTs. Archiving allow for large mailboxes and for automatic archiving to the old data to the archive mailbox.
Reduced Cost. No additional licenses required, it's a part of the Exchange 2010 ECAL
Multi Storage Options. With Exchange 2010 & archiving you can host your data on DAS-SATA storage architecture to reduce storage costs

So looking to the above, I can say that for most of the organizations who will be looking for getting rid of PSTs and giving large mailboxes for their users, Exchange 2010 Archiving will be your arm to do that.

Common Unsupported Exchange 2007 Configurations

mbaher@microsoft.com — Wed, 17 Jun 2009 12:15:00 GMT

This is an interesting section in the Exchange 2007 help file that i wanted to share and recommend you to read it,

http://technet.microsoft.com/en-us/library/dd577077.aspx

Exchange Server 2007 Service Pack 2 available in Q3 2009

mbaher@microsoft.com — Mon, 18 May 2009 18:17:00 GMT

Key new features of Exchange Server 2007 SP2 unveiled today include:

Enhanced Auditing - New Exchange auditing events and audit log repository enable Exchange administrators to more easily audit the activities occurring on their Exchange servers. It allows the right balance of granularity, performance, and easy access to audited events via a dedicated audit log repository. This simplifies the auditing process and makes review of audited events easier by segregating audited events in a dedicated location.
Exchange Volume Snapshot Backup Functionality - A new backup plug-in has been added to the product that will enable customers to create Exchange backups when a backup is invoked through the Windows Server 2008 Backup tool. Exchange Server 2007 didn't have this capability on Windows Server 2008 and additional solutions were required to perform this task.
Dynamic Active Directory Schema Update and Validation - The dynamic AD schema update and validation feature allows for future schema updates to be dynamic deployed as well as proactively preventing conflicts whenever a new property is added to the AD schema. Once this capability is deployed it will enable easier management of future schema updates and will prevent support issues when adding properties that don't exist in the AD schema.
Public Folder Quota Management - SP2 enables a consistent way to manage quotas by improving the current PowerShell cmdlets to perform quota management tasks.
Centralized Organizational Settings - SP2 introduces new PowerShell option that enable centralized management of many of the Exchange organization settings.
Named Properties cmdlets - SP2 enables Exchange administrators to monitor their named property usage per database.
New User Interface for Managing Diagnostic Logging- SP2 enables Exchange administrators to easily configure and manage diagnostic logging from within the Exchange Management Console.

Read the full article at http://msexchangeteam.com/archive/2009/05/11/451281.aspx

Exchange 2010 setup wizard

mbaher@microsoft.com — Fri, 01 May 2009 22:47:00 GMT

In this article I’ll give you a tour for the Exchange 2010 setup wizard, if you deployed Exchange 2007 before you will be familiar of the Exchange 2010 setup wizard, if not then here is how it looks:

Finally here is the Exchange 2010 EMC.

Exchange 2010 setup prerequisites http://technet.microsoft.com/en-us/library/bb691354(EXCHG.140).aspx

Exchange 2010 beta is online

mbaher@microsoft.com — Wed, 15 Apr 2009 15:23:00 GMT

Wanted to test and play with Exchange 2010, go and get the beta from here http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=1898ed2c-2f88-48ac-824e-d3d20fad77d7 also you can find the beta documentation at http://technet.microsoft.com/en-us/library/bb124558(EXCHG.140).aspx

Enjoy,

Exchange 2000/2003 to Exchange 2007 migration guidelines

mbaher@microsoft.com — Wed, 04 Jun 2008 14:37:00 GMT

I've worked on many projects involving migrating Exchange 2000/2003 to Exchange 2007 and from these migration planning i can list the common guidelines that can help in doing your migration planning. The transition process from Exchange 2003 to Exchange 2007 is a relatively straightforward process and involves the following high level tasks:

1. Prepare the Active Directory directory service, you can extend the Active Directory schema and create the Active Directory objects and universal security groups to support Exchange 2007 before you install the server roles. Run setup.com /PrepareAD from the command line on the domain controller that is the schema master at the forest root. When you run setup.com /PrepareAD, the task setup.com /PrepareLegacyExchangePermissions also runs to create the universal security group that is granted permissions to send e-mail to Exchange 2007 Hub Transport servers

2. Supress minor link state updates on all Exchange 2003 front end servers

3. Deploy Exchange 2007 CAS/HUB servers in the same organization/forest as the existing Exchange 2003 servers and choose FE_SERVERNAME as the routing group connector destination

Note: Using an Exchange Server 2003 front-end server together with an Exchange 2007 Mailbox server is not supported.

4. Configure the routing group connector to use all Hubs servers

5. Deploy the Edge servers

6. Configure the external firewall to allow for outbound SMTP traffic from the Edge servers

7. Configure Edge Sync subscription and rest of the Edge settings

8. Configure the external firewall 1:1 NAT for both Edge servers to allow for incoming SMTP traffic

9. Delete the old SMTP connectors

10. Deploy Exchange 2007 mailbox servers

11. Configure Public folder replication

12. Configure the CAS web services virtual directories by following the below steps:

a. Configure CAS OWA virtual directories to support integrated authentication and to set the the external URL

b. Configure Outlook Anywhere internal/external URLs with Integrated authentication and set the External/Internal URLs

c. Configure ActiveSync external URL

d. On the Exchange 2003 backend clusters; configure the ActiveSync virtual directory to use Integrated Authentication

e. Configure OAB URLs using the Exchange Management Console and configure it to use HTTPS instead of HTTP

13. Configure rest of the CAS settings

14. Publish the ActiveSync through ISA 2006

15. Test co-existence between both servers:

a. GAL co-existence is automatically achieved since all servers share the same Active Directory information. This means the GAL will appear the same for both Exchange 2007 and 2003 users

b. Email connectivity between Exchange 2003 to Exchange 2007 servers will automatically be enabled because of the Routing Group Connector created by the installation of the Hub server role

16. Move sample mailboxes using the Exchange 2007 move mailbox wizard or cmdlet as a test for the migration

Note: You do not need to move your existing contacts or distribution groups. They will be available in Active Directory even if you remove your Exchange 2003 or Exchange 2000 servers

17. On an Exchange 2007 server, for each offline address book (OAB), move the generation process to an Exchange 2007 server. For detailed steps, see How to Move the Offline Address Book Generation Process to Another Server.

18. Upgrade the email address policies

19. Upgrade the address lists

Note: Use the LDAP to OPATH converter tool

20. Change the routing group connectors to use the Exchange 2003 backend server

21. Decommission Exchange 2003 Front End servers

22. Move rest of the mailboxes to the Exchange 2007 mailbox cluster

Note: If you have any Exchange 2003 or Exchange 2000 recipient policies that have not been applied, moving the mailboxes to an Exchange 2007 server will force the recipient policies to be re-evaluated and applied. Before you move mailboxes, make sure that you want to apply all of the existing recipient policies. If you have an existing recipient policy that you do not want to apply, clear the automatically update e-mail address based on e-mail address policy check box in Active Directory Users and Computers. For more information, see the Exchange Server Team Blog article Yes, Exchange 2007 really enforces Email Address Policies.

Note: The old Exchange 2003 backend will be running for at least 1 week in order to allow Outlook clients to automatically update their profile to point to the new Exchange 2007 mailbox cluster, otherwise, if the cluster is decommissioned immediately, clients will not be able to access their mailboxes until their profiles are modified to use the new Exchange mailbox server. KOC must ensure that all Outlook clients will logon at least once during this week for their profiles to be updated. Clients that do not logon before the old cluster is decommissioned must be updated manually. Exprofile can be configured to run as a logon script for the MAPI users to automatically configure their Outlook profiles to use the new mailbox server.

23. Remove the Exchange 2003 backend servers. The decommissioning of the Backend servers should be conducted after making sure that all clients are redirected to the Exchange 2007 servers

24. Remove the last Exchange 2003 server from the organization. For detailed steps, see http://technet.microsoft.com/en-us/library/bb288905(EXCHG.80).aspx

Note that these are just a guidelines to help in your migration planning, i'm not covering all of the scenarios and situations which will vary according to the environment design & setup.

Auto complete, Reply-ability, I need my X500!!!!

mbaher@microsoft.com — Sun, 17 Feb 2008 21:07:13 GMT

Sometimes i join discussions regarding Exchange cross-org migrations and the most hot topics is always about migration planning, mailboxes content migration, user’s notification, outlook profiles, etc… but what about the below two important points:

· Outlook auto complete feature (Automatic Name Resolution). After migration of mailboxes users will got NDR if they send an email using the outlook auto complete feature

· Reply-ability to old emails. After migration of mailboxes users will receive NDR if they tried to reply to an old email that is received from a user in the same Exchange organization

Why is this happening? Simply Outlook stores the recipient information using the LegacyExchangeDN value of the user’s mailbox, so when you do the migration of the mailbox you actually migrated the SMTP address & the mailbox content but you didn’t migrate his LegacyExchangeDN…

So how can we solve this? Simply we need to copy the LegacyExchangeDN value of the users’ mailboxes as an X500 address so that Exchange can deliver these emails. I will highlight some of these migrations scenarios to see how we can solve this:

- In managed migrations where there is a GalSync solution between the two organizations using MIIS or ILM the GalSync will stamp the LegacyExchangeDN value of the mailboxes in the source organization as an X500 address on the contacts created in the target organization.

- In Exchange 2007 we can use the move-mailbox cmdlet to move mailboxes between different Exchange organizations knowing that the target must be an Exchange 2007 org. The move-mailbox process will do the migration of the LegacyExchangeDN value of migrated mailboxes and stamp it as an X500 address to relief us from this pain.

But what about non-managed migrations or between organizations who don’t have a connectivity between their data centers, in these type of migrations you have to extract the LegacyExchangeDN value from the source mailboxes and stamp it as the X500 address to the target mailboxes in the target organization. One of the approaches that you can follow is using CSVDE to extract a CSV file from the source organization (Exchange 2000 or 2003) and manipulate it to have only two columns (alias & LegacyExchangeDN) then use the below powershell script to stamp the X500 address on the target mailboxes created in the Exchange 2007 organization,

Import-CSV file.csv | foreach {

$Temp = Get-Mailbox -identity $_.alias

$Temp.EmailAddresses += “X500” + $_.legacyexchangedn

set-mailbox -instance $temp}

so by having the X500 address on the migrated mailboxes Outlook auto complete and reply-ability to old emails will work as usual because the address Outlook tells Exchange to use is still valid.

Excessive paging on Exchange 2007 servers when working sets are trimmed

mbaher@microsoft.com — Sun, 10 Feb 2008 11:44:46 GMT

Mike posted a fantastic explanation for an excessive paging problem that you might get on Exchange 2007 servers on windows 2003. I advise you to read it and might proactively install it on your servers http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx

Who said that transaction goes from Logs to DB!!!!!

mbaher@microsoft.com — Tue, 22 Jan 2008 22:35:48 GMT

I've heard lot of discussions going about how Exchange ESE works and that the transactions goes from the Logs to the DB based on the checkpoint depth. I've collected some small info from different sources trying to explain how ESE writes transactions to the DB.

The following five subcomponents of ESE work together to move the data into the database and to its static form:

Log buffers When ESE first receives a transaction, it stores it in log buffers. These log buffers are used to hold information in memory before it is written to the transaction logs. By default, each buffer unit is the size of a disk sector, which means that it's 512 bytes in size. JET does some sanitation to make sure that the number of buffers is a minimum of 128 sectors, a maximum of 10,240 sectors, and aligns to the largest 64 KB boundary. So, for Exchange 2000 Server (and all service packs) the default number of log buffers is 84, which JET sanitizes to 128 so the actual buffer area is 64 Kbytes. For Exchange Server 2003, the default number of log buffers is 500, which JET sanitizes to 384 so the actual buffer area is 192 KB.
Log writer As the buffers fill up, ESE moves the data from the buffers onto disk and into the log files. In this operation, the transactions are committed to disk to the logs in a synchronous fashion. This process is fast, because it is crucial to move the data from memory and into the transaction logs quickly in case of a system failure.
IS buffers The IS or cache buffers are the first step toward turning a transaction into actual data. The IS buffers are a group of 4 kilobyte (KB)-pages allocated from memory by Exchange for the purpose of caching the database pages before they are written to disk. When first created, these pages are clean, because they have yet to have any transactions written to them. ESE then plays the transactions from the logs into these empty pages in memory, thereby changing their status to dirty. The default value for maximum size these buffers can reach is 900 MB in Exchange 2000 Server SP 3.
Version store ESE writes multiple, different transactions to a single page in memory. The version store keeps track of and manages these transactions. It also structures the pages as the transactions occur
Lazy writer At this point, ESE must flush the dirty pages out of memory. The lazy writer is responsible for moving the pages from the cache buffers to disk. Because there are so many transactions coming in at once and so many pages getting dirtied, the job of the lazy writer is to prioritize them and subsequently handle the task of moving them there without overloading the disk I/O subsystem. This is the last phase and the point at which the transactions have officially become static data. It is also at this point that the dirty pages are cleaned and ready for use again.

What about the checkpoint file and other components???

The Checkpoint File. The database engine maintains a checkpoint file called Edb.chk for every log file sequence in order to keep track of the data that has not yet been written to the database file on disk. The checkpoint file is a pointer in the log sequence that indicates where in the log file the information store needs to start the recovery in case of a failure. The checkpoint file is essential for efficient recovery. Without it, the information store would start from the beginning of the oldest log file on the disk and check every page in every log file to determine whether it had already been written to the database--a time-consuming process, especially if all you want to do is make the database consistent.

The Checkpoint depth. The checkpoint depth is a threshold that defines when ESE begins to aggressively flush dirty pages to the database.

ESE Cache. An area of memory reserved to the Information Store process (Store.exe) that is used to store database pages in memory. By storing pages in memory, this can reduce read I/Os, especially when the page is accessed many times; in addition, this cache can be used in two ways to reduce write I/Os - by storing the dirty page in memory, there is the potential for multiple changes to be made to the page before it is flushed to disk; also, the engine can write multiple pages together (up to 1MB) using write coalescing

In Exchange 2000 & Exchange 2003 we had two main files for each Exchange database which are:

The EDB file. The .edb file is the main repository for the mailbox data. The fundamental construct of the .edb file is the b-tree structure, which is only present in this file, and not in the .stm file. The b-tree is designed for quick access to many pages at once. The .edb file design permits a top level node and many child nodes. Tree depth has the greatest effect on performance. A uniform tree depth across the entire structure, where every leaf node or data page is equidistant from the root node, means database performance is consistent and predictable. In this way, the ESE 4 KB pages are arranged into tables that form a large database file containing Exchange data. The database is actually made up of multiple b-trees. These other ancillary trees hold indexing and views that work with the main tree. The .edb file is accessed by ESE directly.

The STM file. The .stm or streaming media file is used in conjunction with the .edb file to comprise the Exchange database. The purpose of the .stm file is to store streamed native Internet content. Unlike the .edb file mentioned previously, the .stm file does not store data in a b-tree structure. When a message arrives through the Internet or Simple Mail Transfer Protocol (SMTP), it always arrives as a stream of bytes. In Exchange Server 2003 and Exchange 2000 Server, these messages are streamed directly to the .stm file where they are held until accessed by a MAPI client. So the content is not converted. That way, if the end user is consistently accessing mail through POP3, the mail items are pulled directly from the .stm file and are already in the proper state for delivery. In the case that the message is accessed by a MAPI client, however, the message is moved over to the .edb file and converted to Exchange native form, and is never moved back to the .stm file.

Note that we removed STM file from the Exchange 2007.

So how do the above components work together?

An operation occurs against the database (e.g. client sends a new message), and the page that requires updating is read from the file and placed into the ESE cache (if it is not already in memory), while the log buffer is notified and records the operation in memory.
The changes are recorded by the database engine but are not immediately written to disk; instead, these changes are held in the ESE cache and are known as "dirty" pages because they have not been committed to the database. The version store is used to keep track of these changes, thus ensuring isolation and consistency are maintained.
As the database pages are changed, the log buffer is notified to commit the change, and the transaction is recorded in a transaction log file (which may or may not require a log roll and a start of a new log generation).
Eventually the dirty database pages are flushed to the database file.
The checkpoint is advanced.

In Exchange 2007 we changed the database architecture in four significant respects, this is to make use of the 64bit platform so the result is:

The streaming database (.stm) file has been removed from Exchange 2007.
Longer log file names are used, thereby enabling each storage group to generate as many as 2 billion log files before log file generation must be reset.
Transaction log file size has been reduced from 5 MB to 1 MB to support the new continuous replication features in Exchange 2007.
The database page size has increased from 4 KB to 8 KB.

And to push the ESE engine another step forward in terms of performance, Exchange SP1 introduced two major updates and some enhancements in the online maintenance process. I recommend reading part I at http://msexchangeteam.com/archive/2007/11/30/447640.aspx and Part II at http://msexchangeteam.com/archive/2007/12/06/447695.aspx

Installing service packs and hotfixes on Exchange 2007 with Forefront Security for Exchange installed.

mbaher@microsoft.com — Mon, 31 Dec 2007 13:15:35 GMT

Installing service packs and hot fixes on an Exchange 2007 server with Forefront Security for Exchange is a routine task that administrators should do, below are the information for how to install it:

Stop all Exchange-related services. Also, stop any Forefront Security for Exchange Server-related services that may be running after Exchange has stopped.
Run the FSCUtility tool to disconnect Forefront Security for Exchange Server from Exchange. To do this, follow these steps:
1. Start a command prompt, and then change the directory to that of the Forefront Security for Exchange Server installation.
2. Type the following command, and then press ENTER: FSCUtility /disable
Install the service pack or the hotfix.
Start the Exchange-related services, and then verify that all the Exchange-related services are running.
Stop all the Exchange-related services.
Run the FSCUtility tool to connect Forefront Security for Exchange Server to Exchange. To do this, follow these steps:
1. Start a command prompt, and then change the directory to that of the Forefront Security for Exchange Server installation.
2. Type the following command, and then press ENTER: FSCUtility /enable
Start the Exchange-related services.

Exchange 2007 SP1 ESE Changes - Part 2

mbaher@microsoft.com — Fri, 14 Dec 2007 20:02:24 GMT

This is part 2 of the very interesting article about the changes in ESE in Exchange 2007 SP1. Highly recommended for reading,

http://msexchangeteam.com/archive/2007/12/06/447695.aspx