You Had Me At EHLO... : Administration

New Exchange fixes may disrupt Blackberry, Goodlink and other services

Exchange — Fri, 28 Apr 2006 19:19:00 GMT

CALL TO ACTION

A change in Exchange permissioning behavior may impact mobile communications and other add-on applications for Exchange. Shared and resource mailboxes may also be affected. By evaluating in advance whether this change will impact your environment, you can take simple remediation steps to ensure that installation of a new Exchange update will not impact critical services. A script is available to help you identify accounts and applications in your environment that may be affected.

EXECUTIVE SUMMARY

A change has been made in how the "Send As" permission works in Microsoft Exchange. In the past, additional accounts could be granted the "Full Mailbox Access" permission to a mailbox and these accounts could then send mail as the mailbox owner. From now on, the "Send As" permission must be explicitly granted to additional accounts or they will not be able to send mail as the mailbox owner.

This change can affect add-on services that have relied on "Full Mailbox Access" alone for impersonating users to send messages on their behalf. For example, the user of a mobile email device may compose a message on the device. This message is transmitted to the mobile access service, which logs on to Exchange and sends the message as the user.

New rollups and service packs for both Exchange 2000 and Exchange 2003 will include this change, as will all updates and hotfixes for the Exchange Information Store service (store.exe).

The change was made several months ago and has been documented in Microsoft Knowledge Base Article 912918. However, many administrators have been caught by surprise after downloading an Exchange store update for a different issue. Therefore, the Knowledge Base article has been rewritten to more fully explain the change, and Microsoft will be publicizing the article widely, both internally and externally. A sample script has been added to the article that shows administrators how to quickly identify affected accounts and to correct their permissions, if necessary.

This change in permissions behavior will not keep Exchange mailbox owners from sending mail when logged on as themselves. But it may keep them from sending from a mobile device that impersonates them, or affect other applications or users who send mail as them. This change also does not affect "cross-forest," "resource forest" or mixed Exchange 5.5 and Exchange 2000/2003 installations.

Nonetheless, running the script right now is a good idea, both to get familiar with how it works, and to find out whether you have affected accounts you don't know about. You may have created resource or other shared mailboxes and forgotten to grant "Send As." Or you may be running scripts and applications that do not grant "Send As" when they should.

The script for finding affected accounts and granting them the right permissions is available from this link:

http://support.microsoft.com/kb/912918

FREQUENTLY ASKED QUESTIONS

WHY DID YOU MAKE THIS CHANGE IF IT BREAKS SOME APPLICATIONS?

The change was implemented because of multiple requests from customers, and it provides additional security functionality in several scenarios. For example, consider a disaster recovery situation where an Exchange administrator needs full access to all mailboxes in a database in order to merge or salvage mailbox contents. Before, you could not grant such access without also giving the administrator the ability to send as any user in the database.

Correcting problems created by the change is straightforward--just go to the mailbox owner account and grant the "Send As" permission to the account that needs to send as that mailbox.

The old behavior was confusing too. An administrator might explicitly deny "Send As" rights and the Deny would have no effect when an account had "Full Mailbox Access". The way it works now is easier to understand and administer.

IS THERE A REGISTRY KEY OR SOME OTHER WAY TO OVERRIDE THE NEW BEHAVIOR UNTIL I'M READY FOR IT?

No, there is not. Providing a registry key or other override was considered and rejected because it would allow temporarily overriding the enhanced security whenever someone wanted to.

Something to remember is that this change applies only to additional accounts that are granted "Full Mailbox Access." If you are the mailbox owner, you don't need additional "Send As" permission. In cross-forest or Windows NT 4 mixed domain scenarios, the "Associated External Account" is treated like the mailbox owner, and so is a delegate who has been granted "Full Mailbox Access." Microsoft Knowledge Base article 912918 discusses each of these scenarios and exceptions in detail.

WHAT EXACTLY DOES THE SCRIPT DO?

The script is pretty simple. It works on one Active Directory domain at a time. In its Export mode it finds all accounts in the domain that have "Full Mailbox Access" to a particular mailbox, but don't also have "Send As." It ignores accounts that already have both permissions. So the output file only contains accounts that might have a problem.

The Export file is tab-delimited. You can sort it and edit it in Notepad or Excel, and then feed the file right back into the script to grant "Send As" in bulk for all accounts listed in the file.

Full documentation and tips on using the script are included in the Knowledge Base article.

ARE THERE OTHER WAYS OF CORRECTING THE PROBLEM WITHOUT USING THE SCRIPT?

You can use the Active Directory Users & Computers console to set permissions on individual accounts. You can also grant "Send As" for all objects in a domain or container, and thus have the permission take effect by inheritance.

WHERE DO I GO FOR MORE INFORMATION?

Microsoft KnowledgeBase article 912918 is the place to start. It includes the script and the details about how all of this works - including information about Outlook delegation scenarios. To really dig in to how Exchange permissions work, settle in with these white papers:

Working with Active Directory Permissions in Exchange Server 2003

http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/ex2k3ad.mspx

Working with Store Permissions in Microsoft Exchange 2000 and 2003

http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/storperm.mspx

- Mike Lee

Knowing your limits

Exchange — Mon, 17 Apr 2006 21:09:00 GMT

Recently MSIT wanted to make some global changes to the retention time of public folders. In the past we had a global limit of 3 years retention on items in public folders. The plan was to change the default limit to 1 year, and then exempt folders as needed.

But there were a couple of complications...

After examining the public folder tree and gathering feedback from the user base, we found that about 30% of the public folder tree would need an exemption for one reason or another. About 100,000 folders give or take a sub tree. Worse, the folders to be exempted where "sprinkled" throughout the tree in no real pattern.

Secondly, in Exchange 2003, public folder age limits are enforced thru 3 mechanisms, per replica properties, database limits, and per folder properties. Database properties and folder properties could be accessed thru a variety of mechanisms (DAV, LDAP, and Exchange System Manager) but replica properties are accessible only thru ESM.

Age Limit Enforcement	Precedent
Per Replica	1^st
Per Database	2^nd
Per Folder	3^rd

Due to the number of public folders that needed to be exempted, and problems around manually keeping each folder replica in sync, using Exchange System Manager to set per replica limits wasn't going to work. Since database policies had higher precedence then per folder limits, we decided that we would have to set all folders at the folder level and then turn off database and replica limits (fortunately, only a few folders had replica limits.)

In order to get the data we needed, we first used PFDAVAdmin to extract information about the tree. We needed the display name, folder path, folder size, item count, the PR_OVERALL_AGE property (this is the per folder age limit property), and the PERMANENTURL (mapi prop:0x670e001f)

Exchange 2003 SP2 introduced a number of cool new features, one of which is a wizard for doing large scale modifications of public folders. Since the folders where scattered all around, we couldn't use this tool to set the age limits on the folders. PFDavAdmin doesn't support importing modifications of these types of properties so it was out too. Time to write some code! Fortunately, there are lots of examples of how to do this; MSDN has a good one here.

Basically we need to do a DAV propatch something like this on each folder:

queryBuilder = new StringBuilder();

queryBuilder.Append(@"<?xml version=""1.0""?>");

queryBuilder.Append(@"<b:propertyupdate xmlns:a=""http://schemas.microsoft.com/mapi/proptag/"" xmlns:b=""DAV:"" xmlns:z=""urn:schemas-microsoft-com:datatypes"">");

queryBuilder.Append(@"<b:set>");

queryBuilder.Append(@"<b:prop>");

queryBuilder.Append(@"<a:0x66990003 z:dt='int'>" + Int32.Parse(ageLimit) + @"</a:0x66990003>");

queryBuilder.Append(@"</b:prop>");

queryBuilder.Append(@"</b:set>");

queryBuilder.Append(@"</b:propertyupdate>");

We still needed to have a way to identify each folder. Since users can modify display names, move folders, and include symbols which maybe hard to code into a URL, we need a better URL naming mechanism. Fortunately, this already exsisted, the permenateurl property we had gathered ealier. Basically, each public folder in Exchange can be identified by a URL that looks something like the following:

http://myPublicFolderServer/ExAdmin/Admin/<FQDN>/public%20folders/-FlatUrlSpace-/f85c74f57f7280469fa523b239cea3e3-1e834051

We now have all the information we need to set the folder age limits, the desired age limit, the url of the folder, and a tool to stamp each one. Any public folder server will work, since the age limit is a property on a folder, and all public folder servers have a copy of the folder hiearchy. We're ready to go.

The final step was to batch the folders into smaller groups. Making large scale changes to public folder properties can result in mail replication storms as each PF server sends out and recieves change notifications. We needed to ensure that our changes where small enough that we didn't cause any problems. After a little testing in our lab, and carefully monitoring the impact on our infrastrucutre, we found the magic number (which will vary from deployment to deployment, you'll have to do this on your own) and a week or two later had removed all the stale unwanted data from our environment.

- Ted Kolvoord

Surprise your users with an April Fool's Day OWA 2003 Theme

Exchange — Sat, 01 Apr 2006 08:45:00 GMT

I just posted instructions on how to create this OWA theme on my blog:

In addition to the cow look, it also has a moooooooooooo for a new mail sound, and a cowbell for the reminder.

Have fun!

- KC Lemson

Fix available to alleviate event ID 9548

Exchange — Wed, 22 Mar 2006 18:55:00 GMT

No doubt many of you are familiar with the event commonly seen in application logs of Exchange 2000 and 2003 servers, MSExchangeIS event 9548, indicating that the information store came across a disabled user who is missing the msExchMasterAccountSid attribute while processing some various task. There are many KB articles associated with this event, such as 291151, 326990, 278966, 328880, 316047, and possibly more. There have also been countless support cases where this design was at least a contributing factor. Almost every application log from an Exchange 2000 or 2003 server ever seen has likely been littered with 9548 events, to the point where it has become an annoyance event. For additional information on the event and the related information, I suggest reading this article: http://www.msexchange.org/articles/NoMAS-Tool.html.

A CDCR (Critical Design Change Request) was accepted last year to resolve the issue without having to run tools or scripts, and the first version of this fix was released last week. The KB article is 903158.

The problem was that a decision was made during the development of Exchange 2000 that every disabled user account with a mailbox had to have the msExchMasterAccountSid attribute. This is because in order for a mailbox to function within the store, a SID must be associated with the mailbox. The logic worked like this:

1. If the user account associated with the mailbox is disabled, and has msExchMasterAccountSid, AND msExchMasterAccountSid is not the well-known SELF SID, then msExchMasterAccountSid is the SID that is associated with the mailbox.

2. If the user account is enabled, or if it is disabled and has msExchMasterAccountSid set to the well-known SELF SID, then use objectSid.

3. If the user account is disabled, and has no msExchMasterAccountSid, OR if we cannot tell if the user is enabled or disabled due to access control on the user object, or if we cannot read the objectSid due to access control, fail the operation and log the 9548 event.

In the 3^rd case, the vast majority of the 9548 events came from the first part - that is, almost all 9548 events are due to disabled user accounts which are missing msExchMasterAccountSid. The new logic works as follows:

2. If the user account is enabled, or if it is disabled and has msExchMasterAccountSid set to the well-known SELF SID, OR if the user is disabled has does not have msExchMasterAccountSid, then use objectSid. (Big change here)

3. If we cannot tell if the user account is enabled or disabled due to access control on the user object, or if we cannot read the objectSid due to access control, fail the operation and log the 9548 event.

So now, the only way you will get the 9548 event is due to a real problem with the user account associated with a mailbox.

- Alex Seigler

Exchange 2003 post SP2 DSProxy Referral Update

Exchange — Fri, 17 Mar 2006 19:09:00 GMT

In November 2005 I posted an article on the DSProxy referral changes that are included in Exchange Server 2003 Service Pack 2. For the full article, please see http://blogs.technet.com/exchange/archive/2005/11/04/413669.aspx.

To refresh everyone's mind, prior to Exchange 2003 Service Pack 2, a DSProxy Referral would provide the Outlook client with a global catalog server from one of two locations:

1. The global catalog is located within the same AD site as the Exchange server (normal behavior).

2. The global catalog is located within an AD site that is directly connected to the Exchange server's AD site (when all in-site GCs are unavailable).

In addition to honoring site membership, DSProxy prefers to use global catalog servers that are members of the same domain as the Exchange server. If none are available, then DSProxy will utilize the other global catalog servers in the AD site.

This poses a few issues in multi-domain environments where DomainPrep has been executed against the domains. Specifically, if an Outlook client is referred to a global catalog server that does not reside in the same domain as the mailbox-enabled user, then that user will be accessing data that is in a read-only fashion. This means that updates to certain objects (whether it be updates to the mailbox-enabled user like delegate access, or distribution group membership) may fail.

In Exchange 2003 SP2, the referral process will now attempt to provide the Outlook client with a GC that belongs to the same domain as the mailbox-enabled user using a new algorithm. The new algorithm will solve the delegation issue, provided a global catalog server exists and is reachable by the Exchange server for the mail recipient. It may address the distribution group issue if the distribution group resides in the same domain as the mailbox (otherwise it won't because the GC only contains a read-only copy of the distribution group).

Recall that with SP2, we score the desirability of a global catalog based several constraints:

1. A GC that is available (RPC ping) - 16 points

2. A GC that supports the Clients protocol - 8 points

3. A GC belonging to the users domain - 4 points

4. A GC within the same AD site as the Exchange Server - 2 points

5. A GC that is one of the GCs that the Exchange Server is currently using - 1 point

With this model, a client could be referred a global catalog that does not belong in the AD Site of the Exchange server due to the global catalog's domain membership matching that of the mailbox-enabled user.

The new referral mechanism may not be advantageous if:

You deployed distribution groups in a centralized domain and placed GCs of that domain in the Exchange Server's AD site to allow for membership updates to occur through Outlook.
You did not deploy mailbox home-domain GCs in the Exchange Server's AD site.

Post-SP2 DSProxy Referral Mechanism Update

The good news is that the DSProxy Referral behavior can be altered to so that you can simulate Pre-SP2 behavior. In order to do that you must install KB 912584, which provides a Post-SP2 version of DSProxy.dll. This fix allows for the use of the following registry key:

Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\MSExchangeSA\Parameters

REG_DWORD Value: RFR Prefer In-Site GCs

Data: 0 (Disabled, default behavior)

Data: 1 (Enabled)

With this registry key set to a value of 1, the constraint cost will change and utilize the following model:

1. A GC that is available (RPC ping) - 16 points

2. A GC that supports the Clients protocol - 8 points

3. A GC belonging to the users domain - 1 point

4. A GC within the same AD site as the Exchange Server - 2 points

5. A GC that is one of the GCs that the Exchange Server is currently using - 4 points

So if we go back to the SP2 example I provided in the original blog post, enabling this registry key will alter the way in which the DSProxy service assigns points to the global catalog servers:

Server	Active Directory Site	In-Use by DSAccess?	Constraint					Total Points
Server	Active Directory Site	In-Use by DSAccess?	1	2	3	4	5	Total Points
ExDomain GC1	Exchange AD Site	Yes	16	8	0	2	4	16+8+2+4=30
ExDomain GC2	Exchange AD Site	Yes	16	8	0	2	4	16+8+2+4=30
UserDomain GC1	Client AD Site	No	16	8	1	0	0	16+8+1=25
UserDomain GC2	Client AD Site	No	16	8	1	0	0	16+8+1=25
UserDomain GC3	AD Site B	No	16	8	1	0	0	16+8+1=25
UserDomain GC4	AD Site B	No	16	8	1	0	0	16+8+1=25
ExDomain GC4	AD Site A	No	16	8	0	0	0	16+8=24
ExDomain GC5	AD Site B	No	16	8	0	0	0	16+8=24
ExDomain GC6	AD Site B	No	16	8	0	0	0	16+8=24

The net effect of this change is that global catalog servers that are currently being utilized by the Exchange server will be preferred by the DSProxy Referral service.

Please note that even with this registry change, DSProxy can still refer clients to a global catalog server that is in use by the Exchange server and that is a member of the same domain as the mailbox-enabled user due to the constraint cost assignment. Consider the following scenario:

In this scenario, the global catalog servers will be assigned the following points by the DSProxy service (assuming the registry key is enabled):

Server	Active Directory Site	In-Use by DSAccess?	Constraint					Total Points
Server	Active Directory Site	In-Use by DSAccess?	1	2	3	4	5	Total Points
UserDomain GC1	Exchange AD Site	Yes	16	8	1	2	4	16+8+1+2+4=31
ThirdDomain GC1	Exchange AD Site	Yes	16	8	0	2	4	16+8+2+4=30
ThirdDomain GC2	Exchange AD Site	Yes	16	8	0	2	4	16+8+2+4=30
UserDomain GC2	Client AD Site	No	16	8	1	0	0	16+8+1=25
UserDomain GC3	Client AD Site	No	16	8	1	0	0	16+8+1=25

In this particular example, Exchange will provide the UserDomain GC1 server that is in-use by Exchange to the Outlook clients due to that global catalog server having a higher point calculation than the ThirdDomain GC1 or GC2 servers. If UserDomain GC1 were to become unavailable, DSProxy would utilize the ThirdDomain GC1 and GC2 servers before referring clients out of the Exchange AD site.

So to summarize, enabling the "RFR Prefer In-Site GCs" registry key:

Prioritizes GCs that are being used by Exchange over home domain membership.
GCs that are both being utilized and are members of the same domain as the mailbox-enabled user will be preferred over non-home domain GCs.
DSProxy will only refer clients out of site if all GCs within the Exchange server's AD site are unavailable. Constraint assignment will determine which GCs are utilized by the DSProxy Referral Process.

Please contact Microsoft Support to obtain fix KB 912584 if you would like to deploy the updated DSProxy Referral mechanism (the article itself is still being written).

- Ross Smith IV

How to choose server hardware for Exchange 2003 that can be effectively re-utilized for Exchange 12?

Exchange — Mon, 13 Mar 2006 20:05:00 GMT

Introduction

When selecting hardware for your Exchange servers, there are many things that you must consider. Two of the most critical resources are processor and memory.

This blog provides rough guidelines for processor/memory configurations that provide good performance for Exchange 2003 while also providing a strong platform for Exchange Server 12. Recommendations will also be made on how server hardware deployed for Exchange 2003 roles can be re-utilized for Exchange Server 12 on a per server role basis (e.g. Mailbox, Client Access Server (CAS) etc). With this in mind, I have tried to make safe/highly confident predictions concerning what processor/memory configurations work well for E12 server roles to help administrators maintain current Exchange 2003 environments with an eye toward the future.

***This information is provided with the caveat that Exchange 12 is still under development and processor/memory metrics may change. This information will be updated as "Best Practices" are defined for Exchange 12.***

Why are Exchange 12 server hardware requirements different from previous versions of Exchange (2003)?

The primary hardware difference between Exchange 2003 and Exchange 12 is the move from a 32-bit platform (Exchange 2003) to a 64-bit platform (Exchange 12). Exchange 12 will only be supported in production environments when it is running on an x64 edition of Windows Server 2003.

The change from a 32-bit platform to a 64-bit platform requires a new approach to choosing server hardware for Exchange; especially processor and memory:

What processor types should I consider to ensure my server hardware will work with both Exchange 2003 (32-bit) and Exchange 12 (64-bit)?

We recommend that you choose a processor that will work with both 32-bit and x64 versions of Windows 2003. The following server processors support both 32-bit and x64 versions of Windows 2003, thereby supporting Exchange 2003 deployments today, and Exchange 12 deployments tomorrow:

AMD Opteron

Intel Xeon with EM64 Technology

Each of these vendors also ship x64-capable desktop processors which can also run x64 versions of Windows 2003 (e.g. AMD Athlon64 and Intel Pentium D with EM64T) but for the sake of simplicity, this article will concentrate on processors designed for server deployments.

It's important to note that the Intel Itanium (IA64) processor will not work with Windows 2003 x64 Editions, and thus it will not work for Exchange 12 deployments. Exchange 12 is designed to run only on x64 capable processors such as those listed above; Exchange 12 will not run on Itanium based systems.

Regardless of which server processor you select, it is necessary to have the server product pass the Designed for Windows test suite to ensure Microsoft support. Servers listed on the Windows Server Catalog meet this criteria. If your server is not listed, check with your vendor to see if either the "Designed for Windows" logo testing is in progress or the server has passed the testing and is pending a website update.

The Technet X64 Newsgroup is a good place to get more information about moving from 32-bit Windows to 64-bit Windows.

Should I be considering multi-core processors for my Exchange 2003/E12 servers?

The short answer is yes. Extensive testing on dual-core processors has shown that Exchange benefits from dual-core processor technology. The performance benefit for Exchange from dual-core technology depends upon the specific processor utilized. The findings from Exchange 2003 dual-core testing have been summarized in Microsoft Knowledge Base article 827281, CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server. Also, the performance benefit of specific dual-core implementations can be seen by comparing the MMB3 results of a 4-processor, single-core based server to a 2-processor, dual-core based server. These results have been published at the Performance Benchmarks for Computers Running Exchange Server 2003 Web site.

Today, dual-core processors are an attractive option for Exchange 2003 and Exchange 12 servers based on price and performance. Please ask your server vendor about dual-core benefits for Exchange, specific to a given hardware architecture.

What do I have to know to ensure my x64 capable Exchange 2003 server can be upgraded with more memory when I deploy Exchange 12?

Exchange 12 enables much better memory utilization than Exchange 2003 due to its 64-bit architecture. Because of the virtual address space limitations of a 32-bit platform, Exchange 2003 is limited to using 4 GB or less of physical memory. In contrast, Exchange 12 running on Windows 2003 x64 Editions will efficiently utilize upwards of 16 GB of memory and beyond (Mailbox role). This change needs to be factored in when putting together server hardware for Exchange 2003 that can be migrated to Exchange 12 server roles. The following factors should be considered:

Server Maximum Memory Configuration - Different server architectures have different memory limits. We recommend that you check the following technical specifications of the server to determine the criteria that affect the maximum memory configuration to ensure that memory upgrades from Exchange 2003 to Exchange 12 can be accommodated economically:

· Memory Speed - Some server architectures require slower memory to scale up the memory to ten's of gigabytes in a given server (e.g., maximum server memory is limited to 16GB with PC3200 or 32GB using PC2700). You should check with the manufacturer to make sure that the memory configuration target for both Exchange 2003 and Exchange 12 are compatible in terms of speed.

· Memory Module Size - What is the largest memory module size the server will support? Generally, the larger the memory module, the more expensive it is; 2x1GB DDR SDRAM memory modules generally cost much less than 1x2GB DDR SDRAM memory modules. When planning for an Exchange 2003 server, make sure the maximum memory module size allows you to meet your target memory requirements for Exchange 12. It may make sense to spend more money and purchase denser memory modules for an Exchange 2003 deployment to ensure that the memory requirements for Exchange 12 can be met down the road.

· Total Number of Memory Slots - How many memory modules will a given server support? The total number of slots multiplied by the maximum memory module size will provide the maximum memory configuration for the server. Also, keep in mind that memory modules must sometimes be installed in pairs.

Memory upgrade path from Exchange 2003 to Exchange 12 - Once you understand the memory requirements of a given server you can design the optimal memory configurations for Exchange 2003 while simultaneously designing an upgrade path to Exchange 12.

Example - In this example, let's say you wanted to purchase a server for Exchange 2003 (Backend/Mailbox role). The Exchange 2003 best practice for maximum memory is 4GB, but you want to ensure you will have the capability to go to 16GB when you deploy Exchange 12. You analyze the manufacturer technical specs and find the following:

Memory Speed: Specifications say 16GB is possible with PC3200 but 32GB requires PC2700.
Memory Module Size: Specifications say 4GB is max memory module size.
Total Number of Memory Slots: Specifications say there are 8 total memory slots (8*4GB = 32GB Max memory configuration). Specification also states memory must be added in pairs.

With this information it is determined that 2GB PC2700 or 2GB PC3200 memory modules will provide a good memory configuration for Exchange 2003 (2x2GB). Moreover, the server can be upgraded to 16 GB (8x2GB) to significantly reduce the I/O for an Exchange 12 Mailbox Server down the road. If 1GB memory modules were chosen for the Exchange 2003 configuration, the server could not be upgraded to 16GB without having to throw out the 1GB memory modules and replace them with 2GB memory modules. One caveat with this planning method is that some servers experience a performance improvement when more memory slots are filled, while others experience a reduction in performance. Check with your hardware vendor to understand this effect for a given server architecture.

How do I apply the processor and memory configuration factors to specific Exchange 2003/Exchange 12 server roles?

The following chart can be used to assist in purchasing server hardware destined to be used for both Exchange 2003 and Exchange 12 server roles. The goal of this chart is to provide an upper bound of viable processor/memory configurations for Exchange Server 12. This is not a statement of "Best Practice" but a simple guide for purchasing server hardware for Exchange Server 2003 that can be utilized for Exchange Server 12. The "sweet spot" in terms of hardware price/performance for each Exchange Server 12 server role is still being ascertained.

Recommended Maximum Processor and Memory Configuration for Exchange

Exchange 2003			Exchange 12
Role	Max Processor Config	Max Memory Config	Role	Max Processor Config	Max Memory Config
Gateway/ Bridghead	2xDual-core	2GB	Edge Transport/ Hub Transport	2xDual-core	8GB
Front End Server (FE)	2xDual-core	2GB	Client Access Server/ Unified Messaging Server	2xDual-core	8GB
Backend Mailbox Server (BE)/ Public Folder Server	4xDual-core	4GB	Mailbox Server	4xDual-core	32GB

Example In choosing a server platform for an Exchange Server 2003 Mailbox/BE that is destined to be re-utilized as an Exchange Server 12 Mailbox server, then it would make sense to choose a server that has a maximum physical memory capacity of 32GB for a scaled up deployment (~4000 mailboxes/server). It would be unfortunate to choose a server that had a maximum physical memory capacity of 8GB for this role and then not be able to effectively reduce I/O with 16GB or 32GB of memory when transitioned to Exchange Server 12 service. The maximum configurations outlined above for Exchange 12 are not "optimal" configurations, but the current maximums the Exchange Product Group is developing against.

Summary

I hope I have offered some insight on how to choose server hardware for Exchange 2003 that can be utilized for Exchange 12 deployments tomorrow. With effective planning and an understanding of the basic processor and memory requirements of both versions, the investments made today in current Exchange 2003 infrastructures can continue to be leveraged with Exchange 12.

- Matt Gossage

msExchESEParamLogBuffers Storage Group Attribute

Exchange — Fri, 10 Mar 2006 19:49:00 GMT

The msExchESEParamLogBuffers storage group attribute governs the number of Extensible Storage Engine (ESE) log buffers that are used by the Exchange information store. ESE uses a set of log buffers to hold information in RAM before it writes to the transaction logs. As you already know we have two sets of physical data in Exchange (physical data being physical files):

1. Transaction log files

2. Database Files (forget the .edb and .stm at the moment, just think of them as a single entity)

The exchange db file is made up of 4k pages structured across a given number of binary trees. Data is changed in the file by modifications to existing pages, while new data is added to the file through the creation of additional pages. Deletion follows the same process. From an operations point of view working with data through a physical file is not performance friendly. IO operations that have to reach physical disk (array cache) are costly performance wise for any application. Therefore the ESE engine (exchange's db storage engine) works with db page creations, modifications and deletions in volatile memory on the host (aka RAM). The db has a cache reserve in memory which is governed by the msExchESEParamCacheSizeMax attribute. Db pages are read from our physical file (disk/array cache) and placed into the reserve. This reserve is then used to work with pages - thus providing data modification performance. Changed pages (those that have been modified in some manor) are known as 'dirty-pages' whilst their unchanged cousins are referred to as 'clean-pages'. Dirty pages are eventually flushed from the db cache to disk - this happens during a variety of circumstances, that we don't need to go into here - just be aware that our dirty pages are eventually de-staged to physical file.

So, that's fine - that is the db - but where do the transaction logs come into it and especially the msExchESEParamLogBuffers attribute?

Exchange utilizes the Extensible Storage Engine (ESE),which is a transactional based database model that utilizes ACID. This is:

- Atomic: This term indicates that a transaction state change is all or none. Atomic state changes include database changes, and messages and actions on transducers.

- Consistent: This term indicates that a transaction is a correct transformation of the state. The actions taken as a group do not violate any one of the integrity constraints associated with the state. This requires that the transaction be a correct program.

- Isolated: This term indicates that even though transactions execute concurrently, it appears to each transaction (T) that others executed either before T or after T, but not both.

- Durable: This term indicates that once a transaction completes successfully (commits), its changes the state to survive failures.

Each of these 4 components are delivered by our transactional logging capability. Every change to a db page is considered a transaction, even though a complete transaction could (and 99.9% of the time does) involve the modification of more than a single page. The msExchESEParamLogBuffers attribute is exactly the same msExchESEParamCacheSizeMax attribute but it is for the transaction logs. As we change db pages to a dirty state in the db cache, we record the process (known as a transaction) to the physical log file. Just as working with data on physical disk for the db causes performance degradation, so it does for the transaction logs. Therefore we have a volatile memory reserve for our log operations too. The reserve size is governed by the msExchESEParamLogBuffers attribute.

Exchange's ability to recover from a crash comes in the form of a durable commit. A durable commit says once the transaction is logged into the msExchESEParamLogBuffers memory reserve, we must flush this to physical disk straight-way - prior to its corresponding dirty db pages being de-staged to the physical db file. This ensures we keep consistency for recoverability and the db never contains an incomplete transaction (aka corruption). The larger the msExchESEParamLogBuffers memory reserve, the greater the number of transactions that can be simultaneously worked on and the also the larger individual transactions can be.

So, that's what the msExchESEParamLogBuffers attribute is there for, increasing the size can provide better performance for multiple transaction recording operations. This is particularly beneficial to busy corporate Exchange severs. If the msExchESEParamLogBuffers becomes full we start to see log operation stalls, which directly affect the clients performance when changing data - that's why we say WRITE performance for transaction log volumes is critical - transaction records do not get flushed from memory until they have been through the durable commit.

The msExchESEParamLogBuffers attribute can also be taken into consideration during storage sub system testing. The objective of Jetstress is to generate an IO profile as much like a production Exchange server as possible. The msExchESEParamLogBuffers attribute is Active Directory (AD) based and therefore doesn't come into play with Jetstress. In a production Exchange environment the msExchESEParamLogBuffers attribute is read by the server and configured value is used for the memory reserve size. Referencing the above explanation in this mail for how transactional logging works, we can see that the msExchESEParamLogBuffers attribute can directly affect our IO write profile for the transaction log volume. For example: if the memory reserve size is small, we should see increased 'smaller' IO operations. The reason for this is because with a smaller buffer ESE is under a greater pressure to free up reserve space, ensuring we do not encounter a log stall scenario. On the other hand; if the memory reserve size is greater, we can expect to see reduced IO operations but of a larger size. The reasoning behind this is because ESE is not under so much pressure to flush every complete transaction immediately and therefore can group a number of complete transactions together for a single 'cost effective' write IO.

Jetstress has always had the capability for log buffer consideration (existing in the CMD line version and in the UI version). Up until recently, the UI version didn't support entering in a value greater than 8192 and by default was configured to use a value of 64. Jetstress now has the capability to adjust this value, and has a default setting of 9,000. Having a larger msExchESEParamLogBuffers size will prove advantageous for write performance to the log volume. The IO handoff is a single threaded, sequential process and we need to wait for a response back prior to issuing the next IO. Therefore, logic says - if we produce less IO, response times will be quicker as we will mitigate any risk of queuing IO operations on the host that are perhaps awaiting handoff to the storage device.

The new default value of 9,000 in Jetstress will provide advantageous to response time and generate less IO. This would be my recommendation. Providing the production Exchange Sever is configured in the same manor, the setting will produce the nearest profile to Exchange running under normal operations. However, there is a complication as always. Jetstress uses very small updates (replace of a single record). So, if you use 100% durable commits for its options then the size of the log buffer almost doesn't matter because we are always flushing it to commit the transactions to disk ASAP. If you use a mix of durable and lazy commits then this will take advantage of the larger buffer and will also more closely simulate the log behaviour of Exchange. Exchange has a lot of lazy commit transactions in a chain that are then followed up by a single durable commit to flush them all to disk. You should just view this as a larger single transaction because this is just an artefact of how Exchange is architected internally.

Frequently Asked Questions

[Question] Before tweaking log buffers should I check that the disks are behaving properly? i.e. If the disks are slow, will tweaking JET buffers really solve the root cause of the problem.

[Answer] If you are experiencing 'bigger' issues with the storage solution modifying the log buffer value will not make any difference. Although, one thing I would say is that this attribute is only a controlling factor for log IO. A value of 9,000 will purely provide a REAL simulation of Exchange IO. Remember the latest build of Jetstress to uses the 9,000 figure by default. So, whether you have 'poor' performing disks already, you should try to simulate Exchange IO as closely as possible - at least to base line your performance testing. Once you have a base line you can attempt to improve the performance of you storage solution following best practice guidelines from your chosen storage vendor and Microsoft.

[Question] When should we change the msExchESEParamLogBuffers attribute? when average Database\Log Record Stalls/sec exceeds 10ms, and spikes above 100ms?

[Answer] If an Exchange server is seeing log stalling behaviour then increasing the ESE log buffer should offer help. By increasing the value, and thus the memory reserve size we increase our ability to handle more transactions, thus not stalling them causing thread waiting behaviour. The Exchange performance whitepaper details the correct figures based on product team matrix testing. Another good reason for the increase (and why it should be done for a configuration step as part of a server install) is the larger the memory reserve size, the larger the transaction we can handle in single simultaneous process.

[Question] Will changing msExchESEParamLogBuffers to 9,000 increase or decrease the log disk write latency?

[Answer] Changing the log buffer value will change the log IO profile. Therefore, the answer is yes - it may well change the write latency figures. However, what I would say is this doesn't really matter a great deal. As I mention above the default value is now 9,000 - this is a 'recommended' performance tuning step which offers the best working environment for Exchange. Therefore we should work with this figure and perform other steps to ensure our log write latency is acceptable.

[Question] Does the recommendation of changing msExchESEParamLogBuffers to 9,000 apply to both E2K and E2K3?

[Answer] See the following article for the correct Exchange 2000 and Exchange 2003 configuration:

http://www.microsoft.com/technet/prodtechnol/exchange/Analyzer/eb989e36-5f76-49d8-9075-c2f6dd07d37b.mspx

- Wil Westwick

More on Exchange ActiveSync Reporting with Log Parser - COM object available

Exchange — Fri, 03 Mar 2006 21:10:00 GMT

We recently posted details on how to use Log Parser to parse out Exchange ActiveSync (EAS) information to produce helpful reports that can give you a better idea of the usage load of your EAS deployment. In that blog we relied solely on LogParser using the IISW3C input format and your IIS Logs.

In this blog post, we take the capabilities of Log Parser one step further by leveraging the pluggable COM object interface. We have put together an EAS specific Log Parser COM object to help make building queries much easier. You will find this Log Parser COM object very useful in further drilling into sync activity. For example you can create a report that showed the number of actual Calendar items Added, Updated or Deleted from the device of a specific user. The same information can be reported for both Email and Contacts as well. While you could get this without the use of the EAS Log Parser COM Object, it is not a trivial task and involves string parsing contortions that would make your eyes blurry and also take a long time for results to be generated on a typical production IIS log dataset.

For our examples we will show you how to write relatively straight-forward SQL statements that use the EAS Log Parser COM Object. The first sample is a user report that will return a few of the fields exposed via the COM Object. The second sample is a lot simpler, but still relatively important… we will produce a server bandwidth report which can show you how many bytes are being sent and received from your EAS Server. For both of these examples we chose to output the results to an HTML formatted report.

NOTES:

- The COM Object is compatible with all versions of Exchange with ActiveSync built-in... Exchange 2003 or higher (E12).

- This reporting solution, SQL scripts and COM Object are not officially supported by Microsoft.

- Your IIS machine should be setup to produce "W3C Extended Log File Format" logs. IIS sets this format type by default format, but if you changed this format type please reset it in order for the SQL queries to find the correct field names. You can change this on the properties of your website via IIS Mgr.

Getting Started

To install Log Parser go to: http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

Log Parser Basics

By default, log parser will be installed in C:\Program Files\Log Parser 2.2. Run the LogParser.exe from the Log Parser install directory. Examples of the full Log Parser syntax are included in the last section of this blog - "Running Log Parser with the Supplied SQL Scripts".

Log File Location

With the EAS Log Parser COM Object, you can specify the location of the logs via:

- UNC

- Mapped Drives

- Or Locally

NOTE: Log Parser (without the help of the EAS Log Parser COM Object) sometimes has problems opening large amounts of log files through a mapped drive, so it is not a recommended approach.

Installing the EAS Log Parser COM Object

Please copy the EAS Log Parser COM Object to your local machine and register the dll. Download the file from HERE.

Example on how to register it:

Regsvr32 c:\Program Files\Log Parser 2.2\lpeas.dll

SQL Queries

User Hits Report

Create a file and name it EAS_User_Statistics.SQL and place this code block into the newly created file.

SELECT TOP 50

eas-user-alias,

eas-user-domain,

eas-device-type,

SUM (TotalClientEmailSync) AS TotalClientEmailSyncs,

SUM (TotalServerEmailSync) AS TotalServerEmailSyncs,

ADD (TotalClientEmailSyncs, TotalServerEmailSyncs) AS TotalEmailSyncs,

SUM (TotalClientContactSync) AS TotalClientContactSyncs,

SUM (TotalServerContactSync) AS TotalServerContactSyncs,

ADD (TotalClientContactSyncs, TotalServerContactSyncs) AS TotalContactSyncs,

SUM (TotalClientCalendarSync) AS TotalClientCalendarSyncs,

SUM (TotalServerCalendarSync) AS TotalServerCalendarSyncs,

ADD (TotalClientCalendarSyncs, TotalServerCalendarSyncs) AS TotalCalendarSyncs,

ADD (TotalEmailSyncs, ADD (TotalContactSyncs, TotalCalendarSyncs)) AS TotalSyncs

USING

CASE eas-folder-data-type

WHEN 'Em' THEN

ADD(eas-client-adds,

ADD (eas-client-changes,

ADD (eas-client-deletes, eas-client-fetches)))

ELSE 0

END AS TotalClientEmailSync,

CASE eas-folder-data-type

WHEN 'Em' THEN

ADD(eas-server-adds,

ADD (eas-server-changes, eas-server-deletes))

ELSE 0

END AS TotalServerEmailSync,

CASE eas-folder-data-type

WHEN 'Co' THEN

ADD(eas-client-adds,

ADD (eas-client-changes,

ADD (eas-client-deletes, eas-client-fetches)))

ELSE 0

END AS TotalClientContactSync,

CASE eas-folder-data-type

WHEN 'Co' THEN

ADD(eas-server-adds,

ADD (eas-server-changes, eas-server-deletes))

ELSE 0

END AS TotalServerContactSync,

CASE eas-folder-data-type

WHEN 'Ca' THEN

ADD(eas-client-adds,

ADD (eas-client-changes,

ADD (eas-client-deletes, eas-client-fetches)))

ELSE 0

END AS TotalClientCalendarSync,

CASE eas-folder-data-type

WHEN 'Ca' THEN

ADD(eas-server-adds,

ADD (eas-server-changes, eas-server-deletes))

ELSE 0

END AS TotalServerCalendarSync

INTO C:\Public\LogParserOutput\EAS_User_Statistics.html

/* Substitute this directory location with the location of your logs. Wildcards are allowed.

FROM C:\Public\Logs\*.log

GROUP BY

eas-user-alias,

eas-user-domain,

eas-device-type

ORDER BY

TotalSyncs DESC

What's going on in this SQL statement?

This SQL statement calculates totals from several salient EAS activities (Emails, Calendar, Contact Sync'd, etc) for a specific user. It is similar to the example that we provided in our external blog for EAS Reporting, but this time we show you how much cleaner it is without having to parse the cs-uri-query field. As you can see in the sample above each data point (eas-cmd, eas-device-type, eas-client-adds, etc.) is now available as its own field, for example if you wanted to see the items created and sent up to the EAS Server you could query the 'eas-client-adds' field. Taking it a step further, you can get the number of emails created from the device by first looking in the 'eas-folder-data-type' field for the value 'Em' which means email. Once you find the record with 'Em' from the 'eas-folder-data-type' field, you then can SUM the number of 'eas-client-adds'. For our sample, we took it one step further and calculated all client activity for each eas-folder-data-type (we did the same for server activity). We calculated the total by using the ADD function and added each client related sync field together. Since the ADD function only supports adding two arguments together we needed to string together several ADD functions to get the total.

Another interesting feature that this script leverages in Log Parser is the ability to output to an HTML file and being able to customize the HTML output. To customize the HTML, you create a template file that has the extension '.tpl'. In the 'Running Log Parser with the Supplied SQL Scripts' section below, we will show you what cmd-line parameters you need to pass in to get this to work.

Dynamically Created HTML output

To create the template file create another file and name it EAS_User_Statistics.tpl and copy the contents into the file. This file will be used by Log Parser to dynamically create an HTML formatted file output.

<HTML>

<HEAD><TITLE>Summary of Exchange ActiveSync User Statistics</TITLE></HEAD>

<TR>

<TH COLSPAN="13" ALIGN="CENTER"><font size="4">Exchange ActiveSync User Statistics

</font> </TH>

</TR>

<TR>

<TH>UserName</TH>

<TH>Domain</TH>

<TH>Device Type</TH>

<TH>Total Syncs</TH>

<TH>Total Client Email Syncs</TH>

<TH>Total Server Email Syncs</TH>

<TH>Total Email Syncs</TH>

<TH>Total Client Contact Syncs</TH>

<TH>Total Server Contact Syncs</TH>

<TH>Total Contact Syncs</TH>

<TH>Total Client Calendar Syncs</TH>

<TH>Total Server Calendar Syncs</TH>

<TH>Total Calendar Syncs</TH>

</TR>

</LPHEADER>

<TR>

<TD>%eas-user-alias%</TD>

<TD>%eas-user-domain%</TD>

<TD>%eas-device-type%</TD>

<TD align="center"><font color="#FF0000">%TotalSyncs%</font></TD>

<TD align="center">%TotalClientEmailSyncs%</TD>

<TD align="center">%TotalServerEmailSyncs%</TD>

<TD align="center">%TotalEmailSyncs%</TD>

<TD align="center">%TotalClientContactSyncs%</TD>

<TD align="center">%TotalServerContactSyncs%</TD>

<TD align="center">%TotalContactSyncs%</TD>

<TD align="center">%TotalClientCalendarSyncs%</TD>

<TD align="center">%TotalServerCalendarSyncs%</TD>

<TD align="center">%TotalCalendarSyncs%</TD>

</TR>

</LPBODY>

</TABLE>

</BODY>

</HTML>

</LPFOOTER>

EAS Server Bandwidth Report

For the next report, create a file and name it EAS_Server_Bandwidth.SQL and place this code block into the newly created file.

SELECT

s-ip AS ServerIP,

COUNT (eas-cmd) AS TotalHits,

SUM (sc-bytes) AS TotalBytesSent,

SUM (cs-bytes) AS TotalBytesReceived

INTO C:\Public\LogParserOutput\E12_Server_Usage.html

FROM C:\Public\Logs\ex05120714.01.log

GROUP BY

s-ip

ORDER BY

TotalHits DESC

What's going on in this SQL statement?

This SQL statement doesn't quite show the real benefit of the EAS Log Parser COM Object, because this same query could be easily created by simply getting the COUNT(*) instead of using COUNT(eas-cmd), but we wanted to share this report because we've heard from you that you want visibility into the amount of bandwidth flowing into and out of your EAS Servers. To accomplish this, we are getting the bandwidth data directly from the sc-bytes and cs-bytes fields, which we have confirmed through analysis that this information also factors in GZip compression. Similar to the first sample, this sample also creates an HTML formatted output file.

Dynamically Created HTML output

To create the template file create another file and name it EAS_Server_Usage.tpl and copy the contents into the file. This file will be used by Log Parser to dynamically create an HTML formatted file output.

<HTML>

<HEAD><TITLE>Summary of Exchange ActiveSync Server Usage</TITLE></HEAD>

<TR>

<TH COLSPAN="4" ALIGN="CENTER"><font size="2">Summary of Exchange ActiveSync Server Usage</font></TH>

</TR>

<TR>

<TH>Server IP</TH>

<TH>Total Hits</TH>

<TH>Total Bytes Received</TH>

<TH>Total Bytes Sent</TH>

</TR>

</LPHEADER>

<TR>

<TD>%ServerIP%</TD>

<TD align="center"><font color="#FF0000">%TotalHits%</font></TD>

<TD align="center">%TotalBytesReceived%</TD>

<TD align="center">%TotalBytesSent%</TD>

</TR>

</LPBODY>

</TABLE>

</BODY>

</HTML>

</LPFOOTER>

Running Log Parser with the Supplied SQL Scripts

There are several input parameters that will allow you to make log parser aware of this COM Object and to output to html. They include:

COM Object

- -i:COM

- -iProgID:MSUtil.LogQuery.EASInputFormat.

HTML Output

- -o:TPL

- -tpl:<<LOCATION TO TPL FILE>>

Here is the command syntax...

LogParser.exe -i:COM -iProgID:MSUtil.LogQuery.EASInputFormat file:<<LOCATION TO SQL FILE>>\EAS_Server_Bandwidth.sql -o:TPL -tpl:<<LOCATION TO TPL FILE>>\EAS_Server_Usage.tpl

EAS User Statistics EXAMPLE:

LogParser.exe -i:COM -iProgID:MSUtil.LogQuery.EASInputFormat file:c:\Public\LogParser\SQL\EAS_User_Statistics.sql -o:TPL -tpl:c:\Public\LogParser\SQL\EAS_User_Statistics.tpl

EAS User Statistics EXAMPLE:

LogParser.exe -i:COM -iProgID:MSUtil.LogQuery.EASInputFormat file:c:\Public\LogParser\SQL\EAS_Server_Bandwidth.sql -o:TPL -tpl:c:\Public\LogParser\SQL\EAS_Server_Usage.tpl

Log Parser Help

The Log Parser Help is included as part of the Log Parser 2.2 install. We have found this Help file to be very helpful in getting started. The Reference section is especially informative.

Here are some sections that were helpful...

Reference\

- Command-Line Operation\Query Execution Mode

- Query Syntax\*

- Input Formats\*

- Output Formats\*

We hope you find these two scripts helpful in getting started with your EAS reporting with the help of the EAS Log Parser COM Object.

- Michael Higashi and Patrick Tousignant

Exchange 2003 Auto Accept Agent vs. direct booking

Exchange — Wed, 22 Feb 2006 23:29:00 GMT

In my Exchange support experience, customers often ask me about the differences between using the Auto Accept Agent and Outlook Direct Booking. They want to know which they should use for their specific environment. After much research, I compiled the following matrix of advantages and disadvantages of each to help you decide for yourself which you should use.

The Auto Accept Agent is a server-side store event sink available in Exchange 2003 SP1 which provides automatic server-side processing of meeting requests sent to resource mailboxes. The Agent handles initial scheduling, cancellations and updates and sends responses to the meeting organizer - meaning less scheduling conflicts and allowing receptionists and conference room owners to spend their time more productively.

What are the advantages of using Auto Accept Agent?

The agent runs on the server 24 hours per day, 7 days a week. The agent responds immediately and the requestor does not have to wait for the receptionist to return from lunch.
The agent checks the availability by looking at the calendar (resource schedule), not the Free/Busy information. This reduces the number of issues caused by free/busy update lag.
It can decline individual conflicting instances within a recurring meeting request.
In case of declined meeting request with conflicts (or a meeting request accepted with conflicts) the response includes date/time information on which instances conflict.
All comments and attachments are removed from the meeting requests. Only the time and date information will remain on the meeting item. This prevents the delegate from viewing and confidential or private information. It also keeps the mailbox trim.
Meetings marked as private are processed immediately. Additionally, the private flag is stripped from meeting requests so the receptionist will always have clear visibility to what is on the calendar.
It replaces the need for excessive permissions needed to allow direct booking and does not require delegate permissions or configuration
Much more robust than Outlook "auto-booking".
Reminders are removed from the conference room's copy of the meeting. This avoids "reminder hell" in the event the conference room mailbox is directly opened in Outlook.
It handles meeting requests from all clients including OWA.
The resource mailbox owner retains their rights on the resource mailbox and will remain the ultimate authority on the mailbox.

Does Auto Accept Agent have any disadvantages compared to direct booking?

The agent code must run on every Exchange server that has a resource mailbox, since CDOEX is used to process requests and can only access local mailboxes.
Delegates (e.g. Receptionists) will be less aware of the number and nature (urgency) of incoming requests.
Currently the booking window will be limited to 6 months. This can be changed but it's not recommended. It will take a while for requestors to become accustomed to this hard limit. Important: Any request beyond the six-month window, including recurring meetings extending past it, will automatically be declined.
Delegates will have to check the Inbox, Sent Items, and Deleted Items for unexpected items, but can do so more infrequently.

With that said, I would also like to talk about direct booking and its advantages and some considerations while using it.

Why direct booking?

Direct booking is an Outlook-specific feature that uses the organizer's Outlook client (Outlook 2000 or later) to book an appointment directly into a resource mailbox schedule. The Outlook client of the person organizing the meeting performs all the necessary tasks, such as conflict checking and placing the reservation on the resource calendar. To set up direct booking, follow:

291616 How to direct book a resource in Outlook 2002 or in Outlook 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;291616

Advantages of Direct booking:

Immediate confirmation/denial of booking request.
It has the logic to avoid double-booking.

With these advantages in mind, some disadvantages of Direct booking include:

Direct booking requires granting organizer read/write permissions on the calendar. Hence, users can just go onto the resource's calendar and delete existing meetings to free up space for their own meetings.
Organizer must remember to designate resource as "Resource" and not "Required" or "Optional". No conflict checking is done if resource specified as one of the latter (rule on resource's inbox should be created to generate reply to this effect).
Since there is availability against free/busy, latency in free/busy replication can result in double-booking.
Conflicts aren't detected beyond the range of published free/busy and can result in double-booking.
Outlook Web Access does not currently support the "direct booking" of meeting requests.

Direct booking should not be used in conjunction with automatic booking done by the Auto Accept Agent. There is also configuration associated with direct booking that could conflict with Auto Accept Agent settings. For example, the default setting for direct booking is to accept all requests, regardless of conflicts. There is also a direct booking setting that automatically declines all recurring requests. These settings are only honored by Outlook and only when the resource is specified as a resource attendee. You should not register a resource mailbox for Auto Accept Agent in Exchange 2003 when you set up a resource for direct booking in Outlook. This is discussed in the below article

903290 You should not register a resource mailbox for Auto Accept Agent in Exchange 2003 when you set up a resource for direct booking in Outlook

http://support.microsoft.com/default.aspx?scid=kb;EN-US;903290

Cluster Server Installation Notes

I also get asked if the agent can be installed on a cluster. Installing the Auto Accept Agent on a cluster requires only minor considerations. The installation of the binaries is the same as installing on a standalone server. However, Auto Accept Agent must be installed on each node that may be used for failover. If Auto Accept Agent is not installed on a passive node, meeting requests will be orphaned in the resource mailbox Inbox if this node is made active. Since the agent starts only when needed, there is no need to change the service startup settings. Because the configuration is file-based, it is important to use the same settings across all nodes in the cluster. If Auto Accept Agent is running on multiple active nodes, any per-mailbox settings must be put in all AUTOACCEPT.CONFIG.XML files. This file can be copied to all nodes of a cluster.

Mailbox sink registration must be done using the active node which hosts the mailbox. Although passive nodes must have Auto Accept Agent installed, they are not actually running Exchange services and are incapable of performing the mailbox registration. There is no need to make a node active to install or remove the Auto Accept Agent.

In next blog post: Troubleshooting Auto Accept Agent top support issues.

- Nagesh Mahadev

How to deal with users leaving the Organization

Exchange — Fri, 17 Feb 2006 21:09:00 GMT

A user leaving the organization is something that all our customers have to find a method of dealing with. There is no official guidance on the best way to deal with this because it varies depending on the circumstances that you are in. But, since folks keep asking me about the best way to deal with this issue, I thought I would take the time to put up some recommendations and talk about some common scenarios.

There are two primary questions you have to ask when trying to figure out the best way to do this for your organization. First do you want the mailbox to continue receiving email after the person leaves? Second how long do you want to retain the mailbox? Basically the first one is a Yes, No question and the second can be broken down into either a short time period < 30 days or a long time period > 30 days. What we end up with then is four common scenarios we need to deal with.

With Mailflow, Either Short or Long Retention

If you are going to allow email to continue to flow to the mailbox during the retention period then the time of retention is not relevant to our options. This gets us down to only three scenarios we will need to cover.

In this case since the mailbox will still be receiving email we can just leave it in its current physical location. We will want to disable the account in AD so that the user can no longer use the account/mailbox. Also I would recommend moving these accounts to their own OU structure so that you can keep better track of them.

The only gotcha in this scenario is that you need to properly set the MSExchangeMasterAccountSID value otherwise you will end up with 9548 warnings in your application log and NDRs when you send email to the user you just disabled. The following article tells you how to set this value:

319047 You receive a non-delivery report when you send a message to a disabled account

http://support.microsoft.com/default.aspx?scid=kb;EN-US;319047

You will need to develop your own process to track these users and how long they have before you delete their accounts and set their Mailboxes to purge.

In this scenario you can also configure the account to deliver all email to another mailbox or configure another user to access this mailbox. This will assist in transitioning a user's responsibilities over to a new person or group. The best way to do this is thru AD using one of the following articles:

281926 How to configure a mailbox to forward mail to a mail-enabled contact

http://support.microsoft.com/default.aspx?scid=kb;EN-US;281926

268754 How to Assign Users or Groups Full Access to Other User Mailboxes

http://support.microsoft.com/default.aspx?scid=kb;EN-US;268754

Without Mail Flow, Short Retention

This is actually this most simple situation to deal with. In this case the default mailbox retention on information stores is 30 days so if we simply delete the mailbox thru AD. This will prevent the mailbox from receiving email and the mailbox will purge automatically within 30 days. Once the mailbox has been removed from the account you can disable the account until you are ready to delete it from AD.

If at any point during the 30 days you do decide that you need access to the email you can simply reconnect the mailbox to the previous account or any existing account (without a mailbox) this will allow you access to the old email.

274343 How to Recover a Deleted Mailbox in Exchange

http://support.microsoft.com/default.aspx?scid=kb;EN-US;274343

Users sending email to this mailbox will receive an NDR stating that the user does not exist at the organization.

Without Mail Flow, Long Retention

This case crops up most after a customer has already used the first scenario for 30 days or so. After all you don't want to lose a valuable piece of email because the sender got an NDR that the mailbox did not exist.

The recommendation that I make to customers who need to do this is to dedicate a storage group to the long term retention of these mailboxes. This is done because with long term data retention we don't need fast disk we just need a good bit of disk space. So what to do is setup a storage group and place it on a basic disk array with plenty of storage. I would still place the log files on another volume, maybe one shared with another storage group (assuming we are going to be doing our moves at night).

Now we configure the mailbox retention on this database to the value we desire for our environment. We move the mailbox to be retained to this server and then delete the mailbox from AD. This concentrates all of our mailboxes to be retained in one location where we can back them up and manage them and does not incur their space or backup overhead on the other databases.

Conclusion

Once you break things down the actions to take are quite simple and easy to implement. My only other recommendation with this is to make sure your retention and deletion policies are clearly stated and documented to your internal customers. This will help prevent that phone call of "We need to get mail XYZ from user Bob's mailbox but he left the company 2 months ago, do you still have that?"

- Matthew Byrd

Exchange 2003 - Active Sync reporting

Exchange — Wed, 15 Feb 2006 01:51:00 GMT

We have received a lot of feedback recently about your need to better understand the usage of your Exchange 2003 ActiveSync (EAS) deployment. With all of the great features built into EAS, it’s important for you as an IT Professional to know the volume and usage patterns of your deployment. This information can help you more effectively manage your EAS deployment, better understand user productivity, and help you calculate ROI.

Today, the EAS protocol, logs a lot of data in IIS about what’s going on, but parsing out that data can be a daunting task. We understand your pain, so to help you, we have created two sql scripts which will help you get started in better evaluating your EAS deployment. These scripts work in conjunction with the Microsoft Log Parser 2.2 tool to produce informative reports.

DISCLAIMER: Those scripts are samples only, not officially supported by Microsoft.

The first script produces a ‘Hits by User’ Report which will show you which users are hitting your EAS servers. This report provides many of the salient statistics that could help you better understand how your users are interacting with your EAS deployment. It returns information by individual user such as, devices type used, number of Syncs, Emails Sent, Attachments downloaded, Meeting Responses, etc.

The second script produces is a ‘Hits by DeviceType’ Report. This report gives you a breakdown of the DeviceTypes (SmartPhone, PocketPC, etc.) that are hitting your EAS Server. This report can give you a better idea of what types of devices your users prefer, which can help you make more informed planning decisions in the future. This script also can produce a very helpful chart, such as a 3D pie chart using Microsoft Office Web Components that can be used for reporting or presentations purposes to your management.

Here are the details for running Log Parser with these sql scripts…

NOTE: Your IIS machine should be setup to produce “W3C Extended Log File Format” logs. IIS set this format type by default format, but if you changed this format type please reset it in order for the sql queries to find the correct field names. You can change this on the properties of your website via IIS Mgr.

Getting Started

To install Log Parser go to… http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

To install Microsoft Office Web Components go to… http://www.microsoft.com/downloads/details.aspx?FamilyID=982B0359-0A86-4FB2-A7EE-5F3A499515DD&displaylang=en

Log Parser Basics

By default, log parser will be installed in C:\Program Files\Log Parser 2.2. Run the LogParser.exe from the Log Parser install directory. Examples of the full Log Parser syntax are included in the last section of this blog – “Running Log Parser with the Supplied SQL Scripts”.

Log File Location

You can either specify the UNC location of your Logs, or copy them locally to the machine running Log Parser. Mapped drives do not currently work in Log Parser 2.2.

SQL Queries (samples)

Hits by User Query

Create a file and name it Hits_by_User.sql and place this code block into the newly created file.

SELECT

TO_LOWERCASE (cs-username) AS User,

MyDeviceType AS DeviceType,

COUNT(*) AS Hits,

SUM (MySync) AS Syncs,

SUM (MyFolderSync) AS Folder_Syncs,

SUM (MyPing) AS Pings,

SUM (MySendMail) AS Emails_Sent,

SUM (MySmartReply) AS SmartReplies,

SUM (MyMeetingResponse) AS Meeting_Responses,

SUM (MyGetAttachment) AS Get_Attachments

USING

SUBSTR (cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'DeviceType='), 11),

INDEX_OF (SUBSTR(cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'DeviceType='), 11)), '&')) AS MyDeviceType,

SUBSTR (cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'Cmd='), 4),

INDEX_OF (SUBSTR(cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'Cmd='), 4)), '&')) AS MyCmd,

CASE MyCmd

WHEN 'Sync' THEN 1

ELSE 0

END AS MySync,

CASE MyCmd

WHEN 'Ping' THEN 1

ELSE 0

END AS MyPing,

CASE MyCmd

WHEN 'SendMail' THEN 1

ELSE 0

END AS MySendMail,

CASE MyCmd

WHEN 'SmartReply' THEN 1

ELSE 0

END AS MySmartReply,

CASE MyCmd

WHEN 'MeetingResponse' THEN 1

ELSE 0

END AS MyMeetingResponse,

CASE MyCmd

WHEN 'GetAttachment' THEN 1

ELSE 0

END AS MyGetAttachment,

CASE MyCmd

WHEN 'FolderSync' THEN 1

ELSE 0

END AS MyFolderSync

/* CSV

If you want to output to CSV, you will need to add the cmd-line parameter -o:CSV

EXAMPLE: LogParser.exe -i:IISW3C -o:CSV file:c:\Public\LogParser\SQL\User_Activity.sql

INTO C:\Public\LogParserOutput\Hits_By_User.csv

FROM C:\Public\Logs\*.log

WHERE cs-uri-stem = '/Microsoft-Server-ActiveSync'

GROUP BY User, DeviceType

ORDER BY DeviceType, Hits DESC

What’s going on in this sql statement?

With EAS, most of the salient data that will give you valuable reports are stored in the cs-uri-query field. The EAS protocol logs each HTTP Post request and response, such as sending email, downloading attachments, etc. Here in this example, we first parse the cs-uri-query field to look for the ‘DeviceType=’ section in this field (cs-uri-query) and retrieve the value that is between the ‘DeviceType=’ and the ‘&’ (single quotes excluded). Next we go after the different types of sync commands by using the same mechanism that we used to get the DeviceType. To do this we look for the ‘Cmd=’ section and find the value that follows. The parsed out commands are then stored into the MyCmd variable. In order to display the various sync command stored in the MyCmd variable into individual fields we need to place them into their own variable and SUM up the variable to get the total of each sync command. We use the CASE function to evaluate the incoming data of the MyCmd field. In our example, we returned some of the more interesting commands, but there are more commands to report on if you’re interested. See the below list of 'Sync Commands' which you could add to this sql statement...

Finishing up on the details of this first sql statement we select the output format. In this case we place the results into a CSV file which you can quickly view, or you could import the CSV into a SQL reporting solution. In the next sql query example, we’ll show you how to create a 3D pie chart of the results using Microsoft Office Web Components.

Modifying/Customizing this sql statement

We know that you may have unique reporting requirements and would like to have a report that only returned certain data points. You can use the above sample as a starting point to add or remove fields to have the most appropriate report for your needs. For example, if you wanted to have a report that just showed the top 10 users who have downloaded the most attachments you could modify the query to something like this…

SELECT

TO_LOWERCASE (cs-username) AS User,

SUM (MyGetAttachment) AS Get_Attachments

USING

SUBSTR (cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'Cmd='), 4),

INDEX_OF (SUBSTR(cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'Cmd='), 4)), '&')) AS MyCmd,

CASE MyCmd

WHEN 'GetAttachment' THEN 1

ELSE 0

END AS MyGetAttachment

/* CSV

If you want to output to CSV, you will need to add the cmd-line parameter -o:CSV

EXAMPLE: LogParser.exe -i:IISW3C -o:CSV file:c:\Public\LogParser\SQL\Get_Attachment_Hits_by_User.sql

INTO C:\Public\LogParserOutput\Get_Attachment_Hits_By_User.csv

FROM C:\Public\Logs\*.log

WHERE cs-uri-stem = '/Microsoft-Server-ActiveSync'

GROUP BY User

ORDER BY Get_Attachments DESC

Sync Commands

This list is the comprehensive list of sync commands. Depending on your Exchange Server version and the devices that connect to your server, you may not have these commands return data. For example, the Ping command was recently introduced in the Exchange 2003 SP2 release, so if you have a longer version of Exchange you may not get this information returned back in your report.

Sync

FolderSync

GetItemEstimate

Ping

GetAttachment

SmartReply

SendMail

MeetingResponse

ItemOperations

Notify

MoveItems

Settings

SmartForward

GetHierarchy

Provision

Hits by DeviceType Query

Create a file and name it Hits_by_DeviceType.sql and place this code block into the newly created file.

SELECT MyDeviceType AS DeviceType,

COUNT(*) AS Hits

USING SUBSTR (cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'DeviceType='), 11),

INDEX_OF (SUBSTR(cs-uri-query, ADD (INDEX_OF (cs-uri-query, 'DeviceType='), 11)), '&')) AS MyDeviceType

/* Output Types: 1) CSV, 2) Chart */

/* CSV */

/* INTO C:\Public\LogParserOutput\Hits_By_DeviceType.csv */

/* Chart */

INTO C:\Public\LogParserOutput\Hits_By_DeviceType.gif

FROM C:\Public\Logs\*.log

WHERE cs-uri-stem LIKE '%ActiveSync'

GROUP BY MyDeviceType

What’s going on in this sql statement?

In this example we use the same mechanism to parse into the cs-uri-query field to get the DeviceType value. We then group and count the different Device Types. The interesting thing about this example is that the results fit nicely into a chart, such as a pie chart. Log Parser has hooks built into the Microsoft Office Web Component API which will allow you to produce relatively sophisticated charts through the Log Parser tool. We’ve provided you with the option to produce either a CSV or a Chart. Just comment out (using /* COMMENT */) the output type that you do not want.

Running Log Parser with the Supplied SQL Scripts

To generate reports using the sample sql files that you created above open, type in (or copy & paste) this command into the command prompt that you opened up above.

LogParser.exe -i:IISW3C file:<<LOCATION TO SQL FILE>>\Hits_By_User.sql

LogParser.exe -i:IISW3C file:<<LOCATION TO SQL FILE>>\Hits_By_DeviceType.sql

EXAMPLE:

LogParser.exe -i:IISW3C file:c:\Public\LogParser\SQL\Hits_By_DeviceType.sql

Log Parser Help

The Log Parser Help is included as part of the Log Parser 2.2 install. We have found this Help file to be very helpful in getting started. The Reference section is especially informative.

Here are some sections that were helpful…

Reference\

- Command-Line Operation\Query Execution Mode

- Query Syntax\*

- Input Formats\*

- Output Formats\*

We hope you find these two scripts helpful in getting started with your EAS reporting.

- Michael Higashi and Patrick Tousignant

Minimum permissions necessary to access mailbox data

Exchange — Wed, 25 Jan 2006 22:47:00 GMT

I was asked the following question recently: What are the minimum permissions necessary I need to grant a user in order for that user to be able to access the data in another user’s mailbox?

Automatically I referenced following article 821897 that states Send As and Receive As permissions are necessary. What we came to find out was that the Receive As permission was the only permission necessary to access information in the mailbox and the Send As permission was only needed to add data into the mailbox and not simply access the information.

After some research I have the answer to the question:

There are two methods to grant permission for a user to access another user’s mailbox through Outlook by selecting File -> Open -> Other User’s Folder. If it is a custom application that is accessing the mailbox using WebDav or CDO code yet another set of permissions are needed.

MAPI permissions

To begin we can use MAPI permissions assigned at the top level mailbox object.

The “reviewer” role gives the user the ability to read items and files only. In order for all folders to be viewed the Folder Visible permission must be present on the top level. Here are the steps:

Here’s what I would suggest:

1. Open the target mailbox

2. Right click on the particular folder you want to grant the access to and choose Properties

a. Choose the Permissions tab

b. Click Add and from the address list choose the user to grant access to and then click okay

c. Choose the reviewer right and then click okay

3. Right-click on the Mailbox – <Target Mailbox Display Name> folder and choose Properties.

a. Choose the Permissions tab

b. Click Add and from the address list choose the user to grant access to and then click okay

c. Leave the role as None and select the “Folder Visible” permission and then click okay

Outlook folder permissions

http://office.microsoft.com/en-us/assistance/HP052422871033.aspx

Full Mailbox Access at the Active Directory Level

The second way to grant access to access the items in the mailbox through Outlook is to assign the full mailbox permissions to the mailbox in the Active Directory.

There are a couple of scenarios that could occur depending on which version of store is installed on the Exchange server. If no hotfixes have been applied then the user will be able to delete messages, create messages, read items and files. The user will also be able to send email from that mailbox. If the server is at store.exe hotfix builds 7233.51 and higher for Exchange Server 2003 Service Pack 1 (SP1) or Store.exe hotfix builds 7650.23 and higher for Exchange Server 2003 Service Pack 2 (SP2) then the user will have the all the same permissions to delete messages, create messages, read items and files, but will no longer be able to send messages from the mailbox.

The following article has more information:

A delegate user who has "Full mailbox access" permissions for another user's mailbox can send e-mail messages as the mailbox owner in Exchange Server 2003
http://support.microsoft.com/kb/895949/

This change was also discussed in this blog post.

When using a custom application

When a custom application is accessing the mailbox the Receive As permission is necessary. The Receive As permission on the mailbox gives the user access to the same tasks as when granting Full Mailbox Access: delete messages, create messages and send email (as the user who is accessing/logged into the mailbox), read items and files, but also gives the user the permission to copy data out of the mailbox.

1. Open the Exchange System Manager

2. Expand the Organization

3. Right click and choose the Delegate control…

a. Click next and then Add and browse. From the object picker choose the user or group you want to grant access and click okay

b. Make sure the Exchange View Only Administrator is chosen and click okay

c. Click next and finish

Note: The Exchange View Only Administrator can also be set on the administrative group level using step 3

264733 How to enable the Security tab for the organization object in Exchange 2000 and in Exchange 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;264733

4. Next expand the Administrative Groups container

5. Expand <administrative group name>, Servers container, <server name>, the Storage Group and <storage group name>

a. Right click on the store containing the mailboxes you want to grant access to and choose properties

b. Click on the security tab and choose the user from the list. If the user was added as part of a group at the View Only Administrator level then that individual user will need to be added at this time if the entire group is not going to be granted Receive As permissions here.

c. Scroll down the list of permissions and check allow for the Receive As permission and then click okay

Note: The Receive As permission can be set at any level under the Organization

6. The Information store may cache this data and it can take up to 2 hours for this cache to be flushed. Dismount and remount the store to flush this cache immediately

If the user is already an Exchange Full Administrator you will need to follow the steps in either of the following two articles (depending on the version of Exchange you are using) as the Receive As permission will be inherited as a deny:

821897 How to assign service account access to all mailboxes in Exchange Server 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;821897

262054 XADM: How to Get Service Account Access to All Mailboxes in Exchange 2000

http://support.microsoft.com/default.aspx?scid=kb;EN-US;262054

Working with Store Permissions in Microsoft Exchange 2000 and 2003

http://www.microsoft.com/downloads/details.aspx?familyid=2ae266f0-16b7-40d7-94d9-c8be0e968a57&displaylang=en

- Charlotte Raymundo

Public Folder Replication Troubleshooting – Part 3: Troubleshooting Replica Deletion and Common Problems

Exchange — Tue, 24 Jan 2006 02:33:00 GMT

This is a third blog post about troubleshooting public folder replication issues. In first post we covered Troubleshooting the Replication of New Changes. The second blog post covered Troubleshooting the Replication of Existing Data. This post in this series will cover Troubleshooting Replica Deletion and Common Problems. To get the full picture, please read all referenced material!

Troubleshooting Replica Deletion

You removed an old server from the replica list on all your folders. However, when you go to Public Folder Instances for the old store in ESM, you still see a bunch of folders there. This is due to a problem with the replica deletion process. In the Exchange 2003 Sp2 version of ESM, if you try to delete a public store in this state, ESM presents a dialog stating:

“You cannot delete this public folder store because it contains folder replicas. To avoid data loss, right click the public folder store and use Move Replicas to move the replicas to a different server. It may take several hours until the content is replicated to the new server and the local replicas are removed.”

When you remove a store from the replica list on a folder, that store does not immediately delete the data. Instead, it sends out a special 0x20 status request to all the other replicas. This is called a Replica Delete Pending Status Request (RDPSR), and can not be distinguished from a normal status request in the application log. A RDPSR contains a flag that indicates the replica is pending deletion. When the other stores receive this 0x20, they respond with a special 0x10 called a Replica Delete Pending Ack (RDPA). The RDPA indicates that it’s ok to delete that data – but the other stores only send this 0x10 if they already have all the CNs that the pending deletion replica has. The replica will only be deleted once the store has received a 0x10 indicating that someone else has the data.

This means that if you delete the store before Public Folder Instances is empty, you are probably losing data. Only 2003 Sp2 ESM will stop you from doing this – in older versions you must manually check Public Folder Instances to see if it’s ok to delete the store. You should always check Public Folder Instances before deleting a public store, and when 2003 Sp2 ESM gives you this warning, you should not try to ignore it or work around it – instead, you should troubleshoot the replica deletion process.

Note that prior to Exchange 2003 Sp2, the server that was removed from the replica list only sends the RDPSR once. If no one responds, you’ll see that the folder just stays in Public Folder Instances indefinitely, unless you add the store back to the replica list and then remove it again, causing a new RDPSR to be sent. 2003 Sp2 changed this behavior so that the store will retry every hour until it gets a RDPA from someone.

Troubleshooting

This is almost the same as troubleshooting the backfill process.

1. Did the pending deletion replica send a 0x20?

Unless you already had logging turned up when you removed the replica, you won't know. Fortunately, you can just add the replica back and then remove it again. Then watch the application log for the 0x20.

2. Did the 0x20 reach the other replicas?

You should know the drill by now. Check the application logs on the other replicas to see if they received the 0x20.

3. Did any other replica respond with a 0x10?

This is the part you'll probably end up focusing on. If a replica received the 0x20 from the pending deletion replica, but did not respond with a 0x10, that means that the pending deletion replica has data that the other replica doesn't. Since you know it just received a 0x20 from that replica, then you also know that it already knows what data it's missing. Therefore, you'd expect to see a backfill request for that folder every 24-48 hours. Watch the application log, and troubleshoot it exactly like the normal backfill process described earlier.

4. Did the pending deletion replica receive the 0x10?

Once any other replica has all the data, that replica should respond with a 0x10. When the pending deletion replica receives that 0x10, it will finally be willing to delete that data. That doesn't mean it will happen immediately, though. If there are clients using that replica, it won't be deleted until later during online maintenance. If you want, you could speed this up by dismounting and mounting the store to disconnect the clients.

Common Problems

You may find that a server sent some type of replication message to another server, but the receiving server never logged the incoming message in the application log. However, message tracking says it was delivered locally to the store on that server. This behavior usually indicates either a problem with the Replication State table or a permissions problem on the SMTP virtual server.

Let's cover the easy one first. One problem that causes an incoming replication message to be ignored by the receiving server is a problem with the Replication State, or ReplState, table. Note that a problem with the ReplState table may also cause the server to fail to issue backfill requests (0x8) for some folders, so this information also applies to that situation. Each public store uses its ReplState table to track the state of replication for any replicated folders. The table contains multiple rows for each folder - one row per replica. It's possible for the rows in the ReplState table to get out of sync with the replica list, such that it has extra or missing rows. Sometimes you can get it to sync up again just by making a change such as removing a server from the replica list, applying the change, and then immediately adding it back, but this doesn't always work. Fortunately, a ReplState test was added to isinteg. See KB889331 for Exchange 2003, or KB892485 for Exchange 2000. As long as you have the updated isinteg.exe and store.exe, you can use isinteg to correct the problem with the ReplState table. If you run only the ReplState test, it is typically very fast - less than a minute even on a large public store. Once isinteg has been run, you may still need to go back and make a change to the folder to get the ReplState table to sync up with the replica list. After they're in sync, the server should be able to process the incoming replication messages, or should begin issuing backfill requests normally.

The other common problem that causes an incoming replication message to be ignored is an issue specific to Exchange 2003. An Exchange 2003 server requires that the sending server has the Send As right on the receiving SMTP virtual server. That is, if ServerA is Exchange 2003, and ServerB is sending a PF replication message to ServerA, ServerB must have Send As on ServerA's SMTP virtual server. Otherwise, ServerA does not process the incoming replication message. This permission is normally granted through the Exchange Domain Servers groups. If the Send AS right is the problem, all incoming replication messages from a particular server will fail. I find it easiest to identify this problem with a network trace taken while a replication message is being transferred from one server to another. The conversation should go like this:

ServerA: 220 ServerA.microsoft.com Microsoft ESMTP MAIL Service...

ServerB: EHLO ServerB.microsoft.com

ServerA: 250-ServerA.microsoft.com Hello

250-TURN

250-SIZE

250-ETRN

250-PIPELINE

250-DSN

250-ENHANCEDSTATUSCODES

250-8bitmime

250-BINARYMIME

250-CHUNKING

250-VRFY

250-X-EXPS GSSAPI NTLM LOGIN

250-X-EXPS=LOGIN

250-AUTH GSSAPI NTLM LOGIN

250-AUTH=LOGIN

250-X-LINK2STATE

250-X-EXCH50

250 OK

The important part here is that ServerA must be advertising the GSSAPI NTLM LOGIN options. If you don't see these in ServerA's response to the EHLO, it's usually because Integrated Windows Authentication has been unchecked on the SMTP virtual server. This is mentioned in step 1 of KB843106 and step 3 of KB842273. As long as these authentication verbs appear, you should see ServerB try to use them:

ServerB: X-EXPS GSSAPI

ServerA: 334 GSSAPI supported

ServerB: <a bunch of base64 encoded data>

ServerA: 334 <more base64 encoded stuff>

ServerB: CRLF

ServerA: 235 2.7.0 Authentication successful.

If authentication does not succeed, you may have a kerberos problem or an issue with the computer account for ServerB. Next the servers will transmit linkstate information. After that, they finally get around to the business of transferring email:

ServerB: MAIL FROM:<ServerB-IS@microsoft.com>

ServerA: 250 2.1.0 ServerB-IS@microsoft.com....Sender OK

ServerB: RCPT TO:<ServerA-IS@microsoft.com> NOTIFY=NEVER

ServerA: 250 2.1.5 ServerA-IS@microsoft.com

ServerB: XEXCH50 2404 2

ServerA: 354 Send binary data

It's this last response to the XEXCH50 verb that's important. If the response is "354 Send binary data", then everything is fine, at least as far as permissions to the SMTP virtual server are concerned. If the GSSAPI NTLM login options were not advertised, or the authentication attempt failed, then it's expected that ServerA will instead respond with "504 Need to authenticate". If those steps succeeded, but ServerA still says "504 Need to authenticate" instead of "354 Send binary data", then ServerB does not have the Send As right on ServerA's SMTP virtual server. There are several ways this could happen. For one, when you delegate rights such as Exchange Full Administrator in ESM, that user or group inherits a deny on the Send As right. Therefore, using ESM to delegate admin rights to the computer account, the Exchange Domain Servers group, or some other group that contains the Exchange servers will break public folder replication. Another possibility is that the computer account is not in the Exchange Domain Servers group, which is how it normally has the Send As right. You'll need to evaluate the permissions on the SMTP virtual server and determine why the computer account for the sending server does not have the proper rights. See KB843106 and KB842273 for more details about the "504 Need to authenticate" problem.

Conclusion

You may have noticed as you read through this document that Sp2 for Exchange 2003 contains several important enhancements under the hood to prevent replication issues and assist in troubleshooting them. Environments with multiple public stores can really see a huge benefit from Sp2, especially when it comes to moving replicas between servers, and adding and removing public stores.

I hope this was helpful. Thanks to Dave Whitney for reviewing all this!

- Bill Long

Public Folder Replication Troubleshooting – Part 2: Troubleshooting the Replication of Existing Data

Exchange — Thu, 19 Jan 2006 19:53:00 GMT

This is a second blog post about troubleshooting public folder replication issues. In first post we covered Troubleshooting the Replication of New Changes. This blog post covers Troubleshooting the Replication of Existing Data and the last post in this series will cover Troubleshooting Replica Deletion and Common Problems. To get the full picture, please read all referenced material!

Troubleshooting the Replication of Existing Data

When new changes replicate, but old unchanged stuff doesn't, you have a backfill problem. The most typical situation where hierarchy backfill occurs is when a new public store is created. The most typical content backfill scenario is when a public store has been added to the replica list of a folder.

When you have a backfill problem like this, it may have already occurred to you that there’s an easy workaround – just make a change to all the items. By doing this you circumvent the broken backfill process and replicate everything as new changes. Despite the fact that I wrote both of the tools that are typically used for this (PFDAVAdmin and ModifyItems), it’s usually best to troubleshoot the backfill process and fix the root cause. If you just change everything to make it replicate, you may end up with the same backfill problem in the future when the replicas get out of sync again. That said, let’s move on to discussing backfill. To understand the backfill process, it's first necessary to understand how changes are tracked.

Every folder and message in the store is assigned a Change Number (CN) when it is created and every time it is modified. When replication occurs, the CNs on each object are used to determine whether that object needs to be replicated. A group of CNs is called a CNSet. The CNSet for a particular folder on a particular server is called status information. This status information is included on every replication message. Every message type 0x2 contains the hierarchy status for the sending server. Likewise, every message type 0x4 contains the status information for that particular folder for the sending server. All other replication message types contain status information for their respective folders as well.

When a new public store mounts for the first time, it sends a status request (type 0x20) for the hierarchy to all the existing public stores. Similarly, when a new store is added to the replica list of a folder, that store will send a 0x20 to all other replicas of that folder. Like every replication message, a status request contains a CNSet of all CNs for the folder in question (or the hierarchy) that the originating store has, and asks the other stores to respond if they have CNs that the originator doesn't. Note that prior to Exchange 2003 Sp2, every replica was not asked to respond to the status request, so some stores would ignore the status request even if they had changes that the originating store did not. A 2003 Sp2 server will ask for responses from all replicas, and will respond even when the originating server did not specifically ask it to, as long as it has changes that the originating server does not. This can greatly improve the decisions made during the backfill process. The unique thing about a status request is that it doesn't contain any data to replicate - it just has a list of change numbers. The other stores respond with status messages (0x10), which list their own CNSets for that same folder (or the hierarchy). When the originating server receives the 0x10 messages, it compares the CNSet contained within to its own CNSet. If the 0x10 contains changes that the store doesn't already have, the backfill process begins.

The first step in the backfill process is to add entries to the backfill array for the folder in question. These entries have a CNSet that describes the missing changes, and a timeout value describing when the store will request the missing data. The backfill timeout will vary depending on the situation. In the case of a new public store being brought online or a new replica of a folder being added, the initial timeout is 15 minutes.

Backfill entries may be added to the backfill array during the course of normal operation as well. Consider a situation where a particular public store has broadcast two changes in two separate 0x2 messages. Let's say the administrator deletes the first 0x2 message out of the queue, but the second one makes it through. When the other servers receive this 0x2, they will find that the CNSet in the status information contains CNs that they never got. As a result, they will create backfill entries for that data. Backfill entries for missing data that was discovered during the normal course of replication will start with a timeout of 6 hours if the data is available in the same Routing Group (RG), or 12 hours if it is only available in a different RG. Each time a backfill request is issued, the next timeout will be 12 and then 24 hours for intra-RG requests, or 24 and 48 hours for inter-RG requests.

Every five minutes the store will check to see if any backfill entries have reached their timeout. If they have, a backfill request (type 0x8) is issued for the missing CNs, and the timeout is set to the next interval. A backfill request is not a broadcast; it is directed at a single server - one of the servers that previously indicated it had the missing CNs in the status information it sent to the requesting server. When that server receives the incoming 0x8, it immediately processes the request and responds with one or more backfill responses (0x80000002 for hierarchy or 0x80000004 for content), which contain the actual data for the requested change numbers. Like backfill requests, backfill responses are not broadcasts - they are sent only to the requesting server.

If the requesting server successfully processes the incoming backfill response, the CNs it contained are cleared from the backfill array on that store. Actually, any incoming message that contains CNs that are outstanding in the backfill array will cause those CNs to be cleared from the array.

Troubleshooting

As you can see, there are a lot more questions to answer when troubleshooting the backfill process.

1. Does the store know it's missing data?

First you should determine if the server even realizes that other stores have changes that it needs to request. Unfortunately, there is no supported tool or utility that will let you view the backfill array directly to see if it has anything in it. However, there are other more indirect ways of figuring this out.

One way is to wait. If the server knows it's missing data, it will be requesting it at least once every 24 or 48 hours. This means you can simply turn up logging and wait to see if a 0x8 message ever goes out. If you never see a 0x8 for the folder in question, but you are seeing 0x8's for other folders, you may have hit the outstanding backfill limit, which we'll discuss shortly.

Another option is to make sure the server receives the latest status information. Remember, the server only sends a status request that one time after you add the new replica. After that, the only status information it receives will be through the normal course of replication. So if its initial attempt to get status was lost because the 0x20 or the 0x10 in response was lost or deleted, it may sit there indefinitely and not even realize that it's missing anything. There are several ways to make sure the server has received status information for a folder.

- Go to a server that has all the data and make a change to the folder by adding, deleting, or modifying a message. In the case of the hierarchy, create, delete, or change the properties of a folder. The resulting 0x4 or 0x2 will contain status information for that folder or the hierarchy, respectively. When the server that's missing the data successfully processes the incoming replication message, you know that it has added any appropriate entries to the backfill array.

- Use the Synchronize Content option in Exchange 2003 ESM. This is a well-hidden but very useful option. To find it, go under the Public Folders tree and go to the folder in question. Highlight the folder in the left-hand pane. In the right-hand pane click the Status tab. Right-click on the server that has all the data and choose Synchronize Content. This does two things - it causes the server to issue a status request 0x20 for the folder, and it causes it to immediately timeout any backfill entries. Notice that I said you should Synchronize Content from the server that already has the data. You may wonder why you would do that, when it's the other server that has the backfill entries that need to be timed out. Remember that at this point we're just trying to ensure that the server missing the data KNOWS it has something to backfill. To that end, we can use Synchronize Content from the server that has the data to send a 0x20 to the server that doesn't. In this case we're not really interested in seeing a status 0x10 response to the 0x20. We just want the store missing content to receive a replication message for the folder from a store with content, so it can add the appropriate entries to the backfill array. The 0x20 from the server with the data serves this purpose. Note that in Exchange 2003 Sp2, Synchronize Content is now available for the hierarchy by right-clicking on the Public Folders node itself.

- Use the Replication Flags registry value (KB813629). If you put this value in place, along with the Enable Replication Messages At Startup value from KB321082, it causes the store to send a status request 0x20 for every folder on startup. Again, you would want to use this on the server that has the content - the point of this step is to get the server that has content to send its status information to the server that's missing content.

- Use 2003 ESM to send a backfill response. In 2003 Sp1, you could use the Send Hierarchy option to send a hierarchy backfill response and the Send Contents option to send a folder content backfill response. In 2003 Sp2, both of these options became Resend Changes. This sends a backfill response for the range of data you specify, but you probably shouldn't specify the whole range of data since that might satisfy all outstanding backfill entries and end up working around the original problem. Instead, specify a range of only a day or two. This causes a 0x80000002 or 0x80000004 to go to the target server, which again serves the purpose of giving it status information for the store that has the data.

Once you've used one of these options to force status information, and you've verified that the store missing the data received the incoming message by watching the application log, then you know it knows it's missing the data.

2. Does the store request the missing data?

After you've made sure the store know it needs to backfill some data, does it ever issue a backfill request? Recall that after it has tried to backfill the data a couple of times, You may have to wait 24 or 48 hours for the next backfill request, since that will be the longest timeout interval for intrasite and intersite backfills, respectively. There is one way to speed this up, and that is to use Synchronize Content again, but this time from the server that's missing the data. This will immediately timeout the backfill entries for that folder. However, you may still find that the store does not issue a backfill request for the folder you're focusing on. If this is the case, watch the app log for the next 24-48 hours. If the store is sending backfill requests for other folders, but not for the folder you're focusing on, it may have hit the outstanding backfill limit.

When you experience a situation where you've added replicas of a lot of folders to a new store, and replication seems fine at first but then grinds to a halt over the next day or two, you have probably hit the outstanding backfill limit. The outstanding backfill limit is a mechanism intended to throttle replication. By default, the store will only allow 50 outstanding backfill requests at a time. Once it has 50 outstanding, it will re-request those 50 over and over until they are satisfied. Once any one outstanding entry has been satisfied, that opens up a slot in the OBL for a new set of data to be requested. This means that if 50 requests are having problems being satisfied for whatever reason, replication can not proceed.

If you are seeing this behavior, you should watch the application log to see what the store is requesting. You'll be seeing periodic 0x8 messages for the current 50 outstanding backfill requests, and you'll find that no backfill response is received, which is why they're still outstanding. At that point you should change your focus to troubleshooting one of the folders the store is currently trying to backfill, since resolving the problem will allow it to move on to other folders.

There is one other option, and that is to increase the Oustanding Backfill Limit (OBL). You can do this by creating a registry value called Replication Oustanding Backfill Limit under the registry key for that store. The maximum value is 5000 decimal. However, once you do this the replication floodgates will open and it will be hard to determine which 50 folders caused it to choke. You'll need to postpone troubleshooting until things settle down again. Typically I recommend leaving the limit at 50 and fixing the problem, instead of working around it by increasing the limit.

If the OBL doesn’t appear to be a problem, and you still aren’t seeing outgoing 0x8 messages for the folder in question, see the “Common Problems” in last post of this series.

3. Does the other store respond to the request?

Once you have a backfill request to focus on, you need to determine if the backfill target ever got the request. Check the application log on that server for the incoming 0x8. You can also search the application log for the message ID mentioned in the outgoing event from the sending side. If you can find no sign of it in the application log, use message tracking to see how far it got. If it received the 0x8, it should respond almost immediately with one or more 0x80000002 or 0x80000004 messages (you will often see many backfill responses to a single backfill request, since the changes are not all sent in a single message). Of course, the time it takes to generate the backfill response messages will vary based on the data in the folder and the replication message size limit. For instance, if you set the maximum replication message size to 1 GB, the responding server could try to pack the entire hierarchy into a single backfill response, which might take an hour or more just to pack up!

4. Does the requesting server get the response?

Now it's time to check the application log on the requesting server to see if it received the backfill response. If not, track the message and see how far it got. If it received the backfill response and logged it in the application log, then that backfill request should have been satisfied and it should be able to move on.

As mentioned earlier, if you find that message tracking shows that one of these messages was delivered to the store, yet the application log does not show the incoming replication message, have a look at “Common Problems” in last post of this series.

In next blog post: Troubleshooting Replica Deletion and Common Problems.

- Bill Long

Public Folder Replication Troubleshooting – Part 1: Troubleshooting the Replication of New Changes

Exchange — Tue, 17 Jan 2006 23:21:00 GMT

Introduction

The purpose of this blog post series is to serve as a guide to troubleshooting public folder replication problems. It will not tell you exactly how to fix every possible replication problem. However, it will show you how to isolate every possible replication problem so that you focus your troubleshooting on the point of failure. Put another way, this post is intended to take you from a problem description like “The content on my old server isn’t replicating to my new server” to a much narrower problem description like “My old server isn’t responding to the status requests from my new server, therefore the new server doesn’t know it’s missing data and isn’t trying to backfill. This means the problem is actually with the old server.” This post will also describe how to identify a few of the most common replication problems. Before I get into the details of troubleshooting, I want to give an overview of my general approach to these issues.

The best troubleshooting tool for public folder replication is the application log. In order to isolate a replication problem, you must be able to follow the replication events in the log to see exactly where the process is breaking down. Typically, you should turn up Diagnostics Logging on Replication Incoming and Replication Outgoing to maximum as a starting point for troubleshooting. Each time a store sends or receives a replication message, it will log an event to that effect (assuming logging is turned up). The various kinds of replication messages can be differentiated by the 'Type' shown in the description of the event. I prefer to focus on the Type rather than the event ID for several reasons:

- Event IDs change between versions of Exchange. For instance, in Exchange 5.5 an outbound backfill request was a 3014. In Exchange 2000 and 2003 it's a 3016.

- Incoming and outgoing event IDs are different for each type. An outgoing hierarchy message is a 3018, while an incoming hierarchy message is 3028.

- Status requests and status messages use the same event IDs, even though they are two different types of messages. Thus, you can't distinguish between them from event ID alone.

Focusing on the Type is a little easier. You can easily correlate these with Event IDs by examining your application log. There are only 7 types, and you can see if the message is outgoing or incoming by looking at the Category of the event. If you focus on types instead of event IDs, all you need to remember is:

Hierarchy - 0x2

Content - 0x4

Backfill Request - 0x8

Backfill Response - 0x80000002 (for hierarchy) or 0x80000004 (for content)

Status - 0x10

Status Request - 0x20

Also note that Replication Errors logging is rarely helpful. Even when replication is working normally, most servers will generate lots of replication error events, such as event ID 3093 indicating there was an error reading a property. In most cases the property has no effect on replication and the error can be ignored. I recommend leaving Replication Errors logging at None unless you're looking for something specific, such as the problem described in this previous blog post.

Troubleshooting the Replication of New Changes

To troubleshoot public folder replication, you must first be familiar with the normal message flow that is expected when replication is working. Based on that knowledge, you can identify the point of failure when a problem arises. Before I discuss how to troubleshoot the replication of new changes, let's describe what the expected behavior is.

Hierarchy Replication

The replication of new hierarchy changes takes place whenever a folder is created or deleted, or a change is made to the properties of a public folder, such as the replica list, client permissions, description, administrative note, or storage limits. Note that this does not include the email addresses on a mail-enabled folder. The email addresses are stored on the directory object in the Active Directory, so changing them does not result in hierarchy replication. Only changing the properties stored in the public store itself will cause hierarchy replication.

Every 15 minutes (by default), the store broadcasts any changes that were made to the folder properties in one or more hierarchy replication messages. With logging turned up to Maximum on Replication Outgoing, you'll see an event like this on the server where the hierarchy was modified:

Event Type: Information

Event Source:     MSExchangeIS Public Store

Event Category:   Replication Outgoing Messages

Event ID:   3018

Description:

An outgoing replication message was issued.

Type: 0x2

Message ID: <91ACCD0758385549A56A10971798985572D5@bilongexch1.bilong.test>

Database "First Storage Group\Public Folder Store (BILONGEXCH1)"

CN min: 1-72CF, CN max: 1-72D3

RFIs: 1

1) FID: 1-6994, PFID: 1-1, Offset: 28

      IPM_SUBTREE\NewFolder

Notice the "Type: 0x2" at the beginning of the description, identifying this as a hierarchy replication message.

A hierarchy replication message is sent from the originating server directly to all other public stores. There's no concept of topology for the replication of new changes. All changes to the hierarchy are sent directly from the server on which the changes were made to all other servers that have a public store associated with the same hierarchy. Every other server should log an incoming event showing a type 0x2 when they successfully process the incoming replication message.

Content Replication

Content replication takes place whenever a message is created or deleted, or the properties of a message are changed. The times at which the store broadcasts content changes for a folder can be modified by changing the replication schedule on that folder, but by default the changes will be broadcast every 15 minutes just like the hierarchy. A content replication message is identified by the type 0x4 in the event description. Once again, there is no concept of topology for the broadcast of new changes. When the content of a folder is modified on any given replica, that replica will send a 0x4 message directly to all other replicas of the folder. And again, every receiving server should log an incoming 0x4 event when they successfully process the incoming message.

Troubleshooting Steps

Those are the two most basic scenarios for replication. When new hierarchy changes or new content changes are not making it from one server to another, troubleshooting is very straightforward.

1. Did the server generate an outbound replication message?

So you made a change to a folder or you added content to a folder on particular server, and that content didn't make it to some other server. The first question to answer is did the target server broadcast the changes? When troubleshooting, it's important to keep track of what server you're working against when you make changes. There are several ways to accomplish this. In ESM, you can right-click on the Public Folders node and choose "Connect To" to point to a particular store. For the most part ESM will make any changes on the specified store, but be aware of one exception - Client Permissions. Prior to Exchange 2003 Sp2, when you changed Client Permissions through ESM, ESM would attempt to make the change against a store that houses a replica of the folder, even though this isn't necessary since the permissions are stored as a property of the folder in the hierarchy. With the 2003 Sp2 version of ESM, this was changed so that it now does make the change on the hierarchy you’ve pointed it to. When you're testing hierarchy replication by making changes through ESM, you should avoid using the permissions for testing since it may be hard to predict which store the change will be made against, unless you’re running ESM from 2003 Sp2. If you're using Outlook and you want to verify which replica of a folder you are hitting, you can use MFCMAPI or a similar tool to view the PR_REPLICA_SERVER property of the folder. This will show you the name of the server that Outlook is using to access the content of that folder.

If the replication schedule is Always for the folder in question (which is always true for the hierarchy), and you don't see an outbound 0x2 or 0x4 within 15 minutes, then you know something is wrong on the originating server. If the server is not generating any outbound hierarchy or content broadcasts, the replication agent may have failed to start. One of the common scenarios is described in KB272999. The important thing to note here is the 3079 event:

Event ID: 3079

Source: MSExchangeIS Public

Type: Error

Category: Replication Errors

Description:

Unexpected replication thread error 0x3f0.

EcGetReplMsg

EcReplStartup

FReplAgent

This will be logged even with no additional logging turned up when you mount the public store. If the 3079 includes the "EcReplStartup" function, this means the replication agent failed to start and no new changes will be broadcast until the problem is corrected and the store is mounted again.

Hierarchy replication is also vulnerable to certain permissions problems when Exchange 5.5 public stores are present in the organization. When an Exchange 2000 or 2003 server sends a hierarchy replication message to an Exchange 5.5 server, it must produce a ptagACLData property (the 5.5-style permissions based on legacyExchangeDN) from the ptagNTSD property (the 2000-style permissions based on SID). This means each SID must be converted to a legacyExchangeDN. This SID to legacyExchangeDN conversion can fail for several reasons. For instance, if a SID resolves to more than one user, an event like this may be generated:

Event ID: 9528

Category: General

Source: MSExchangeIS

Type: Error

Description:

The SID S-1-5-21-408248388-469072634-37170099-1391 was found on 2 users in the DS, so the store cannot map this SID to a unique user.

The users involved are:

/DC=com/DC=company/CN=Users/CN=User1

/DC=com/DC=company/CN=Users/CN=User2

Since the SID can't be converted to a legacyExchangeDN, the store will fail to generate an outbound hierarchy broadcast message.

2. Was the message addressed to the server that didn't receive the change?

If the originating server generated the outbound message, the next step is to be sure it was addressed to the server that didn't receive the data. The easiest way to verify this is to track the message. In message tracking, you can just track the message ID that was reported in the outbound replication event. In the Message History window, the To: line will be visible. If the store that didn't receive the changes is not listed here, then again the focus should be on the originating server. Does it see that server in the organization? Does that server have email addresses on its public store object? Does the originating server see that store in the replica list on the folder in question?

3. Did the message make it to the destination server?

Once you've verified that the message was addressed to the destination server, the next question to answer is - did the message make it that far? You can determine this through message tracking. If message tracking says the message was delivered to the store, but you don't see an incoming replication event acknowledging the message, see the "Common Problems" section in the last post of this series.

In next blog post: Troubleshooting the Replication of Existing Data.

- Bill Long