Mike Lagase : Outlook

Windows Desktop Search and the implications on WAN performance

mikelag — Mon, 04 May 2009 17:26:29 GMT

Windows Desktop search (WDS) is a great tool to help you to search through the unwieldy plethora of documents or emails that you may have scattered across your desktop. With the addition of 3rd party IFilter add-ins, it makes it even easier to find what you are looking for.

As of version 3.01, Desktop search has disabled the indexing of online mailbox on a default installation due to performance implications on the Exchange server side. Companies sometimes have the need to still run Outlook in Online mode due to security requirements of not having local OST’s, or they need real-time email for business purposes. With some of those requirements, companies have the need to also have fast message/document retrieval which Windows Desktop Search can surely do without a problem.

WDS does have some group policy settings that will now allow online indexing of mailboxes and a listing of all the settings for WDS 4 is at http://technet.microsoft.com/en-us/library/cc732491.aspx. This setting that allows indexing of online mode Outlook profiles is "Enable Indexing uncached Exchange Folders". Once this is deployed via group policy, WDS will now start indexing online mode Outlook profiles. This of course could put a huge strain on the server as all of the users data is being indexed if deployed to a large user base. Recommended guidance states that you should deploy this policy to smaller subsets of users to prevent possible server performance problems. This is similar guidance to what Microsoft recommends for cached mode deployments.

With that said, I would like to now take us down a road where certain combinations of WDS policies can not only affect Exchange server side performance, but can also have serious implications on WAN performance. If you currently have a centralized Exchange deployment and users are accessing all of their email across WAN circuits, read on.

Let’s say you have an administrative assistant running in cached mode that needs to gain access to another users complete mailbox with a requirement that data in that mailbox is easily discoverable. This requirement can be easily met by using Windows Desktop Search and is very common in law firms. A default Outlook 2007 installation will have the "Download shared folders (exclude mail folders)" option selected for their user profile, so if this assistant had previously opened a another users non-email folder such as Contacts, Calendar and Tasks, WDS will index those items without any issue. This feature unfortunately does not meet the complete requirement as we need to index all items in the other users mailbox. After Full mailbox permissions is added for this assistant, they can now add this other users mailbox to their profile to view their data. Once you do this, you will now see that WDS will still not find any email items unless you have selected the folder in the mailbox and then performing the search. Everything so far is the default behavior.

WDS has a feature which will allow you to index online delegate mailboxes and is deployed via the GPO setting "Enable Indexing of online delegate Mailboxes". Once this setting is deployed, any user that has another users mail related folder in their profile will now get indexed. So that seems like a good thing, no? Well, we all know that indexing any type of mailbox in online mode will increase overall performance on the Exchange server and if users are doing this over a WAN, you will now see increased WAN utilization while WDS is indexing this data making direct RPC calls to the Exchange server. If this setting was deployed to a large user base while there are a number of profiles that have other mailboxes added to their profile, you could potentially saturate this network circuit. Your network administrator at this point would obviously not be too happy and your users would then start complaining that email access is really slow or Outlook may get disconnected due to this saturation. Our best practices dictate that this setting should be deployed to smaller user bases at a time to prevent increased client traffic to the Exchange Server.

Imagine deploying this policy to 1000 users all accessing Exchange across a WAN and all have an added mailbox to their profile. By default, WDS will only index 120 items per minute which should help keep the Indexing traffic under control. If all users workstations were indexing this amount of data at a time, we would be seeing about 120,000 items per minute of traffic. Couple that with any attachments that WDS is configured to index for such as PDF or Word documents, and this will make for a very bad network day.

There are ways to change the amount of items that are indexed per minute by modifying the GPO setting "Enable Throttling Online Mailboxes". Setting this policy to a lower value will help reduce the amount of items that are indexed per minute per mailbox and should also help keep some of the network traffic down to a minimum. The caveat here is that it will take longer to index these mailboxes. Keep in mind that is still going to be direct RPC traffic to the Exchange server with minimal amount of throttling.

To help reduce some of this overhead, Outlook 2007 has a registry entry ( CacheOthersMail ) that will allow you to cache other users mail folders in an OST file. This was first introduced in KB955572 and requires that you disable the downloading of headers. This was then rolled up in to the Outlook 2007 post SP1 Sept. 24, 2008 hotfix package (957909) . If the indexing of delegate mailboxes policy has been deployed to these users and you add this Outlook registry key, you will now see a mixture of traffic being generated by WDS. One is direct RPC traffic to the Exchange server and the other is Outlook FxGetBuffer function calls or otherwise known as Outlook sync (ICS). The Outlook sync traffic will become more prevalent over time as the other users mailbox is cached locally in the OST file. FxGetBuffer calls are a lot less expensive than direct RPC calls to the Exchange server, so deploying the CacheOthersMail registry key may help with overall WAN utilization during initial indexing. You still need to plan on increased WAN traffic as synch traffic coming from many clients could also cause potential WAN degradation issues.

WDS Registry Reference

Registry data to index data in your mailbox if you have an Online mode profile
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: PreventIndexingUncachedExchangeFolders
Value: 0

Registry data to index shared mailboxes:
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: EnableIndexingDelegateMailboxes
Value: 1

Registry data to change the amount of mail items that are indexed per minute.
Key: HKLM\software\policies\Microsoft\windows\windows search
DWORD: EnableThrottlingOnlineMailboxes
Value: 120
Accepted Values (Default: 120, Min: 6, Max:600)

Outlook Registry Reference

Registry data to Cache others users mail data in an OST
One-off users
Key: HKCU\Software\Microsoft\Office\12.0\Outlook\Cached Mode
DWORD: CacheOthersMail
Value: 1

GPO deployed
Key: HKCU\Software\Policies\Microsoft\Office\12.0\Outlook\Cached Mode
DWORD: CacheOthersMail
Value: 1

One of the most taxing combinations of WDS settings with relationship to Exchange Server and WAN performance is deploying "Enable Indexing uncached Exchange Folders" and "Enable Indexing of online delegate Mailboxes" simultaneously. If you also index attachments which is the default behavior, this could put increased burden on network resources and could cause considerable downtime for your users. Deploying these settings needs to be carefully planned out especially in centralized Exchange installations to prevent the situations that I describe.

Outlook 2007 Performance Improvements Hotfix

mikelag — Thu, 12 Mar 2009 18:40:14 GMT

If you haven't heard already, we have released a Pre-SP2 hotfix that help improve Outlook performance and responsiveness in a big way. Here is an excerpt from the article.

Performance improvements

Performance and responsiveness are key concerns for all our customers. That is why we made the large performance tuning and optimization changes that are included in Office suite Service Pack 2 (SP2).

Outlook 2007 SP2 delivers performance improvements in four major areas:

General Responsiveness
SP2 reduces I/O disk usage and UI response time.
Startup
SP2 removes long operations from initial startup.
Shutdown
SP2 makes Outlook exit predictably despite pending activities.
Folder/View Switch
SP2 improves view rendering and folder switching.

Before you go out applying this on your machine, you need to be warned of the first startup experience as we rebuild the tables and indexes in your OST. If you have a large OST, this is going to take some time, but I can tell you that the wait is well worth it. It is actually an entirely new experience at least for me anyway's since I have a good deal of email item counts in my folders. Switching between folders with large item counts is no longer painful and this hotfix provides immediate viewing of these folders.

Grab the hotfix from the following article:

Description of the Outlook 2007 hotfix package (Outlook.msp): February 24, 2009
http://support.microsoft.com/kb/961752

Check out the plethora of improvements in this release in the following article as there are many.

Outlook 2007 improvements in the February 2009 cumulative update
http://support.microsoft.com/kb/968009

Hope this helps tame some of the larger mailboxes that you have.

Mike

Client RPC Dialog box questionnaire for Administrators

mikelag — Thu, 12 Feb 2009 20:00:00 GMT

There are times when you are troubleshooting an Exchange Server issue where it appears that the server is performing OK, but the users are still complaining of the dreaded RPC dialog box and hangs in their client. Most of the time an Exchange administrator or helpdesk personnel needs to speak directly with the end user to determine what actions they were taking at the time the RPC dialog box occurs. Since there are numerous ways which can promote this dialog box, an administrator needs to understand specific actions that users were taking at the time of the problem. A lot of the times, this has nothing to do with server side performance problems, but rather something that is installed on the client or something the user is doing.

I have created a simple document in which the users can answer to allow you to gain some insight in to a users actions and their habits that are aggravating this RPC dialog box.

The document is password protected so that the fields are checkable. The password currently is "Microsoft".

Please provide feedback regarding this document to help make this better.

Thanks,

Mike

Slow Outlook Online Mode Performance

mikelag — Sun, 31 Aug 2008 19:40:00 GMT

Switching to or running Outlook in Online mode against an Exchange 2003 server may exhibit slower than normal performance when clicking on particular folders in your mailbox that contain high item counts.

This is a known issue when the “Show in Groups” option is turned on in Outlook when many items exist in a particular folder. The fix is actually two parts. One for the server side and one for the client side. Note: The server side fix is not necessary for Exchange 2007.

What you might see in Performance Monitor is similar to the load that Client side search engines add. This counter in particular is “Slow Findrow Rate” located under MSExchangeIS Mailbox. One might think that this counter is only used for detecting client search engines, but this counter increases whenever an application is crawling through a users mailbox.

In this case, Outlook creates a search restriction (“Show in Groups” with a descending date sort order) initially using a Fast FindRow call, but the server immediately switches to a Slow FindRow call which causes a large delay in returning the view for that newly created restriction while items in that particular folder are being queried. This could lead up to the dreaded RPC dialog box or the balloon stating the Outlook is requesting data from the server. While we are creating this restriction on the Exchange Server, this may also cause an added CPU performance hit. Since these restrictions are not cached, these problems may occur each time the user clicks on these folders that may have high item counts.

To better optimize this search restriction behavior, an optimized search was added server side so that these restrictions created by Outlook will use the Fast FindRow call method instead of switching to the Slow Findrow method. This provides a huge performance gain and mitigates the Outlook delay problems.

To be able to handle these optimized searches efficiently, your server performance has to be within our recommended performance guidelines, so even though you have applied these fixes, there is still a possibility that these problems could occur due to underlying performance problems server side.

Here are the links to the articles and fixes for each.

Exchange 2003 Server Side fix
Error message when you enable the "Show in Groups" option for a folder that contains thousands of e-mail messages in Outlook: "Outlook is retrieving data from the Microsoft Exchange Server <ServerName>"
http://support.microsoft.com/kb/948828

Outlook fixes
Description of the Outlook 2003 hotfix package: February 21, 2008
http://support.microsoft.com/kb/949289

Description of the Outlook 2007 hotfix package: February 27, 2008
http://support.microsoft.com/kb/949401

Outlook 2007 Performance tips and Messaging Guide for Office 2007

mikelag — Mon, 18 Aug 2008 20:56:17 GMT

Need help troubleshooting Outlook 2007 performance issues? Well, this link will help you do just that to rule out the specific things that might be causing significant performance degradation.

Performance tips for deploying Outlook 2007
http://technet.microsoft.com/en-us/library/cc179071.aspx

Here is a hidden little gem (Messaging guide for Office 2007) that has all kinds of useful information in it, but unfortunately you cannot find it since it is posted as a document and not as a nicely formatted HTML page so our Indexing services can consume it.

Downloadable book: Messaging guide for the 2007 Office release
http://technet.microsoft.com/en-us/library/cc179109.aspx

Mike

WDS 3.01 and online mode crawling

mikelag — Tue, 20 May 2008 20:04:00 GMT

Starting with WDS 3.01, to be able to use Windows Desktop Search to index an outlook profile running in online mode requires that you use group policy to enable this functionality. We index cached mode profiles by default, but online mode crawling is a large performance hit on the server causing the Slow Findrow rate for Mailbox Stores to soar. This has always been an issue with the default behavior, but it is nice to see that we now disable this by default for Online mode profiles.

More information on this can had from the WDS Administration guide. Here is a snippet from this guide.

Using WDS and Microsoft Outlook\Exchange

Finding items in e-mail has been a concern for business people for many years. The hard part is finding a way to keep a current index of all e-mail messages and attachments without excessively taxing the mail server. WDS supports Outlook indexing in both cached and uncached mode. However, WDS 3.01 must be configured via Group Policy to index e-mail when cached mode is turned off (Note, this is different from WDS 2.6x which indexed online content by default).

http://www.microsoft.com/downloads/details.aspx?FamilyID=00645e54-70a8-4d05-906d-af8773cbc728&DisplayLang=en

The two group policy settings that are used to enable/disable this functionality and to force cached mode outlook profiles is the following:

“Disable Cached Exchange Mode on New Profiles”
“Cached Exchange Mode (File | Cached Exchange Mode)”.

This has also been updated in the Windows 2008 online documentation at http://technet2.microsoft.com/windowsserver2008/en/library/584732fa-ad15-4cc6-9c1b-469add7da5621033.mspx?mfr=true

Mike

Outlook 2003 Delegate hotfix available to help in multi-domain environment scenarios

mikelag — Sat, 26 Jan 2008 13:17:00 GMT

I wrote an update to my original blog on the EHLO site regarding Delegate and DL problems in multi-domain environments. This Outlook update helps address the delegate issues that you were seeing previously that caused a lot of folks to redesign their AD infrastructure to work around some of the pitfalls associated with this.

Check it out at http://msexchangeteam.com/archive/2008/01/24/447928.aspx

Update!!
---------------
This change has now been ported to Outlook 2007 and the associated KB Article is http://support.microsoft.com/default.aspx?scid=kb;EN-US;950794

Outlook 2007 users cannot view Free/Busy times for any user.

mikelag — Wed, 19 Dec 2007 00:20:00 GMT

After looking at this, we verified that Autodiscover was in fact working properly and the correct URLS were being returned.

With that said, we then opened https://servername.domain.com/ews/services.wsdl and this returned successfully. This file is used to describe the operations and properties exposed to the client when a SOAP request is made to determine how to handle the free/busy requests amongst other things. More information on the services.wsdl file can be had at http://msdn2.microsoft.com/en-us/library/bb408417.aspx

Enabling Outlook logging per http://support.microsoft.com/kb/831053 , we scheduled a meeting request to view free/busy which of course failed. The good thing is with the logging turned up, we could then looking at the Free/Busy log in the %temp%\olkas directory to see what was up.

Here is what the SOAP request consisted of

<soap:Body><GetUserAvailabilityResponse xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
<FreeBusyResponseArray><FreeBusyResponse>
<ResponseMessage ResponseClass="Error"><MessageText>Mailbox logon failed., inner exception: Cannot open mailbox /o=Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Test1.</MessageText>
<ResponseCode>ErrorMailboxLogonFailed</ResponseCode>
<DescriptiveLinkKey>0</DescriptiveLinkKey><MessageXml>
<ExceptionType
xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">Microsoft.Exchange.InfoWorker.Common.Availability.MailboxLogonFailedException</ExceptionType>
<ExceptionCode
xmlns="http://schemas.microsoft.com/exchange/services/2006/errors">5008</ExceptionCode></MessageXml></ResponseMessage><FreeBusyView>
<FreeBusyViewType
xmlns="http://schemas.microsoft.com/exchange/services/2006/types">None</FreeBusyViewType>
</FreeBusyView></FreeBusyResponse></FreeBusyResponseArray></GetUserAvailabilityResponse></soap:Body></soap:Envelope> 2007/12/17 09:22:48.281: XML response processed successfully

So we received an error code of 5008 which is MailboxLogonFailed

We turned on service side tracing and noticed in the call, we received this error.
Microsoft.Mapi.MapiExceptionUnknownUser: MapiExceptionUnknownUser: Unable to make connection to the server. (hr=0x80004005, ec=1003)

Unknown user, eh? We ran get-mailbox against the mailbox GUID of the user and we found the user no problem. So what gives?

Oddly enough, runnning test-mapiconnectivity -verbose, we see the same failure with this unknown user since 1003 is ecUnknownUser, so this was definately not a client side issue, but a server side issue.

StorageGroup : First Storage Group
Database     : Mailbox Database
Mailbox      : SystemMailbox{81E05D2A-1321-427B-A12D-B78A7931E3D5}
Result       : *FAILURE*
Latency      : 00:00:00
Error        : [Microsoft.Exchange.Data.Storage.MailboxUnavailableException]: C
               annot open mailbox . Inner error [Microsoft.Mapi.MapiExceptionUn
               knownUser]: MapiExceptionUnknownUser: Unable to make connection
               to the server. (hr=0x80004005, ec=1003)
               Diagnostic context:
                   ......
                   Lid: 28153   EcDoConnectEx exception [rpc_status=0x6BA][latency=0]
                   Lid: 16280   dwParam: 0x6BA      Msg: EEInfo: ComputerName: n/a
                   Lid: 8600    dwParam: 0x6BA      Msg: EEInfo: ProcessID: 4932
                   Lid: 12696   dwParam: 0x6BA      Msg: EEInfo: Generation Time: 2007-12-17 16:33:29:752
                   Lid: 10648   dwParam: 0x6BA      Msg: EEInfo: Generating component: 2
                   Lid: 14744   dwParam: 0x6BA      Msg: EEInfo: Status: 1722
                   Lid: 9624    dwParam: 0x6BA      Msg: EEInfo: Detection location: 390
                   Lid: 13720   dwParam: 0x6BA      Msg: EEInfo: Flags: 0
                   Lid: 11672   dwParam: 0x6BA      Msg: EEInfo: NumberOfParameters: 2
                   Lid: 8856    dwParam: 0x6BA      Msg: EEInfo: prm[0]: Unicode string: ipc017.indelan.com
                   Lid: 8856    dwParam: 0x6BA      Msg: EEInfo: prm[1]: Unicode string: IPC017
                   Lid: 23065   EcDoConnectEx called [length=182]
                   Lid: 17913   EcDoConnectEx returned [ec=0x3EB][length=56][latency=0]
                   Lid: 19778
                   Lid: 27970   StoreEc: 0x3EB
                   Lid: 17730
                   Lid: 25922   StoreEc: 0x3EB

We found out that there was a failed SP1 installation on the server and many things were done to the server before I got my hands on this issue. We had to fix that problem and shortly thereafter, we saw the following in the application log and this is the same Unknown user error that we received before.

Event Type: Warning
Event Source: MSExchange Search Indexer
Event Category: General
Event ID: 107
User:  N/A
Description:
Exchange Search Indexer has temporarily disabled indexing of the Mailbox Database First Storage Group\Mailbox Database (GUID = 81e05d2a-1321-427b-a12d-b78a7931e3d5) due to an error (Microsoft.Mapi.MapiExceptionUnknownUser: MapiExceptionUnknownUser: Unable to make connection to the server. (hr=0x80004005, ec=1003)
Diagnostic context:
    Lid: 23065   EcDoConnectEx called [length=182]
    Lid: 17913   EcDoConnectEx returned [ec=0x3EB][length=56][latency=0]
    Lid: 19778
    Lid: 27970   StoreEc: 0x3EB
    Lid: 17730
    Lid: 25922   StoreEc: 0x3EB
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Int32 ec, DiagnosticContext diagCtx)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize)
   at Microsoft.Mapi.ConnectionCache.OpenMapiStore(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)
   at Microsoft.Mapi.ConnectionCache.OpenMailbox(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)
   at Microsoft.Exchange.Search.MailboxCache.GetMailbox(ServerMdbAndMailboxGuids guids)
   at Microsoft.Exchange.Search.RetriableOperations.GetMailboxForUser(ThreadLocalCrawlData crawlData, Guid mailboxGuid, Object unusedSourceParam, Object unusedParam1, Object unusedParam2)
   at Microsoft.Exchange.Search.RetriableOperations.DoRetriableMapiOperation[SourceType,ReturnType,Parameter1Type,Parameter2Type](ThreadLocalCrawlData crawlData, Guid mailboxGuid, SourceType source, Parameter1Type parameter1, Parameter2Type parameter2, MapiOperationDelegate`4 operationDelegate)
   at Microsoft.Exchange.Search.Crawler.CrawlUser(Guid mailboxGuidToCrawl, ThreadLocalCrawlData crawlData, Boolean finalRetry)).

Since most of these functions rely on the System Attendant Mailbox to perform these functions, we opened the EMC and navigated to Disconnected Mailboxes. Low and behold, the SA mailbox was disconnected for whatever reason, possibly due to the failed SP1 installation.

To verify this, we opened up ADSIEDIT and navigated to the System Attendant object under this servers configuration object and found that HomeMDB was missing for the System Attendant.

Now to fix this, we added the correct DN of the First Storage Group\Mailbox Store which is where this mailbox normally resides to the System Attendant's objects HomeMDB attribute and saved it. After restarting the System Attendant service, Free/Busy, Content Indexing and the test-mapiconnectivity cmdlets started working again. Life was good once again.

Keep in mind that a lot of the server side functions rely on the System Attendant mailbox, so check that this mailbox is in tact and connected in your troubleshooting efforts.

-Mike

Outlook 2007 clients unable to perform Check name

mikelag — Thu, 13 Dec 2007 19:46:00 GMT

Yesterday,

I had an interesting issue where no Outlook 2007 clients were not able to perform a check name operation against any Exchange Server or Global Catalog server and we were continually prompted for authentication. Outlook 2003 clients and earlier did not have this problem. This environment was a simple setup with a single Exchange 2007 server hosting all roles on one Global Catalog server.

Went over the generics in http://support.microsoft.com/default.aspx?scid=kb;EN-US;297801, but that didn't provide any help. Oddly enough, Outlook 2007 was installed on the Exchange 2007 server as that in now currently supported and check name actually worked there.

So looking at this from the network side with a network trace, we see the following information being passed.

Example Server Names we are dealing with in these traces.
---------------------------
Exchange Server - EXSERVER
Global Catalog - GC.DOMAIN.COM

Netmon snippets
------------------------
Here we see the request for a kerberos ticket, but we are trying to connect to the Exchange server to do the address book lookups based on the exchangeAB/EXSERVER principal name.

2394 108.092812 000FEA71865E 0019B9B0A9C5 KERBEROS KRB_TGS_REQ 10.0.0.11 GC.DOMAIN.COM IP

KERBEROS: KRB_TGS_REQ
    KERBEROS: Protocol version number (pvno[1]) = 5 (0x5)
    KERBEROS: Message type (msg-type[2]) = KRB_TGS_REQ (0x0C)
    KERBEROS: Pre-authentication Data (padata[3])
        KERBEROS: Data type = PA-{AP|TGS}-REQ
            KERBEROS: Protocol version numer (pvno[0]) = 5 (0x5)
            KERBEROS: Message type (msg-type[1]) = 14 (0xE)
            KERBEROS: Ticket (ticket[3])
                KERBEROS: Ticket version number (tkt-vno[0]) = 5 (0x5)
                KERBEROS: Realm (realm[1]) =DOMAIN.COM
                KERBEROS: Server name (sname[2]) =krbtgt/DOMAIN.COM
                    KERBEROS: Principal name type (name-type[0]) = KRB_NT_SRV_INST (Service & other unique instance)
                    KERBEROS: Principal name value (name-string[1]) =krbtgt/DOMAIN.COM
                KERBEROS: Encrypted part (enc-part[3])
                    KERBEROS: Encryption type (etype[0]) = RC4-HMAC-NT
                    KERBEROS: Key version number (kvno[1]) = 2 (0x2)
                    KERBEROS: Ciphertext (cipher[2])
            KERBEROS: Authenticator (authenticator[4])
                KERBEROS: Encryption type (etype[0]) = RC4-HMAC-NT
                KERBEROS: Ciphertext (cipher[2])
    KERBEROS: Request body (req-body[4])
        KERBEROS: Realm (realm[2]) =DOMAIN.COM
        KERBEROS: Server name (sname[3]) =exchangeAB/EXSERVER
            KERBEROS: Principal name type (name-type[0]) = KRB_NT_SRV_INST (Service & other unique instance)
            KERBEROS: Principal name value (name-string[1]) =exchangeAB/EXSERVER

Here is the response back from the domain controller that we receive

2395 108.092812 0019B9B0A9C5 000FEA71865E KERBEROS KRB_TGS_REP GC.DOMAIN.COM 10.0.0.11 IP

KERBEROS: KRB_TGS_REP
    KERBEROS: Protocol version number (pvno[0]) = 5 (0x5)
    KERBEROS: Message type (msg-type[1]) = KRB_TGS_REP (0x0D)
    KERBEROS: Client realm (crealm[3]) =DOMAIN.COM
    KERBEROS: Client name (cname[4]) =Administrator
        KERBEROS: Principal name type (name-type[0]) = KRB_NT_PRINCIPAL (Name of Principal)
        KERBEROS: Principal name value (name-string[1]) =Administrator
    KERBEROS: Ticket (ticket[5])
        KERBEROS: Ticket version number (tkt-vno[0]) = 5 (0x5)
        KERBEROS: Realm (realm[1]) =DOMAIN.COM
        KERBEROS: Server name (sname[2]) =exchangeAB/EXSERVER
            KERBEROS: Principal name type (name-type[0]) = KRB_NT_SRV_INST (Service & other unique instance)
            KERBEROS: Principal name value (name-string[1]) =exchangeAB/EXSERVER

So again, we are trying to connect to the Exchange server to perform the lookups and since referrals were turned on, we couldn't lookup the users account in Active Directory because the Exchange server itself was not a domain controller.

What did happen on this server recently is that this server was demoted from being a Domain controller while Exchange 2007 server was on the server. During this demotion process, the ExchangeAB SPN entries were not removed from the Exchange server causing this problem. After removing these entries, Outlook was happy once again.

On a side note, Outlook 2007 has some new changes so that if Kerberos authentication fails, we no longer fall back to NTLM which is what the previous Outlook versions did.

The steps to fix this problem is listed in http://support.microsoft.com/?id=927612.

To help detect this in the future, an ExBPA bug has been submitted so that we can detect this now on the Exchange servers having these ExchangeAB SPN entries as ExBPA already detected these SPN's on the domain controllers.

-Mike