Written by Gerald Ramich, Senior Microsoft Premier Field Engineer.

This article is for folks who are trying to troubleshoot Microsoft Outlook Connectivity issues to Exchange servers. I’ll look at a wide number of troubleshooting items, including:

RPC through CAS for Mailbox Access: How does it work?
Troubleshooting Common RPC issues
Common Root Causes For Receiving the RPC Dialog Box
Verifying Exchange Client Access Ports
Troubleshooting Address Book Lookup or Check name Failures.
Troubleshooting Kerberos
Troubleshooting Networking Issues
Troubleshooting Connections to the Store
Troubleshooting Outlook 2007/2010 OOF, Free/Busy, OWA/ECP and OAB Links

So let’s dive in.

1.0 - RPC through CAS for Mailbox Access: How does it work?

After an Outlook Profile is created, Outlook needs to connect to the Mailbox and AD. Note Outlook 2007 and higher will connect to Autodiscover if it deems the profile needs to be changed.
When Outlook launches, it connects to an Endpoint Mapper (EPM) using port 135
Outlook 2003 and higher only. I am not covering older Outlook versions as these act differently.
Outlook has two other settings: “DS Server” and “Closest GC.” You should note that “Closest GC” is not supported in an Exchange 2010 environment, and “DS Server” is not recommended. Outlook needs to connect to the NSPI directory endpoint on CAS vs. some AD server. Features that break are delegate / multiple mailbox access for older Outlook clients (2003) and support for archived mailboxes.
Outlook queries for Three UUIDs (Universally Unique Identifiers):
MS Exchange RPC Client Access Services (a4f1db00-ca47-1067-b31f-00dd010662da and/or 5261574a-4572-206e-b268-6b199213b4e4) formally assigned to the Exchange Information store in legacy versions of Exchange
MS Exchange Address book Service (1544f5e0-613c-11d1-93df-00c04fd7bd09) for the MS Exchange RFR Interface
MS Exchange Address book Service(f5cc5a18-4264-101a-8c59-08002b2f8426) for the MS Exchange NSP Interface
The MAPI client has knowledge of the needed UUIDs, however it will not know the port number the server is listening on for each of these UUIDs since they are random at startup of the service.
The End Point Mapper returns the listening port number for each of the UUIDs on the CAS server.

2.0 - Troubleshooting Common RPC issues

2.1 - Common Root Causes For Receiving the RPC Dialog Box

High network latency
Loss of network connectivity on the client side
Loss of a network path within a network
Exchange server outages and crashes
Active Directory/Domain Controller outages and crashes
High database and/or log disk latencies
High server CPU and/or context switching
Long running MAPI operations

A few thoughts to keep in mind when debugging latency issues:

High disk latencies usually affect multiple users, not just single users
If max server latencies are high and cannot be explained by high disk latencies, also check for Jet Log Stalls and high server context switching or CPU usage.
Disconnects and reconnects always start with calls to Logon, so seeing lots of Logon calls from a particular user is a sign of connection problems (in addition to the RPC failures)
Outlook/COM add-ins, VBA code, and MAPI code running on the user’s workstation can cause problems that are intermixed with Outlook requests. Those functions may make expensive calls (like Findrow, SetColumns, SortTable, and IMAIL conversions). While Outlook from time to time makes expensive calls, 3^rd party applications are common culprits.

2.1.1 - Troubleshooting the RPC Dialog Box

First, determine if this issue is occurring for a single user or multiple users. Narrow down the client versions and locations if possible.
How long is the pop-up message on the screen? Is the Pop up on screen longer than 5-10 seconds for multiple clients? The pop up can be expected for short periods under even the best conditions. For single client issues it may not be server related. Reference: http://support.microsoft.com/kb/940226
Verify TCP Chimney is disabled on Exchange and GCs. This is required. Reference: http://support.microsoft.com/kb/948496
If clients are online, check critical folder size as this affects overall performance. The numbers are higher for Exchange 2007/2010, however cached mode is recommended.
Exchange 2007 information: http://technet.microsoft.com/en-us/library/cc535025(EXCHG.80).aspx
Outlook users experience poor performance when they work with a folder that contains many items on a server that is running Exchange Server: http://support.microsoft.com/kb/905803
Recommended Mailbox Size Limits – Misleading title, it’s really about Item counts. http://msexchangeteam.com/archive/2005/03/14/395229.aspx (impacts Outlook 2003 OSTs, newer versions are not affected)
How to troubleshoot the RPC Cancel Request dialog box in Outlook 2003 or in Outlook 2002: http://support.microsoft.com/kb/839862
Start Perfwiz on Exchange using all counters for at least 6 hours.
Use an Exchange Performance Troubleshooter (ExTRA) to get a snapshot of performance on Exchange.
Capture a concurrent client and server Netmon trace while reproducing the issue. The only way to track if this is a server or network related issue is to follow any delayed “Response” packets from server to the client.
Use PFDavAdmin to get mailbox item count if necessary.
Exmon can be used to determine various items such as cached or online, how many CPU cycles a user is using, etc. Yes, it has been updated for 2010! Place on Mailbox role…not CAS.

· Determine the suspected server from the RPC popup. The specific server will be listed and can include the user’s home Exchange server, an Exchange server the user was referred to for public folders, another user’s home Exchange server in the case of Calendar Details in the F/B UI, shared calendar/shared folder, or in delegate access, and Active Directory Servers.

· Collect Exmon ETL data via one of the supported methods. Consistent problems may need only 5 minutes worth of data collection to trace the event. It is important to trace for a period afterwards to allow collection and tracing by the Exchange server. Outlook buffers some monitoring data until its next server communication. Problems that happen sporadically may require multiple hours or days’ worth of collection. ETL file size and server impact is documented in the Frequently Asked Questions.

· Open the ETL data file with the Exmon tool.

· Verify that the user made RPC calls and those calls were traced. Find the user’s display name in the By User view. If the Exchange Server is Exchange 2003 or higher, verify that the IP address of the client appears as the IP address of the client machine in question. If the user’s display name does not appear in the By User view, an RPC call may have been issued and received by Exchange, but no successful Logon operation from that user was received (and thus could not be attributed to any user). Alternatively find the “” (BLANK) user name in the By User view and look for the user’s IP Address. If the IP Address appears in this list, an RPC was received by the Exchange server, but the Logon call failed.

· See if any MAPI operations took longer than 500 milliseconds. Within the By User View, the Max Server Latency will indicate the longest time spent processing a single MAPI operation, but an RPC could contain multiple operations.

· If the Max Server Latency is above 500 milliseconds, double click on the user’s name in the By User view. This will cause a reparse of the ETL file (which can take minutes for extremely large files) and will eventually display a detailed view of the user’s MAPI operations. Find the time frame in question (we have accuracy to about 15 milliseconds) in the By Time view. Verify if other operations took a long amount of time that would have been in the same packet or in packets within close range. It is prudent at this point to verify disk latencies are acceptable within the guidelines given in the Exchange Performance Tuning Whitepaper since the overall latency is determined both by the CPU and Store processing, as well as the timeliness of Jet database accesses by the disk subsystem.

· If roughly 5000 milliseconds cannot be accounted for, network latency may be involved. Check the By Clientmon view (if you’re using both Exchange 2003 and higher and Outlook 2003 and higher) for high max and/or average latencies. Using the By Clientmon view, find the user in the list and verify the user’s IP Address is in the list of IP Addresses. If the IP Address of the client is not in the user’s IP Address list, it is possible no client monitoring data was received. Check both the local and other average and max latencies. High average latencies could indicate an overall bad network condition. If the average is acceptable, the max latencies could be high on account of a momentary network issue or because of a long running MAPI operation. Remember, these latencies are the total round trip time of the packet including network transit and store latencies.

· If latencies are acceptable, check for failed RPC Packets. Failures happen from time to time and do not always indicate a problem, but are a useful step.

· Look out for IP Addresses reported in the By Clientmon view (IP addresses that Outlook thinks it is using based on the NIC/VPN) that differ from the IP Addresses in the By User view (IP address as seen by Store). Differences indicate some sort of proxy server or NAT. Client IP Addresses starting with 192.168.X.X are notoriously Wireless routers (but not a requirement nor definitive). These also indicate that the user may be using RPC/HTTP from a remote location.

2.2 - Verifying Exchange Client Access Ports

Verify TCP Ports on Exchange Server are listening using the RPCDump –i command. Below is example of what to look for.

Note: this is a truncated Output. Search for UUIDs. You will see these twice: once for Outlook Anywhere(RPC/HTTP) and once for regular RPC.

Connection-oriented TCP/IP using Microsoft Internet Information Server as HTTP proxy

LAB-E2K10-CSHT[6001] [5261574a-4572-206e-b268-6b199213b4e4] :ACCESS_DENIED

LAB-E2K10-CSHT[6001] [a4f1db00-ca47-1067-b31f-00dd010662da] :ACCESS_DENIED

LAB-E2K10-CSHT[6002] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft Exchange RFR Interface :ACCESS_DENIED

LAB-E2K10-CSHT[6004] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft Exchange RFR Interface :ACCESS_DENIED

LAB-E2K10-CSHT[6002] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :ACCESS_DENIED

LAB-E2K10-CSHT[6004] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :ACCESS_DENIED

Connection-oriented RPC TCP/IP

LAB-E2K10-CSHT[39627] [5261574a-4572-206e-b268-6b199213b4e4] :YES

LAB-E2K10-CSHT[39627] [a4f1db00-ca47-1067-b31f-00dd010662da] :YES

LAB-E2K10-CSHT[63534] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft Exchange RFR Interface :YES

LAB-E2K10-CSHT[63534] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :YES

A breakdown of example above:

Server: LAB-E2K10-CSHT

Port: [39627]

UUID: [a4f1db00-ca47-1067-b31f-00dd010662da]

Accessible: YES

Resource Kit tools: RPC Dump.

If one or more of these ports are not listening. You can use “Netstat –ano” and compare the ports that are listed in RPCDump to the PID that is listed in Netstat. Verify if another service has this port.

TCP 0.0.0.0:39627 0.0.0.0:0 LISTENING 2804 ß MSExchangeRPC

TCP 0.0.0.0:63534 0.0.0.0:0 LISTENING 5368 ßMSExchnageAB

Restarting the Information store will not re-register a stolen port, A restart is required to register TCP ports.

2.3 – Troubleshooting Address Book Lookup or Check name Failures.

Typically this error will resemble something like “The name could not be resolved. The name could not be matched to a name in the address list.”

Determine if the client is connecting to a GC or CAS in one of two ways: hold the Ctrl Key then “right click” the Outlook Icon on the Task Bar and choose Connection Status; and/or look at the Type Directory and look at the server Name.
Capture a trace from the client to see which GC/CAS we are trying to connect to.
If it is a GC, several things on the GC should be checked using DCDiag, NetDiag.
If it is a CAS, RPCDump can also show if F5CC (NT Directory NSPI) is listening
Also Verify the user is showing in the GAL and not hidden.
Verify Kerberos is working.

2.4 – Troubleshooting Kerberos

Netmon will show most Kerberos errors. Testing with NTLM in the Outlook profile under the “Security Tab” is also a good option to eliminate Kerberos issues. If Kerberos fails but NTLM auth works, Verify SPNs using SetSPN tool.

setspn -L ExchangeServerName

SPNs should be registered as follows on Exchange Server:

http/ For Exchange Web Services and the Autodiscover service
exchangeMDB/ For RPC Client Access
exchangeRFR/ For the Address Book service
exchangeAB/ For the Address Book service

Note: Load Balancers require the Alternative Service Account and SPN registered to the Load balancer FQDN instead of the individual server names.

Note: SPNs could be registered as follows pointing to GCs on Exchange2003/ 2007 servers, this should not be done on Exchange 2010: exchangeAB/<GlobalCatalogServerName>

Once SPNs are verified, I recommend this whitepaper: Troubleshooting Kerberos Errors

2.5 - Troubleshooting Networking Issues

Capture concurrent Netmon from client and server(s) affected.
Look for RPC Fault, dropped packets, TCP retransmit.
Devices can cause several connection issues. For example: context 0x0 status 0x1C00001A errors are typically a device issue, as outlined in this MSDN article.
Don’t forget Chimney/TCP Offloading can cause connectivity failures. Check NIC drivers, these need to be up to date.
Firewall between client and servers. It’s necessary to insure all listed ports open for Exchange 2010.
Check out the List of Extended MAPI numeric result codes in this KB article.

2.6 - Troubleshooting Connection to the Store

This error will typically show up as “Unable to open your default mail folders. The information store could not be opened.”

The netmon trace should show RPC Fault and then the corresponding error may indicate “Access Denied.”.
There can be several causes for this. In some cases it may be as simple as the “access this computer from network right”.
Dump both AD Permissions and Exchange Permissions for extended rights, as follows:
Get-ADPermissions MailboxAlias | where [($_.ExtendedRights –Like “*-as*”)} |ft User,ExtendedRights,Deny –Auto (This finds all Receive-as / Send-as extended permissions)
Get-MailboxPermission MailboxAlias –User <person checking access for> |ft User,AccessRights, Deny –Auto

On the get-mailboxpermission you could add | where [($_.AccessRights –Like “*full*”)}

3.0 - Troubleshooting Outlook 2007/2010 OOF, Free/Busy, OWA/ECP and OAB Links

3.1 - HTTP troubleshooting

These steps are true for any HTTP application.
IIS Logs and Status Codes are you friends. The following KB article points to common causes for most of the codes, so turn up protocol logging in IIS: HTTP status codes in IIS 7.0
For HTTP status code definitions, visit the World Wide Web Consortium (W3C) Web site:
High level Codes

a. 1XX – Informational

b. 2XX - Success

c. 3XX - Redirection

d. 4XX - Client Error

e. 5XX - Server Error

5. Mainly you will have to focus on the 4XX and 5XX codes.

6. 4XX Codes have Sub codes to further describe the issue, as follows:

a. 400 – Bad Request

i. 400.1 - Invalid Destination Header.

ii. 400.2 - Invalid Depth Header.

iii. 400.3 - Invalid If Header.

iv. 400.4 - Invalid Overwrite Header.

v. 400.5 - Invalid Translate Header.

vi. 400.6 - Invalid Request Body.

vii. 400.7 - Invalid Content Length.

viii. 400.8 - Invalid Timeout.

ix. 400.9 - Invalid Lock Token.

b. 401 – Access Denied (logon issues)

i. 401.1 - Logon failed.

ii. 401.2 - Logon failed due to server configuration.

iii. 401.3 - Unauthorized due to ACL on resource.

iv. 401.4 - Authorization failed by filter.

v. 401.5 - Authorization failed by ISAPI/CGI application.

c. 403 – Forbidden (Access Restrictions)

i. 403.1 - Execute access forbidden.

ii. 403.2 - Read access forbidden.

iii. 403.3 - Write access forbidden.

iv. 403.4 - SSL required.

v. 403.5 - SSL 128 required.

vi. 403.6 - IP address rejected.

vii. 403.7 - Client certificate required.

viii. 403.8 - Site access denied.

ix. 403.9 - Forbidden: Too many clients are trying to connect to the Web server.

x. 403.10 - Forbidden: Web server is configured to deny Execute access.

xi. 403.11 - Forbidden: Password has been changed.

xii. 403.12 - Mapper denied access.

xiii. 403.13 - Client certificate revoked.

xiv. 403.14 - Directory listing denied.

xv. 403.15 - Forbidden: Client access licenses have exceeded limits on the Web server.

xvi. 403.16 - Client certificate is untrusted or invalid.

xvii. 403.17 - Client certificate has expired or is not yet valid.

xviii. 403.18 - Cannot execute requested URL in the current application pool.

xix. 403.19 - Cannot execute CGI applications for the client in this application pool.

xx. 403.20 - Forbidden: Passport logon failed.

xxi. 403.21 - Forbidden: Source access denied.

xxii. 403.22 - Forbidden: Infinite depth is denied.

d. 404 – Not Found

i. 404.0 - Not found.

ii. 404.1 - Site Not Found.

iii. 404.2 - ISAPI or CGI restriction.

iv. 404.3 - MIME type restriction.

v. 404.4 - No handler configured.

vi. 404.5 - Denied by request filtering configuration.

vii. 404.6 - Verb denied.

viii. 404.7 - File extension denied.

ix. 404.8 - Hidden namespace.

x. 404.9 - File attribute hidden.

xi. 404.10 - Request header too long.

xii. 404.11 - Request contains double escape sequence.

xiii. 404.12 - Request contains high-bit characters.

xiv. 404.13 - Content length too large.

xv. 404.14 - Request URL too long.

xvi. 404.15 - Query string too long.

xvii. 404.16 - DAV request sent to the static file handler.

xviii. 404.17 - Dynamic content mapped to the static file handler via a wildcard MIME mapping.

xix. 404.18 - Querystring sequence denied.

xx. 404.19 - Denied by filtering rule.

e. 405 – Method Not allowed

f. 406 – Client browser does not accept Mime Type Request page

g. 408 – Request timed out

h. 412 – Precondition Failed.

7. The Sub number will be in one of two formats

a. 401.1 – Displayed in browser (note: for IIS 7+, the substatus code is only visible from the server console, by default. For security, remote clients are only given the basic 3-digit HTTP status code)

b. 401 1 – Displayed IIS logs at end of string, as long as http substatus logging is enabled (it is by default)

8. 5XX Codes have Sub codes to further describe the issue, as follows:

a. 500 – Internal Server error

i. 500.0 - Module or ISAPI error occurred.

ii. 500.11 - Application is shutting down on the Web server.

iii. 500.12 - Application is busy restarting on the Web server.

iv. 500.13 - Web server is too busy.

v. 500.15 - Direct requests for Global.asax are not allowed.

vi. 500.19 - Configuration data is invalid.

vii. 500.21 - Module not recognized.

viii. 500.22 - An ASP.NET httpModules configuration does not apply in Managed Pipeline mode.

ix. 500.23 - An ASP.NET httpHandlers configuration does not apply in Managed Pipeline mode.

x. 500.24 - An ASP.NET impersonation configuration does not apply in Managed Pipeline mode.

xi. 500.50 - A rewrite error occurred during RQ_BEGIN_REQUEST notification handling. A configuration or inbound rule execution error occurred.

xii. Note Here is where the distributed rules configuration is read for both inbound and outbound rules.

xiii. 500.51 - A rewrite error occurred during GL_PRE_BEGIN_REQUEST notification handling. A global configuration or global rule execution error occurred.

xiv. Note Here is where the global rules configuration is read.

xv. 500.52 - A rewrite error occurred during RQ_SEND_RESPONSE notification handling. An outbound rule execution occurred.

xvi. 500.53 - A rewrite error occurred during RQ_RELEASE_REQUEST_STATE notification handling. An outbound rule execution error occurred. The rule is configured to be executed before the output user cache gets updated.

xvii. 500.100 - Internal ASP error

b. 501 – Header values specify a configuration that is not implemented

c. 502 – Web Server received an invalid response while acting as a gateway or proxy

i. 502.1 - CGI application timeout.

ii. 502.2 - Bad gateway.

d. 503 – Service unavailable

i. 503.0 - Application pool unavailable.

ii. 503.2 - Concurrent request limit exceeded.

9. The Sub number will be in one of two formats

a. 500.0 – Displayed in browser on server console only (see above)

b. 500 0 – Displayed IIS logs at end of string, if http substatus logging is enabled (on by default)

10. So what logging can I turn up outside of IIS?

a. Diags Logging in Exchange

i. MSExchange AutoDiscover

ii. MSExchange Availability (EWS/OOF/AS), Calendar and Free / Busy

iii. MSExchange Control Panel (Option Page in OWA). Outlook 2010 will use the ECP for some features.

iv. MSExchange WebServices (RPC/HTTP, also EWS/OOF/AS)

b. Test Command lets

i. Test-OutlookWebServices (AutoDiscover)

ii. Test-CalendarConnectivity (AS)

1. Only Anonymous and not very useful.

2. Use URL in Apps events

iii. Test-ECPConnectivity

iv. Test-WebServicesConnectivity(RPC/HTTP access)

1. Note: It does not test Calendar, OOF or ECP!

2. Test-OutlookConnectivity (AutoDiscover, Profile Creation, MAPI or RPC/HTTP access)

Troubleshooting Microsoft Exchange: Outlook Connectivity Issues