A quick and easy check you can do to ensure your O365 connections complete quickly is to check proxy authentication is completing quickly, or better still not being done at all. If you're not using a proxy and are going direct, then you can move along…nothing to see here!

It's surprisingly common and I've run into numerous customers experiencing unnecessary delays on connecting to O365, caused by proxy authentication. With Outlook this can cause a delay on start-up, or the 'polo mint' hang when switching mailboxes/calendars etc, anything that requires a new TCP connection to be spun up. With SharePoint this will manifest itself as slow initial page loads.

To view this proxy authentication stage of a TCP connection to Office 365 (if indeed there is one) I'd recommend using a packet capture tool on the client, such as Netmon or Wireshark. If enabled, proxy authentication needs to occur with every TCP session setup and is the first thing which has to complete after the TCP 3 way handshake, usually triggered with the first GET or CONNECT request. We should expect this process to complete in milliseconds. This is what it looks like in Netmon:

In Netmon it's wise to add the 'NTLM SSP Summary' column to show what stage of authentication we're at, also add the "Time Delta" column to show you the time delay from the previous packet shown.

The easiest way to find the session is to

  • Close all browser windows and open a single one on a new tab.
  • For Outlook close the application completely
  • Start Netmon
  • Connect to your SharePoint page or start Outlook
  • On the left hand side, Netmon should show your browser (or Outlook) with its TCP connections to your site as follows:

 

For each of these examples, you'll see multiple TCP sessions per process, each will perform proxy authentication if enabled. The number will differ on the version of Outlook or browser you are using.

Initially we'll connect with no authentication and be told 'proxy authentication required'. In this example response in this instance takes 0.02 seconds to come back, indicating no network performance issue and no proxy performance issue per-se.

Here is an example of this problem occurring. Showing the stages following the TCP 3-way handshake. To just see these packets and ignore the pure TCP ones, use the filter 'HTTP'.

 

Initial connect:

14:12:24.6483418 19.0046514 0.0003578 iexplore.exe 10.200.30.40 MyProxy-01.Contoso.sig HTTP:Request, CONNECT Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443 , Using NTLM Authorization NTLM NEGOTIATE MESSAGE

Proxy Response:

14:12:24.6876389 19.0439485 0.0283000 iexplore.exe MyProxy-01.Contoso.sig 10.200.30.40 HTTP:Response, HTTP/1.1, Status: Proxy authentication required, URL: Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443 NTLM CHALLENGE MESSAGE

We then send the request again, this time with NTLM authentication for the proxy as requested:

Second request with NTLM Auth:

14:12:24.6883198 19.0446294 0.0004838 iexplore.exe 10.200.30.40 MyProxy-01.Contoso.sig HTTP HTTP:Request, CONNECT Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443 , Using NTLM Authorization NTLM AUTHENTICATE MESSAGE Version:NTLM v2, Domain: headoffdom, User: paul.collinge, Workstation: W7TEST20

200 OK response from proxy but this takes 3 seconds.

14:12:27.7859643 22.1422739 3.0878394 iexplore.exe MyProxy-01.Contoso.sig 10.200.30.40 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443

Subsequent call:

Once the above is complete, subsequent calls are back to millisecond response times.

14:12:27.7868062 22.1431158 0.0008419 iexplore.exe 10.200.30.40 MyProxy-01.Contoso.sig TLS TLS:TLS Rec Layer-1 HandShake: Client Hello.

14:12:27.8445642 22.2008738 0.0485304 iexplore.exe MyProxy-01.Contoso.sig 10.200.30.40 TLS TLS:TLS Rec Layer-1 HandShake: Server Hello.; TLS Rec Layer-2 Cipher Change Spec; TLS Rec Layer-3 HandShake: Encrypted Handshake Message

 

As the delay is only seen when performing the proxy authentication stage of the session setup , this indicates the delay is caused by the process of authentication itself.

As we're using NTLM in this example and in this instance the Proxy is in a different domain to the users, it's feasible this delay could be caused by congestion on the secure channels between the proxy and it's DC, or its DC to the users DC.

This would have to be investigated separately to see if it this, or something else is causing the delay but here is some information on this and the fix (maxconcurrentapi registry key).

http://blogs.technet.com/b/ad/archive/2008/09/23/ntlm-and-maxconcurrentapi-concerns.aspx

http://support.microsoft.com/kb/975363

Regardless of the root cause, this behaviour, if occurring will be causing intermittent performance issues with both Office 365 and Internet browsing. When the above is the cause, you may find that the response times are good at times and very bad at others. The slow times seemed may coincide with high utilisation times, such as first time in the morning and after lunch and as such you should run these tests at various times of the day.

The issue could also be occurring due to loading issues on the proxy itself, however, delays would be apparent in more than just the authentication packets.

 

From an Office 365 perspective, we can easily remove this problem by following the recommended setup and making an exception in the proxy for authentication on the Office 365 urls as per: http://support.microsoft.com/kb/2637629

 

Firewall or proxy servers require additional authentication

To resolve this issue, configure an exception for Microsoft Office 365 URLs and applications from the authentication proxy. For example, if you are running Microsoft Internet Security and Acceleration Server (ISA) 2006, create an "allow" rule that meets the following criteria:

  • Allow outbound connections to the following destination: *.microsoftonline.com
  • Allow outbound connections to the following destination: *.microsoftonline-p.com
  • Allow outbound connections to the following destination: *.sharepoint.com
  • Allow outbound connections to the following destination: *.outlook.com
  • Allow outbound connections to the following destination: *.lync.com
  • Allow outbound connections to the following destination: osub.microsoft.com
  • Ports 80/443
  • Protocols TCP and HTTPS
  • Rule must apply to all users.

HTTPS/SSL time-out set to 8 hours

With these bypassed, we've removed one possible cause of a delay on your Office 365 connections, and also taken away some load from both your proxies, and your DCs.

So in summary, if you're using a proxy, ensure there is no authentication performed on the TCP session out to Office 365 by whitelisting the URLs above. If you must use proxy auth, make sure it's completing quickly, especially at peak times. It should complete in milliseconds, i.e. not much more than the time between the initial SYN and SYN ACK. Even a delay as small as 2-3 seconds like the one demonstrated, will have a noticeable impact for your users.