Over the past 18 months I have been dealing with more and more cloud engagements. One thing that crops up during the planning phase of all of these engagements is network link latency and bandwidth requirements.
Bandwidth prediction is something I am working on in the background, but network link latency seemed like a mystery!. Everywhere I looked and everyone I spoke to had a different opinion on “maximum latency” for Outlook clients.
So.. I decided to try to devise a test that would let me model the impact of network link latency on Outlook client performance. My lab already had a network emulator installed that was capable of simulating link latency so all that I needed was a script to automate Outlook to perform some common tasks, then record how long the script took to run – easy right?
I already had some scripts for Outlook 2010 from my bandwidth prediction work, so I decided to make use of them. The scripts were written with AutoIT which is a regular part of my lab testing toolkit these days!
Software versions in use for all tests…
Note: All tests began from a cold start i.e Outlook was not running.
Note: All clients were tested from 0ms added latency to 1000ms in steps of 100ms and then up to failure point from 1000ms onwards in steps of 500ms. Failure was deemed to be if any single test failed.
Test 1. Sending
Test 2. Opening
Test 3. Delegates
Test 4. Out of Office
Test 5. Calendar
I decided to run tests 1-5 sequentially for three types of client connections
Note: All latency values represented are round-trip times RTT as reported by ping.exe from a command line.
What is evident from this set of results is that the impact of network link latency is linearly related to outlook client performance. Certainly when we consider a broad range of functions. Cached mode dramatically reduces the user wait time for send/receive functions, but it was obvious watching the test run that whenever actions were requested that ran outside of the OST file, such as delegate access or out-of-office requests large delays were experienced at higher latency values. I decided to set the maximum test run time limit at 200 seconds, which was double the test run time with no additional latency applied. Client behaviour when the test duration was taking 200 seconds was the poorest performance level that I perceived to be acceptable by anyone for this test. I judged that if any single operation took 10 seconds or more to display a dialog box or the client appeared to hang then it was unacceptable for the end user.
In this combined test the limiting task was working with the delegate permissions, which was taking approximately 9 seconds to complete at the maximum test run time. Accessing delegate permissions was similar amongst all connection modes.
The results for Outlook Anywhere were confusing initially, although the majority of the difference between Cached and Cached-OA is due to the additional time it takes for an Outlook Anywhere connected client to establish its initial connection. Typically an OA client will take around 10 seconds longer to establish a connection when compared to a normal RPC MAPI connection. This rises to around 30 seconds at 800ms latency. Under use both Outlook Anywhere and Cached mode performed with very similar results.
The test results for me were quite interesting. Prior to this testing I had assumed that Outlook tolerance for link latency was was much worse than the test results show. This is no doubt due to changes in adaptive TCP windowing on Windows 7 plus improvements in Outlook 2010. The bottom line is clear though, Outlook 2010 on Windows 7 is able to tolerate quite high network link latency values before performance begins to suffer noticeably. In the case of Cached mode and Outlook Anywhere both clients were able to tolerate 500ms round-trip-time latency without problem and if your main tasks are sending and receiving mail only, cached mode actually provided a good experience at up to 1000ms round-trip-time latency. Online mode was far more sensitive to network link latency as expected and performance became noticeably slower at 190ms round-trip-time and above. Online mode was still usable above 200ms latency but felt sluggish.
One thing that I didn't account for during this test was that the clients would behave differently as latency increase. My automated script initially didn't cope very well when Outlook in online mode was subjected to high latency values. The long pauses before dialog boxes were displayed made things quite tricky at times, AutoIT is extremely capable though which really helped in this testing!.
The following failure points were noted..
Note: Although the cached mode clients failed at 2000ms for delegate access the client continued to provide usable mail performance. I plan to run further tests in cached mode to see where the break point is for mail only users since this represents the majority of users. Keep watching my blog for more information on this test
This test was designed to isolate the effects of network link latency on client performance, however the reality is that network link latency is only one part of the Outlook client performance experience. In practice it is more likely that poor client performance is caused by local hardware limitations or insufficient resources allocated on the Exchange Server itself. This testing does show however, that consolidating Exchange Server resources into central locations is viable with Windows 7 and Outlook 2010.
So.. this is an interesting question that came to mind while I was performing this testing. We can provide a good user experience in normal day to day tasks over quite latent network links, but what happens if I need to rebuild my OST file? OST file resync is a fairly common occurrence; even if we just consider cases of SATA HDD failure (5% AFR typically), that means in an organisation of 30,000 laptop users there will be approximately 1500 OST resync’s per year (7.5 per working day!) due to HDD failure. I was curious about how long it would take to synchronise an OST file under specific latency conditions, so I ran some tests….
I created a 100MB mailbox then timed how long that took to synchronise down as latency increased, I then calculated how many seconds it took per 1MB of data. My plan was to use this to extrapolate how long larger mailboxes would take to synchronise down.
Note: I had initially planned on doing this experimentally but then the reality of having to run 10 or more tests each taking 20 hours or longer to complete set in and I decided to deal with it theoretically. I may run a few tests to validate the theory when i get time though.
Using this data I was able to extrapolate how long it would take to fully synchronise down a 25GB mailbox (default Office 365 quota limit). Given the assumption that there is always sufficient network bandwidth available, the data looks as follows…
This data suggests that even if a cached mode client were running over a 500ms round-trip-time network latency connection that it would be possible to fully resynchronise their 25GB OST file within 43 Hours. This would require a throughput of 1.34Mbits/sec.
One of the customers I am working with is considering a 5GB quota within their Office 365 tenant so I promised them I would consider their case specifically, so here is the data for a 5GB OST sync.
Taking the same point of 500ms, this prediction suggests that their 5GB OST file would be fully synchronised after 8.6 Hours, assuming that the client had 1.33Mbits/sec of network bandwidth available.
To perform your own predictions using this data use the following form…
T = ( M ) x (( 0.0114 x L ) + 0.3208 )
This testing sparked a few thoughts in my mind, that I just didn't have time to follow up (and I was being harassed by various colleagues and customers to release the data i had!) … anyway, the following are areas i plan to research further when time allows…
For me this test data suggests that network latency is not as critical for client performance on Windows 7 with Outlook 2010 cached mode as it used to be on legacy operating systems. It also shows that the adaptive TCP windowing in Windows 7 goes some way to alleviate the negative effects of network latency. Certainly for my current projects it has allowed me to consolidate more mailboxes into a central location than i would have initially thought possible and the client involved tells me that end user performance is good.
If you are considering running Outlook clients over high latency connections I would urge you to perform a full user pilot to establish your users tolerance of client performance. Also some Outlook plugins can have a dramatic effect on how the client performs under very latent conditions.
Terrific article, Neil! Thanks for sharing your test results.
I'd be curious to know what the results look like for Outlook 2007, since I run into quite a few clients who have not upgraded yet and are looking for a business reason to do so.
I too am curious about this and for legacy operating systems. If I find time I will definitely look to analyse some legacy technology also... the trouble with this kind of testing is that it takes lots of lab time to prepare and perform the testing. I usually try to fit this stuff in between customer engagements, but we are frantically busy at the moment so unless an engagement I am working on needs the data its difficult to make the time to do it.. still its definitely something i want to analyse further when i get the time.
Hi, for your info, I did some tests before we migrated to exch2010 and compared Outlook2007 and 2010.
And for me, 1gb mailbox took about 15min to sync on 2010 (with ~ 45ms latency) compared to 45min on Outlook 2007 (both in cache mode).
Great post Neil, thanks for putting in the hard work needed for this and doing the write up!
Very Nice Job Neil :)
This is excellent, Would you be able to share you scripts? Most of the estate I am in at the moment in still XP and Outlook 2007. It would be good to try and do a comparison
I am happy to share, but to be honest they are not much use since I resorted to using mouseclick() for most things since it seemed more reliable than keyboard shortcuts when the client started to run slowly... and I didnt have Outlook maximised so the scripts are very tightly integrated to a single test machine in my lab! :(
Its on my "to do" list to repeat this on Outlook 2007 though... hopefully this week
Excellent Article! Thanks for sharing this Neil!
Hi Neil. Thanks for the article. Did you ever do the comparison between Outlook 2007 vs Outlook 2010 to compare performance? My organisation is also looking to deploy Outlook 2010 to a smaller set of high latency site to aid in perceived performance issues, but I'm being pushed to show some evidence that O2010 actually performs better on high latency links than O2003 or O2007?
Hi Neil, great article! Is there by chance a test conducted against O365 (MS cloud exchange) with the same parameters? Would you consider there to be any variance of your findings above with anything unique with respect to O365? We're trying to determine how to optimize our O365 deployment and would like to use your study as a benchmark. Thanks!