I lead the engineering organization responsible for BPOS.  My team builds, operates and supports our BPOS service, and over the last few days, we have not satisfied our customer’s needs.  On Tuesday and today we experienced three separate service issues that impacted customers served from our Americas data center.  All of these issues have been resolved and the service is now running smoothly. These incidents were unique to BPOS and not related to Office 365 or any other Microsoft services.

I’d like to apologize to you, our customers and partners, for the obvious inconveniences these issues caused.  We know that email is a critical part of your business communication, and my team and I fully recognize our responsibility as your partner and service provider. We will provide a full post mortem, and will also provide additional updates on how our service level agreement (SLA) was impacted.   We will be proactively issuing a service credit to our impacted customers.

I also want to provide more detail about the recent issues.

On Tuesday at 9:30am PDT, the BPOS-S Exchange service experienced an issue with one of the hub components due to malformed email traffic on the service.   Exchange has the built-in capability to handle such traffic, but encountered an obscure case where that capability did not work correctly.  The result was a growing backlog of email.  By 12:00pm PDT, the malformed traffic was isolated and the mail queues cleared.  The delays encountered by customers varied, on the order of 6-9 hours.   Short term mitigation was implemented and a fix was under development.

At 9:10am PDT today, service monitoring again detected malformed email traffic on the service.   The problem was resolved at 10:03am, but users experienced up to 45 minute email delays during this time.   A second, but related issue was detected via monitoring at 11:35am PDT, resulting in email stuck in some end users’ outboxes. The issue was remediated at 12:04pm PDT. During this time, more than 1.5 million messages had queued on the service awaiting delivery.   The backlog was 90% clear by 4:12 PM, but because of this large backlog of email, customers may have experienced delays of as long as 3 hours.   We are implementing a comprehensive fix to both problems.  

As a result of Tuesday’s incident, we feel we could have communicated earlier and been more specific.  Effective today, we updated our communications procedures to be more extensive and timely.   We understand that it is critical for our customers to be as fully informed as possible during service impacting events.  We will continue to improve the timeliness and specificity of our communications.  The primary mechanism for communicating to our customers on issues has been and will continue to be the Service Health Dashboard.  For North America, that dashboard is at https://health.noam.microsoftonline.com/.

 In an unrelated incident, starting at 1:04am PDT, service monitoring detected a failure in the Domain Name Service (DNS) hosting the http://mail.microsoftonline.com domain.  This failure, prevented users from accessing Outlook Web Access hosted in the Americas, and partially impacted some functionality of Microsoft Outlook and Microsoft Exchange ActiveSync devices.  The team diagnosed, and fixed, an underlying problem in the servers hosting Domain Name Service (DNS) for the http://mail.microsoftonline.com domain, and restored service at 4:52am PDT.  The team identified a number of improvements in our handling of problems associated with DNS, and will provide a full post mortem of this incident available through Microsoft Support.

 As I’ve said before, all of us in the BPOS team and at Microsoft appreciate the serious responsibility we have as a service provider to you, and we know that any issue with the service is a disruption to your business – that’s not acceptable.  I want to assure you that we are investing the time and resources required to ensure we are living up to your – and our own – expectations for a quality service experience every day.

As always, if you are experiencing any service issues, we encourage customers to check the Service Health Dashboard for the latest information or contact our customer support team. Our customer support is available 24 hours a day by telephone or via Service Requests submitted from the Microsoft Online Services Administration Center.

May 17, 2011 Update:
I wanted to thank you for taking the time to read the comments, and though JRG has been responding to many of them, I wanted to address some of the common themes:

• Thanks for the feedback on the dashboard. I know that we have not been timely enough with information. While it can be difficult to trade off timeliness against certainty and accuracy, we know it is critical and we have an ongoing initiative to track and improve communications performance. We specifically failed to provide adequate updates last Tuesday, with the first update about 90 minutes into the incident. I commit that we will work harder to improve our communications.
  
  Many of you have argued that the health dashboard should not require a login. Let me explain why we designed it that way. While today’s health dashboard provides service-wide information, our ultimate goal is to provide a customer-specific view of service health. To provide information that is tailored and appropriate for both end-users and administrators, authentication is required. This updated dashboard which provides customer specific information is under development right now and will be available in 6 – 8 weeks. As with the dashboard today, there is no special password required – any BPOS user or administrator can access it. Finally, we’re increasing the timeout period so the dashboard won’t log you out as quickly.
  
  
• While our phone support is available 24x7, our system was overloaded on Tuesday and Thursday. As well, there was a problem with the voice message issued on Tuesday (corrected on Thursday). We are adding more phone lines as a result of this overload. But a better fix is to release more and timely information on the dashboard. In the rare (though not rare enough last week!) case of a broad outage, phone support is not a truly scalable way of providing information, so we are working diligently to improve our communications procedures and deploy the customer-specific dashboard.
  
  
• To enable faster recovery time from backlogged message queues, we have already added and continue to add more hub server capacity and have deployed a software update to all datacenters to improve how BPOS handles malformed email messages.

I want to again sincerely apologize for the inconvenience last week’s issues caused you, our customers and partners.

Dave Thompson

Corporate Vice-President, Microsoft Online Services