• Concerning Trends Discovered During Several Critical Escalations

    Over the last several months, I have been involved in several critical customer escalations (what we refer to as critsits) for Exchange 2010 and Exchange 2013. As a result of my involvement, I have noticed several common themes and trends. The intent of this blog post is to describe some of these common issues and problems, and hopefully this post will lead you to come to the same conclusion that I have – that many of these issues could have been avoided by taking sensible, proactive steps.

    Software Patching

    By far, the most common issue was that almost every customer was running out-of-date software. This included OS patches, Exchange patches, Outlook client patches, drivers, and firmware. One might think that being out-of-date is not such a bad thing, but in almost every case, the customer was experiencing known issues that were resolved in current releases. Maintaining currency also ensures an environment is protected from known security defects. In addition, as the software version ages, it eventually goes out of support (e.g., Exchange Server 2010 Service Pack 2).

    Software patching is not simply an issue for Microsoft software. You must also ensure that all inter-dependent solutions (e.g., Blackberry Enterprise Server, backup software, etc.) are kept up-to-date for a specific release as this ensures optimal reliability and compatibility.

    Microsoft recommends adopting a software update strategy that ensures all software follows N to N-1 policy, where N is a service pack, update rollup, cumulative update, maintenance release, or whatever terminology is used by the software vendor. We strongly recommend that our customers also adopt a similar strategy with respect to hardware firmware and drivers ensuring that network cards, BIOS, and storage controllers/interfaces are kept up to date.

    Customers must also follow the software vendor’s Software Lifecycle and appropriately plan on upgrading to a supported version in the event that support for a specific version is about to expire or is already out of support.

    For Exchange 2010, this means having all servers deployed with Service Pack 3 and either Rollup 7 or Rollup 8 (at the time of this writing). For Exchange 2013, this means having all servers deployed with Cumulative Update 6 or Cumulative Update 7 (at the time of this writing).

    For environments that have a hybrid configuration with Office 365, the servers participating in the hybrid configuration must be running the latest version (e.g., Exchange 2010 SP3 RU8 or Exchange 2013 CU7) or the prior version (e.g., Exchange 2010 SP3 RU7 or Exchange 2013 CU6) in order to maintain and ensure compatibility with Office 365. There are some required dependencies for hybrid deployments, so it’s even more critical you keep your software up to date if you choose to go hybrid.

    Change Control

    Change control is a critical process that is used to ensure an environment remains healthy. Change control enables you to build a process by which you can identify, approve, and reject proposed changes. It also provides a means by which you can develop a historical accounting of changes that occur. Often times I find that customers only leverage a change control process for “big ticket” items, and forego the change control process for what are deemed as “simple changes.”

    In addition to building a change control process, it is also critical to ensure that all proposed changes are vetted in a lab environment that closely mirrors production, and includes any 3rdparty applications you have integrated (the number of times I have seen Exchange get updated and heard the integrated app has failed is non-zero, to use a developer’s phrase).

    While lab environments provide a great means to validate the functionality of a proposed change, they often do not provide a view on the scalability impact of a change. One way to address this is to leverage a “slice in production” where a change is deployed to a subset of the user population. This subset of the user population can be isolated using a variety of means, depending on the technology (e.g., dedicated forests, dedicated hardware, etc.). Within Office 365, we use slices in productions a variety of different ways; for example, we leverage them to test (or what we call dogfood) new functionality prior to customer release and we use it as a First Release mechanism so that customers can experience new functionality prior to worldwide deployment.

    If you can’t build a scale impact lab, you should at a minimum build an environment that includes all of the component pieces you have in place, and make sure you keep it updated so you can validate changes within your core usage scenarios.

    The other common theme I saw is bundling multiple changes together in a single change control request. While bundling multiple changes together may seem innocuous, when you are troubleshooting an issue, the last thing you want to do is make multiple changes. First, if the issue gets resolved, you do not know which particular change resolved the issue. Second, it is entirely possible the changes may exacerbate the current issue.

    Complexity

    Failure happens. There is no technology that can change that fact. Disks, servers, racks, network appliances, cables, power substations, pumps, generators, operating systems, applications, drivers, and other services – there is simply no part of an IT service that is not subject to failure.

    This is why we use built-in redundancy to mitigate failures. Where one entity is likely to fail, two or more entities are used. This pattern can be observed in Web server arrays, disk arrays, front-end and back-end pools, and the like. But redundancy can be prohibitively expensive (as a simple multiplication of cost). For example, the cost and complexity of the SAN-based storage system that was at the heart of Exchange until the 2007 release, drove the Exchange Team to evolve Exchange to integrate key elements of storage directly into its architecture. Every SAN system and every disk will ultimately fail, and implementing a highly-redundant system using SAN technology is cost-prohibitive, so Exchange evolved from requiring expensive, scaled-up, high-performance storage systems, to being optimized for commodity scaled-out servers with commodity low-performance SAS/SATA drives in a JBOD configuration with commodity disk controllers. This architecture enables Exchange to be resilient to any storage failure.

    By building a replication architecture into Exchange and optimizing Exchange for commodity hardware, failure modes are predictable from a hardware perspective, and that redundancy can removed from other hardware layers, as well. Redundant NICs, redundant power supplies, etc., can also be removed from the server hardware. Whether it is a disk, a controller, or a motherboard that fails, the end result is the same: another database copy is activated on another server.

    The more complex the hardware or software architecture, the more unpredictable failure events can be. Managing failure at scale requires making recovery predictable, which drives the necessity for predictable failure modes. Examples of complex redundancy are active/passive network appliance pairs, aggregation points on a network with complex routing configurations, network teaming, RAID, multiple fiber pathways, and so forth.

    Removing complex redundancy seems counter-intuitive – how can removing hardware redundancy increase availability? Moving away from complex redundancy models to a software-based redundancy model creates a predictable failure mode.

    Several of my critsit escalations involved customers with complex architectures where components within the architecture were part of the systemic issue trying to be resolved:

    1. Load balancers were not configured to use round robin or least connection management for Exchange 2013. Customers that did implement least connection management, did not have the “slow start” feature enabled. Slow start ensures that when a server is returned to a load-balanced pool, it is not immediately flooded with connections. Instead, the connections are slowly ramped up on that server. If your load balancer does not provide a slow start function for least connection management, we strongly recommend using round robin connection management.
    2. Hypervisor hosts were not configured in accordance with vendor recommendations for large socket/pCPU machines.
    3. Firewalls between Exchange servers, Active Directory servers, or Lync servers. As discussed in Exchange, Firewalls, and Support…Oh, my!, Microsoft does not support configurations when Exchange servers have network port restrictions that interfere with communicating with other Exchange servers, Active Directory servers, or Lync servers.
    4. Ensuring the correct file-based anti-virus exclusions are in place.
    5. Deploying asymmetric designs in a “failover datacenter.” In all instances, there were fewer servers in the failover datacenter than the primary datacenter. The logic used in designing these architectures was that the failover datacenter would only be used during maintenance activities or during catastrophic events. The fundamental flaw in this logic is that it assumes there will not be 100% user activity. As a result, users are affected by higher response latencies, slower mail delivery, and other performance issues when the failover datacenter is activated.
    6. SSL offloading (another supported, but rarely recommended scenario) was not configured per our guidance.
    7. Storage area networks were not designed to deliver the capacity and IO requirements necessary to support the messaging environment. We have seen customers invest in tiered storage to help Exchange and other applications; however, due to the way the Extensible Storage Engine and the Managed Store work and the random nature of the requests being made, tiered storage is not beneficial for Exchange. The IO is simply not available when needed.

    How can the complexity be reduced? For Exchange, we use predictable recovery models (for example, activation of a database copy). Our Preferred Architecture is designed to reduce complexity and deliver a symmetrical design that ensures that the user experience is maintained when failures occur.

    Ignoring Recommendations

    Another concerning trend I witnessed is that customers repeatedly ignored recommendations from their product vendors. There are many reasons I’ve heard to explain away why a vendor’s advice about configuring or managing their own product was ignored, but it’s rare to see a case where a customer honestly knows more about how a vendor’s product works than does the vendor. If the vendor tells you to configure X or update to version Y, chances are they are telling you for a reason, and you would be wise to follow that advice and not ignore it.

    Microsoft’s recommendations are grounded upon data- the data we collect during a support call, the data we collect during a Risk Assessment, and the data we get from you. All of this data is analyzed before recommendations are made. And because we have a lot of customers, the collective learnings we get from you plays a big part.

    Deployment Practices

    When deploying a new version of software, whether it's Exchange or another product, it's important to follow an appropriate deployment plan. Customers that don't take on the unnecessary risk of running into unexpected issues during the deployment.

    Proper planning of an Exchange deployment is imperative. At a minimum, any deployment plan you use should include the following steps:

    1. Identify the business and technical requirements that need to be solved.
    2. You'll need to know your peak usage time(s) and you will collect IO and message profile data during your peak usage time(s).
    3. Design a solution based on the requirements and data collected.
    4. Then, you use the Exchange Server Role Requirements Calculator to model the design based on this collected data and any extrapolations required for your design.
    5. Then, you'll procure the necessary hardware based on the calculator output, design choices, and leverage the advice of your hardware vendor.
    6. Next, you'll configure the hardware according to your design.
    7. Before going into production, you'll validate the storage system with Jetstress (following the recommendations in the Jetstress Field Guide) to verify that your storage configuration can meet the requirements defined in the calculator.
    8. Once the hardware has been validated you can deploy a pilot that mirrors your expected production load.
    9. Be sure to collect performance data and analyze it. Verify that the data matches your theoretical projections. If the pilot requires additional hardware to meet the demands of the user base, optimize the design accordingly.
    10. Deploy the optimized design and start onboarding the remainder of your users.
    11. Continue collecting data and analyzing it, and adjust if changes occur.

    The last step is important. Far too often, I see customers implement an architecture and then question why the system is overloaded. The landscape is constantly evolving. Years ago, bring your own device (BYOD) was not an option in many customer environments, whereas, now it is becoming the norm. As a result, your messaging environment is constantly changing – users are adapting to the larger mailbox quotas, the proliferation of devices, the capabilities within the devices, etc. These changes affect your design and can consume more resources. In order to account for this, you must baseline, monitor, and evaluate how the system is performing and make changes, if necessary.

    Historical Data

    To run a successful service at any scale, you must be able to monitor the solution to not only identify issues as they occur in real-time, but to also proactively predict and trend how the user base or user base activity is growing. Performance, event log and protocol logging data provides two valuable functions:

    1. It allows you to trend and determine how your users’ message profile evolves over time.
    2. When an issue occurs, it allows you to go back in time and see whether there were indicators that were missed.

    The data collected can also be used to build intelligent reports that expose the overall health of the environment. These reports can then be shared at monthly service reviews that outline the health and metrics, actions taken within the last month, plans for the next month, issues occurring within the environment and steps being taken to resolve the issues.

    If you do not have a monitoring solution capable of collecting and storing historical data, you can still collect the data you need.

    • Exchange 2013 captures performance data automatically and stores it in the Microsoft\Exchange Server \V15\Logging\Diagnostics\DailyPerformanceLogs folder. If you are not running Exchange 2013, you can use Experfwiz to capture the data.
    • Event logs capture all relevant events that Exchange writes natively. Unfortunately, I often see customers configure Event logs to flush after a short period of time (one day). Event logs should collect and retain information for one week at a minimum.
    • Exchange automatically writes a ton of useful information into protocol logs that can tell you how your users and their devices behave. Log Parser Studio 2.2 provides means to interact with this data easily.
    • Message tracking data is stored on Hub Transport servers and/or Mailbox servers and provides a wealth of information on the message flow in an environment.

    Summary

    As I said at the beginning of this article, many of these customer issues could have been avoided by taking sensible, proactive steps. I hope this article inspires you to investigate how many of these might affect your environments, and more importantly, to take steps to resolve them, before you are my next critsit escalation.

    Ross Smith IV
    Principal Program Manager
    Office 365 Customer Experience

  • How to Configure S/MIME in Office 365

    S/MIME in Office 365

    S/MIME (Secure/Multipurpose Internet Mail Extensions) is a standard for public key encryption and digital signing of MIME data. Configuring S/MIME in Office 365 is a slightly different procedure than configuring S/MIME on-premises. This blog is for people who want to move from on-premises to Exchange Online and want to continue to use S/MIME. This article will also apply to any Office 365 customers who want to use S/MIME for sending digitally signed and encrypted mails.

    Configuring S/MIME will allow users to encrypt and/or digitally sign an email. S/MIME provides the following cryptographic security services for electronic messaging applications: authentication, message integrity, non-repudiation of origin (using digital signatures), privacy, and data security (using encryption). Further, Office 365 also provides the capability for end users to compose, encrypt, decrypt, read, and digitally sign emails between two users in an organization using Outlook, Outlook Web App (OWA) or Exchange ActiveSync (EAS) clients.

    Below, we will take you through the configuration steps that you will need to follow to configure S/MIME for Exchange Online Only (Scenario 1), and for Exchange Hybrid (Scenario 2).

    Scenario 1: Exchange Online

    In this scenario, all the users are hosted on cloud and there is no on-premises Exchange organization.

    Requirements

    1. .SST File (Serialized store): The SST file contains all the root and intermediate certificates that are used when validating the S/MIME message in Office 365. The .SST file is created from certificate store explained below.
    2. End user’s certificate for signing and encrypting the message issued from Certificate Authorities(CA) either Windows based CA or Third party CA.

    Configuration

    Remember that in Exchange Online, only the SST will be used for S/MIME certificate validation.

    1. Create a .SST file for the Trusted Root CA / Intermediate CA of the certificate issued to the users:
    You can use either Certificate MMC or PowerShell cmdlets to export SST file. I am using Certificate console to export the .SST here:

    Open certmgr.msc snap-in, expand Trusted Root Certificate Authorities > Certificates > select the CA Certificates which issued the certificates to end users for S/MIME and right click > All Tasks > Export…

    image

    Note: There may be some Intermediate CA’s. You can move them to Trust Root CA folder and select them (including the Trusted CA certificates) and export it all in one .SST file.

    2. Select Microsoft Serialized Certificate Store(.SST) > Click Next and save the SST file:

    image

    3. Upload .SST to office 365 server:
    Update the SST on office 365 exchange server by executing the following commands using remote PowerShell.

    $sst = Get-Content <sst file copied from the box>.sst -Encoding Byte

    (Example: $sst = Get-Content TenantRoot.sst -Encoding Byte)

    Set-SmimeConfig -SMIMECertificateIssuingCA $sst

    4. Publish user’s certificate to the Exchange Online GAL (Global Address List) using Outlook. If not published, users will not be able to exchange S/MIME encrypted messages.

    Note: To publish the certificate, the user must first have the certificate installed on their local machine.

    • On the File menu in Outlook 2013, click Options.
    • On the Outlook Options window, click Trust Center, click Trust Center Settings..., and then click Email Security.
    • In the Trust Center window, click Settings… (Here, you need to choose certificate issued by the CA you are going to use for S/MIME).
    • In the Change Security Settings window, type the Security Settings Name (you can name it anything) and choose Signing and Encryption certificate. Select the appropriate certificate assigned in previous steps, leave the Algorithm default and click OK.

    image

    • Once the information is selected, you will notice the Default Setting is populated with Security Settings Name. Now you can click the Publish to GAL button. To publish the certificate to the GAL, click OK.

    image

    5. To confirm the certificate is published in AAD (Azure Active Directory), connect to Exchange Online using remote PowerShell and run following command. Check to make sure that the UserSMimeCertificate attribute is populated with the certificate information. If not, return to step 4.

    Get-Mailbox <user> | FL or FT *user*

    image

    6. Once you confirm the end user has the certificate on their machine under certificates > personal store and also published in AAD, the users can use Outlook, OWA, or EAS to send and receive S/MIME messages.

    Note: Make sure you check S/MIME Supported Clients section below before exchanging S/MIME messages.

    Scenario 2: Exchange Hybrid

    In Exchange Hybrid topology, some mailboxes are homed on-premises and some mailboxes are homed online, and users share the same e-mail address space.

    Requirements:

    1. Public Key Infrastructure (PKI). You can use Active Directory Certificate Services to issue certificates to the end users.
    2. SST File (Microsoft serialized certificate store). Tenant admins will have to configure their tenant in O365 with signing certificates issuing CA & Intermediate certs information. They will have to produce a SST file, which is a collection of certificates, and then later import it into O365 to validate S/MIME.
    3. DirSync. You will need version 6593.0012 or higher of the DirSync tool. DirSync is used to synchronize the Active Directory user object to the Azure AD, so that cloud users can also see the certificate information of recipients when performing S/MIME (encrypt) operation.

    You can verify the DirSync version following these steps:

    • Open Control Panel.
    • Click Programs.
    • Click Programs and Features.
    • Click Windows Azure Active Directory Sync tool.
    • Check the version as the screenshot below:

    image

    Configuration:

    1. Public Key Infrastructure (PKI)

    The users in your organization must have certificates issued for digitally signing and encryption purposes. You can either install Certificate Authority On-premises to issue certificates to the end users or have third party certificates issued to them. There are two attributes in a user object where certificate information stored: 1) UserCertificate and 2) UserSMimeCertificate.

    UserCertificate is populated automatically in on-premises deployment with a Windows root CA. This is populated at the time the user enrolls for a user certificate. This could be done manually for each user, or an administrator can set a GPO to automatically enroll all users.

    Certificates are stored in the userSMimeCertificate attribute when an Outlook client publishes a certificate to GAL. Outlook 2010 and above will populate both attributes with the same certificate http://support.microsoft.com/kb/2840546. But Outlook 2007 and below will not. http://support.microsoft.com/kb/822504

    2. When setting a SST file, remember in Exchange online, only the SST will be used for S/MIME certificate validation.

    Create a SST file for the Trusted Root CA / Intermediate CA of the certificate issued to the users:
    You can use either Certificate MMC or PowerShell cmdlets to export the SST file. I am using the Certificate console to export the SST here:

    Open certmgr.msc snap-in, Expand Trusted Root Certificate Authorities > Certificates > select the CA Certificates which issued the certificates to end users for S/MIME, and right click > All Tasks > Export

    image

    Note: There may be some Intermediate CA. If there are, move them to Trust Root CA folder and select them, including the Trusted CA certificates, and export them all in one .SST file.

    Select SST > Click Next and save the SST file:

    image

    Upload .SST to Office 365 server:
    Update the SST on Office 365 Exchange server by running the commands below using remote PowerShell:

    $sst = Get-Content <sst file copied from the box>.sst -Encoding Byte

    (Example: $sst = Get-Content TenantRoot.sst -Encoding Byte)

    Set-SmimeConfig -SMIMECertificateIssuingCA $sst

    3. If end users are issued third party certificates, they can publish the certificate information to the GAL by following these steps:

    Note: To publish the certificate, the users must first have the certificate installed on their local machine.

    • On the File menu in Outlook 2013, click Options.
    • On the Outlook Options window, click Trust Center, click Trust Center Settings..., then Email Security.
    • On Trust Center window, click Settings… (Here, you need to choose which certificate you are going to use for S/MIME).
    • In the Change Security Settings window, type the Security Settings Name (you can name it anything), Choose Signing and Encryption certificate, select the appropriate certificate assigned in previous steps, leave the Algorithm default, and click OK.

    image

    • Once the information is selected, you will notice the Default Setting is populated with Security Settings Name. Now you can click the Publish to GAL button. To publish the certificate to the GAL, click OK.

    image

    • To confirm that the certificate is published in AAD (Azure Active Directory), connect to Exchange Online using remote PowerShell and run the following command. Check to see if the UserSMimeCertificate attribute is populated with the certificate information. If not, return to step 4.

    Get-Mailbox <user> | FL or FT *user*

    image

    If Windows Certificate Authority is used, then the CA will publish the certificate information into the user object. In both cases, you need to use DirSync to replicate the on-premises Active Directory information to the cloud so that cloud users can exchange S/MIME messages.

    4. After the above steps, your end users can use Outlook, OWA, or EAS to send and receive S/MIME messages.

    Note: Make sure you check S/MIME Supported Clients section below before exchanging S/MIME messages.

    S/MIME Supported Clients

    All the client machines should have the PKI issued user certificate installed under (whichever is applicable)
    Certificates - Current User
    - Personal - Certificates
    ­­­­- Trusted Root Certification Authorities - Certificates
    - Intermediate Certification Authorities - Certificates

    If the PKI issued certificate is not available, users will not be able to send digitally signed messages or decrypt the S/MIME encrypted messages.

    Outlook Web App:

    • OWA for S/MIME - Supported only on Windows Vista or greater with browser IE9 and above. Not supported on other browsers or on MOWA (Mobile for Outlook Web Access).
    • Third party certificates aren’t supported for OWA S/MIME; only Windows Certificate Authority issued certificates are supported.
    • To use Outlook Web Access with the S/MIME control, the client system on which the user is running Internet Explorer must have Outlook Web Access with the S/MIME control installed. S/MIME functionality in Outlook Web Access cannot be used on a system that does not have Outlook Web Access with the S/MIME control installed.

    SMIME control in OWA requires .Net 4.5. All users accessing their mailboxes using OWA should install this on their machine. .Net 4.5 can be installed from Microsoft Downloads page.

    Outlook

    • Outlook 2010 and above are supported.

    EAS Clients

    • Windows phone 8.1 is a supported EAS client for S/MIME. To learn how to install a certificate on Windows Phone 8.1, see Installing digital certificates.
    • For any other devices, you need to check with the device vendors.

    FAQ

    1. Do both of these user object attributes (UserSMIMECertificate and UserCertificate) need to be populated with certificate information?

    Either, or both.

    2. Do we support S/MIME for Cross Org/Cross Tenant?

    Cross Org/Cross Tenant S/MIME is not supported in Outlook Web App and EAS (Exchange Active Sync)

    With Outlook, it is a supported scenario. So, when we are looking for certificates for recipients, we check in all the Address Books.  This includes the Global Address Book (GAL), the Contact Address Book (contacts folder), as well as any other address books (which includes LDAP address books). As long as we can find an entry in an address book for the recipient and it contains a certificate that we trust, then we can use it and send S/MIME mail.

    Note: Certificate in Exchange online GAL (Contact) currently not supported.

    3. When I select Encrypt mail and click on Send button in Outlook/OWA, I get error saying that the sender does not have a certificate. Why?

    In the example below, David is a sender. He was trying to send an S/MIME encrypted email message to a couple of recipients who have certificates published in the Active Directory, but David himself doesn’t have a certificate. When he clicks Send, he gets the below error.

    image

    So, when sending an S/MIME encrypted message, we always check the sender’s certificate so that the message is encrypted such that the sender himself can see it from his Outlook ‘sent items’ folder.

    References

    Understanding S/MIME

    Special thanks to Frank Brown, Mike Brown, Timothy Heeney, Tariq Sharif, Vikas Malhotra and Eduardo Melo for reviewing this post!

    Suresh Kumar

  • Exchange releases: December 2014

    Editor's Note: Updates added below for important information related to Exchange Server 2010 SP3 Update Rollup 8.

    The Exchange team is announcing today a number of releases. Today’s releases include updates for Exchange Server 2013, 2010, and 2007. The following packages are now available on the Microsoft download center.

    These releases represent the latest set of fixes available for each of their respective products. The releases include fixes for customer reported issues and minor feature improvements. The cumulative updates and rollup updates for each product version contain important updates for recently introduced Russian time zones, as well as fixes for the security issues identified in MS14-075. Also available for release today are MS14-075 Security Updates for Exchange Server 2013 Service Pack 1 and Exchange Server 2013 Cumulative Update 6.

    Exchange Server 2013 Cumulative Update 7 includes updates which make migrating to Exchange Server 2013 easier. These include:

    • Support for Public Folder Hierarchies in Exchange Server 2013 which contain 250,000 public folders
    • Improved support for OAB distribution in large Exchange Server 2013 environments

    Customers with Public Folders deployed in an environment where multiple Exchange versions co-exist will want to read Brian Day’s post for additional information.

    Cumulative Update 7 includes minor improvements in the area of backup. We encourage all customers who backup their Exchange databases to upgrade to Cumulative Update 7 as soon as possible and complete a full backup once the upgrade has been completed. These improvements remove potential challenges restoring a previously backed up database.

    For the latest information and product announcements about Exchange 2013, please read What's New in Exchange 2013, Release Notes and Exchange 2013 documentation on TechNet.

    Cumulative Update 7 includes Exchange-related updates to Active Directory schema and configuration. For information on extending schema and configuring Active Directory, please review Prepare Active Directory and Domains in Exchange 2013 documentation.

    Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, or who are using Exchange Online Archiving (EOA) with their on-premises Exchange deployment are required to deploy the most current (e.g., CU7) or the prior (e.g., CU6) Cumulative Update release.

    Update 12/12/2014:

    Exchange Server 2010 SP3 Update Rollup 8 has been re-released to the Microsoft download center resolving a regression discovered in the initial release. The update RU8 package corrects the issue which impacted users connecting to Exchange from Outlook. The issue was insulated to the MAPI RPC layer and was able to be isolated to quickly deliver the updated RU8 package. The updated RU8 package is version number 14.03.0224.002 if you need to confirm you have the updated package. The updates for Exchange Server 2013 and 2007 were not impacted by this regression and have not been updated.

    Update 12/10/2014:

    An issue has been identified in the Exchange Server 2010 SP3 Update Rollup 8. The update has been recalled and is no longer available on the download center pending a new RU8 release. Customers should not proceed with deployments of this update until the new RU8 version is made available. Customers who have already started deployment of RU8 should rollback this update.

    The issue impacts the ability of Outlook to connect to Exchange, thus we are taking the action to recall the RU8 to resolve this problem. We will deliver a revised RU8 package as soon as the issue can be isolated, corrected, and validated. We will publish further updates to this blog post regarding RU8.

    This issue only impacts the Exchange Server 2010 SP3 RU8 update, the other updates remain valid and customers can continue with deployment of these packages.

    The Exchange Team

  • November Exchange Releases delayed until December

    We know that many of you are anxiously awaiting the release of our quarterly Exchange updates planned for November. Earlier today the Exchange Team decided to hold the release of these packages until December. We made this decision to provide more time to resolve a late breaking issue in the Installer package used with Exchange Server 2013. We have discovered that in some instances, OWA files will be corrupted by installation of a Security Update. The issue is resolved by executing an MSI repair operation before a Security Update is installed. We do not believe this is acceptable behavior and is unfortunately something that customers might only discover after they install a Security Update.

    As of this blog announcement, we believe the installer defect is limited to Exchange Server 2013. However, we are also evaluating previous versions of Exchange Server and are delaying the planned 2007 and 2010 releases as well to complete that investigation.

    The Exchange team remains committed to ensuring that our customers have the best possible experience and because of that we have opted to delay the November releases to address this issue.

    Exchange Team

  • Introducing the IMAP Migration Troubleshooter

    Situation

    If transitioning your organization from a non-Exchange system such as Google or Lotus Notes to Office 365, you typically need to follow the IMAP migration path. As diagnosing and remediating any issues you might run into during such a migration can be difficult for people unfamiliar with the matter, we worked with our Support teams to provide some guidance for exactly such cases and packaged it for you in a wizard-like package.

    Introducing the IMAP Migration Troubleshooter

    IMAP Migrations provide organizations with an effective way to move email from any environment that supports the IMAP protocol. This is usually used for any non-Exchange source email system. If Exchange is the source you would most likely opt for the Staged, Cutover, or Hybrid migration path.

    To help you with troubleshooting, we have released a new guided walkthrough (GWT) for IMAP migrations. This link will soon be surfaced in various KB articles as well as the support ticket creation process via the Office 365 portal.

    image

    The intent of this GWT is to take a tenant administrator step-by-step though the common troubleshooting tasks to solve their IMAP migration issues. We took the most common migration failure scenarios and put them into this easy to follow guide.

    http://aka.ms/IMAPMigrationGWT

    Feedback

    If you see any issues with the IMAP migration troubleshooter or think there are scenarios that should be added or improved, please let us know at MigrationGWT@microsoft.com.

    Special thanks to all that contributed to the creation of the GWT: Kevyn Pietsch, Nagesh Mahadev, Timothy Heeney, Shawn Sullivan, and to the writers and KE team for assisting with collaboration and coordination efforts: Charlotte Raymundo and Sharon Shen.

    If you are looking for assistance on any troubleshooting Hybrid migrations, see the Exchange Online Migration Guided Walk Through (GWT).

    Kevyn Pietsch, Nagesh Mahadev, Timothy Heeney