Taking a departure from my promised post on C2R for a minute to celebrate the passing of another Patch Tuesday.
A term recognized across the IT industry is "Patch Tuesday."
I'm hopeful this post will remind folks about Patch Tuesday stuff they already know, and hopefully to ring the bell of "common sense." Unfortunately Patch Tuesday isn't universally known, and most people aren't aware of the background or rationale for our approach. It is my ongoing hope that people reading this content are either learning new things, benefitting from having the basics explained to them, or surrendering myths or misperceptions about our update businesses. In any case the history of Patch Tuesday is interesting, and explaining our approach may help. In summary, the question asked and answered is "how do you update hundreds of millions of endpoints every 30 days without creating massive disruption?" The answer: "Very carefully."
Vague metaphor: Some people see streaks of light. Others see two boats passing each other at night in a busy harbor reading each other's navigation lights and following some basic conventions.
Patch Tuesday emerged as part of a broader security initiative undertaken by the company in 2002-2003 as a result of an increase in the number of threats to the Windows environment. Where Microsoft had previously shipped updates on a weekly schedule, it was decided that a monthly schedule combined with advance notice for time to prepare would be a more tenable customer experience for installing updates. At that time all products moved to the Patch Tuesday schedule.
Concurrently, the Windows Update, Microsoft Update and Office Update methods were evolving. Originally released for Windows 95 as a web site, Windows Update allowed users to manually visit a web page, connect and download updates via the web. Eventually the concepts of critical updates and notifications were added to Windows 98 products, giving way to Automatic Updates added for Windows ME and beyond.
In 2005, Microsoft announced a change to Windows Update, introducing the transition to Microsoft Update, to include updates for other products. This began the deprecation and eventual demise of Office Update, leaving Microsoft Update as the primary channel for updating Microsoft Office.
Fast forward 8 years, where today we have hundreds of millions of regularly updating clients, and billions of downloads each year. SAP, Adobe and other software companies have adopted a similar update method, and Patch Tuesday is almost taken for granted. Patch Tuesday has been proven as an effective method of keeping desktops up to date. We have invested quite a bit in update management, beginning with technologies designed to facilitate easy updating, as well as policies and communication guidelines to make the updates more understandable.
Microsoft Update delivers updates to hundreds of millions of users, scaling to large installation numbers very quickly after publishing. The scale and speed of distribution of these updates necessitates a level of caution when deciding what to ship and when. If even a small %age of customers are affected, that can quickly scale to millions or tens of millions of people. We watch the uptake of an installation with a microscope. We carefully monitor installation successes, failures, their causes and we try to learn as much as we can about the environments in which they manifest.
The combination of what ships when can be a tricky area. Our goal is to go as fast as possible. When vulnerabilities or critical updates span platforms or products, however, an alignment of schedules an priorities is necessary to ensure we don't unnecessarily expose one product for the sake of patching another. In some core components, we must sim-ship our updates. For example our Mac products, IE, Office and SQL might need to ship an update on the same day. Keeping 4 or 5 major product lines on cadence with update releases can be daunting, given the multitude of branch, build, automation, development environments in use.
For Public Updates / Patch Tuesday updates, we typically offer only those updates that are most critical to our global customer base. When everyone is affected by an issue, such as a security vulnerability, we offer that update globally. Through the history of Office Updates, you can review the list of what we've shipped, and quickly determine that little falls outside the domain of critical or security. The "Critical" vs. "Important" designation is a criteria of MSRC documented on the Security TechCenter.
In other words, if we take the risk to offer it to everybody, it is an update which we feel strongly that should be applied as early as possible.
An interesting area where we're seeing some change relate to who Office365 and other services that have heavy interaction with Office client applications. Services evolve, as data is understood from their usage, we learn more about how to better optimize client interactions with the services. For example, we encountered a handful of scenarios where sync to online folders wasn't always working as expected, or performed poorly in certain configurations. This update was a change we made for the benefit of service customers, and for the benefit of SharePoint customers syncing document libraries.
In many ways, this is "just another update," but in many ways it is a signal of the future of updating Office. We learned about these issues partly from customer reports or external reports, but performance and other aspects of service interaction are things we learn from telemetry. We are investing heavily to ensure that we are aware of any potential trouble before our customers are aware. And when we are discovering those things, we'll use the Public Update channel to distribute those updates. As our services grow, we want to ensure Office has a successful interaction and users are satisfied with the experience.
This is a departure (an addition) to our traditional response-oriented critical update strategy, and it is part of how Office begins to look more and more like a service. We are supplementing our common escalation channels (the usual mix of Microsoft Support, Communities, MSRC, Dr. Watson, etc.) with more direct line of sight into service health and customer experience. This addition of new sources of telemetry and incident information will increase our speed of issue resolution. The more we listen, the more we learn.
Before we get into the considerations of release cadence and Service Compatibility, it is probably good to start at the beginning. I find these posts are always useful as a digest to initiate the new users, and to refresh those who haven't visited this subject in a while. A myth defrag is usually in order. In this post, I'll break down some of the fundamentals of our update strategy, including what we ship, where, to whom. Beyond explaining the basics, I'll get into some of the lesser known aspects of updating the clients. All of what I am describing applies equally to our MSI-based installations as well as Click 2 Run. We'll get into the practical differences between MSI and C2R in a future post.
Gratuitous image as a vague temporal marker.
Office client updates are distributed on a regular basis to improve the quality of our products. Updates generally fall into a basic category: Security, Customer-requested hotfix, or just a plan "change." In the past, virtually all fixes for Office in-market products were a direct result of customer escalations. That is less true today with the increase of telemetry sources, our alignment to Office 365 and other factors. Over time we have moved from more of the cause / effect or direct engagement model to one where we are better at anticipating customer challenges and resolving them before they become actual problems. The most obvious example is our work with security agencies to address vulnerabilities before they become public, or looking at installation failure rates for a given update before their broad-scale release.
The most important factor in determining which products we update and with what changes is best outlined in the Microsoft Product Support Lifecycle. You can read the details beneath the link, but generally:
Our current releases of Office benefit from a 10-year product lifecycle. In other words, we will be publishing updates for our existing releases until 10 years after their initial availability. The dates for Office product support can be found here. As you will see from the site, Office 2000 and Office XP are no longer supported. We are no longer issuing updates for those products. Office 2003 is set to expire next year, along with Windows XP. You will also observe that Office 2013 is supported until October of 2023.
The first 5 years of product release are for hotfix release, service packs and security / critical changes. In this initial 60-month period, we respond to Microsoft Premier Customer requests for hotfixes, we respond to other requests, for example, from our consumer support teams or from other product groups seeking changes to improve compatibility across our product lines. We call this phase "Mainstream Support."
In mainstream support, we have historically offered 3 service packs during the 60-month mainstream support window. Service Packs are important milestones in deployment - they are a roll-up of all fixes to date, and they reset the support baseline for a given release. For example, the recently released Office 2010 Service Pack 2 is the current baseline for the 2010 release. Per the terms of the Office Service Pack Support Lifecycle, the prior baseline will expire from support 12 months from this release date, giving customers a full year to deploy the release.
The second 5-year period of support is termed "Extended Support" and we offer only security, critical and compatibility changes for products that are in this phase. Office 2003 and Office 2007 are both in their extended support phase, and if you take a look at Microsoft Update for Office 2003, for example, you'd see that the only available updates are marked as either security or critical.
Public Updates ship on "Patch Tuesday," or the 2nd Tuesday of each month. They contain Security Updates, Critical Updates and Global Updates. These are provided to everyone and designed to be installed by every user. These are offered to users via Microsoft Update, and WSUS Catalogs, and as Click 2 Run images.
Cumulative Updates ship on the 2nd Tuesday of even numbered months. (Feb, April, June, Aug, Oct, Dec). These are generally unadvertised, because they are releases of hotfixes requested by our Microsoft Premier customers. These are delivered to the hotfix server, and require an email address to download. For Click 2 Run customers, an image containing these updates is available, but the changes are not automatically downloaded until the next public update. More on how PU and CU relate to each other later.
Service Packs ship on the 4th Tuesday of the month. We tend to ship them once per 18 months, although there is no pre-determined schedule for those releases. The purpose of a Service Pack is to reset the support baseline, so that we can keep customers current and offer them the best support possible. Service Packs are published to Microsoft Update, the Download Center and in WSUS Catalogs.
The contents of a release can be determined by an examination of the associated KB Articles, which detail the affected packages or changes we're delivering. We keep a handy blog here, and we maintain an Update Center on TechNet, which has a handy feed to which you may subscribe. We also publish content to known channels like the Security Bulletin center on TechNet and other places.
This aspect of our strategy is not well understood, but has been true for a long time. All Office updates are cumulative. Every change or update installation contains all changes to date for that component. For example, a Security Update for Word contains all changes to date for Word, which includes all prior security fixes, hotfixes, bug fixes and other changes. What we don't do is create separate versions for specific updates or release channels - there isn't a "Public Update only" version of Word or a "Hotfix only" version. All changes are cumulative.
For Click 2 Run, this makes things very simple. Each time a new image is delivered to you, you have all changes to Office that have been released. Because those images currently ship on Patch Tuesday, you are getting the most current version of Office on a 30-day cadence.
For MSI installations, the problem is somewhat more nuanced, but the end effect is the same. If you install all MSI packages, you will be current on all changes for those packages. For example the Outlook.MSP package delivered in a PU release will contain all changes to date for the files in that package. By design, these packages interoperate with the existing support baseline, corresponding to a released service pack level. In other words, Office 2010 changes apply to the new SP2 baseline, and they will apply to the SP1 baseline for one year after the release of SP2. After this 12-month period expires, those updates will no longer target the SP1 baseline.
This detail is destructive to the myth we frequently encounter related to Service Pack testing. Because Service Packs are roll-ups of existing fixes plus some additional change, customers current on Public Updates have already installed the bulk of changes in a given Service Pack. The binary versions are sometimes updated and there are incremental changes, but these are generally of the variety of changes that don't meet the bar for a customer release - e.g., too small. I covered this a while back in a blog post on my team's site.
I'm asked this question a lot. The answer is pretty simple -- Use Automatic Updates.
If you are a Click 2 Run customer (e.g. you are a user of the Office365 product, or you bought Office 2013 in a retail outlet) keep updates turned on. The image updates on Patch Tuesday will be streamed to you when they are available. This is a very low-maintenance approach to getting current.
To enable automatic updates for Click 2 Run products like Office365 or Office retail products, visit the "Account" Tab under the file menu.
For MSI customers, use Microsoft Update to take automatic updates. You won't update as much as we change in any given Click 2 Run update, but you will accumulate all changes to relevant parts of Office over time.
For managed deployments, I'd encourage you to examine your desktop updates policy closely, to determine whether it is appropriate to take updates through Microsoft Updates. In most cases it is perfectly appropriate (better), and in those cases C2R versions of Office are worth considering because they make the update process so much easier. We will continue to invest heavily in C2R technology moving forward.
Of course all the fine-grained control of MSI installations remains available. This is appropriate in many environments, and we will continue to support this customer need.
In my next post I'll dive into some of the practical differences between MSI and Click 2 Run technologies, and explain the implications for managing desktops.
Customers give us a lot of feedback. We depend on that feedback to improve the quality of our products.
Since I joined our release management team 3 years ago, we have been operating with a mantra that our updates will improve the quality of the product over time. We continuously strive to make the product more stable, more secure, more performant.
At this moment the software updates business is at a crossroads. Patch Tuesday is well known at this point. Alternative updating strategies are becoming more mainstream. The most striking examples are the rapidity of single or few-instance services which are continually updating and refining in response to feedback and incidents within the service. Game client applications backed by online services require updates to log into the server and play. Stores filled with mini "Apps" are usually a complete re-installation when "updates" are published.
This mode of hyper-currency has great benefit to provider and to consumer. In many cases the rapid update cadence is paired with an aggressive, Agile development methodology which forbids / reduces bug backlog. Thus the time from discovery to resolution can be very short if the backlog is of a manageable size. Fixing bugs delights customers. Doing it faster is a good thing for everyone, so long as the experience doesn't get in the way.
In the case of a product like Microsoft Office, updating our products is a carefully managed exercise. Perhaps we are unique or on a very short list of peers in this regard, we have hundreds of millions of endpoints, all at various patching levels, with an infinite number of hardware and software permutations. Unlike gaming clients or single-instance services, Office client applications are usually connected to line-of-business or 3rd party add-ins. Within businesses this is more often the case than it is the exception.
Combining the diversity of the ecosystem with the complexity of the environments in which Office does its best work raises interesting questions in figuring the best approach to the new world of software updates.
Recently Jeff Teper was describing our intent to accelerate the cadence with which we deliver updates. As part of the Office 365 offering, we must consider how we are going to approach this problem for all of the connected client endpoints participating in the service.
I am going to treat this post as an ongoing dialog on our approach. This post will grow over time - stay with it and watch it unfold.
When it is complete, I hope to have a full description of the factors we're looking at for a faster release cadence, the approaches we're going to enable, considerations our customers might weigh, and hopefully collect some feedback along the way.
Feel free to participate in the discussion.