Microsoft Office 2010 Engineering

The official blog of the Microsoft Office product development group

How does usage data improve the Office User Experience?

How does usage data improve the Office User Experience?

  • Comments 18
  • Likes

Hello, my name is Tim Briggs – I am a user experience (UX) Researcher in the Office Design Group (ODG). Usage data is one of the important feedback mechanisms we look at to understand how Office is being used in the real world. Answering questions like “how frequently is a command used?” and “how many files contain this feature?” are critical parts of our entire engineering process.

Today, I’d like to give you a better sense of how we collect, interpret, and design based on usage data. To get the most out of this discussion, some background might be helpful to you. I encourage you to check out Shawn’s post on Designing with Customers in Mind post—it will give you an idea of our research and design process. I’ll also point you to Peter’s post on Data Driven Engineering—it introduces the Customer Experience Improvement Program (CEIP), which is the source of the usage data I’ll discuss.

What types of usage data do we collect?

We collect two main types of usage data: command usage and feature usage. Each gives us a different perspective but the goal of analyzing both is to find patterns of usage in customer data. It’s also important to note that because the data are coming from CEIP, they are anonymous and contain nothing that can identify individuals or their content. But seen in aggregate, general patterns of usage emerge that help us explore, confirm, or challenge our understanding of how all of the Office applications and services are being used.

How do we use Command Usage data?

Command usage data (aka “click data”) tell us how a specific command is used. For instance, take the example of the Paste command. If we combine all of the clicks on Paste, we can mine that data to get a better understanding of:

  • How frequently is it used—should we make it more or less prominent?
  • How many people use it—what’s the impact of improving the experience?
  • What’s the most frequent way of accessing this command—can we make it easier?
  • Does this command occur as part of a clear workflow—how do we better support that?

clip_image002

The Paste command in Word 2010 – command usage data helps us understand how it’s used

Command usage data also tells us about how different parts of the user interface are being used.

  • Within a ribbon tab, what are the most used commands—should we rearrange them?
  • Across tabs, what commands cause the most tab switches—where should the command live?
  • How and when do people use right-click contextual menus—how can we improve that mechanism?

clip_image004

The Home tab in Word 2010 – the arrangement and size of buttons is informed by command usage data

This is basic but super useful information…and a bit daunting. Did you know that in Word alone there are over 2,000 commands? Want to know the most common Word command operating on text? Paste.

How do we use Feature Usage data?

Feature usage data is a little more difficult to describe. Let me try. Think of a “feature” as a general capability of Office, like Tables in Word. There might be many commands associated with using tables (e.g., Insert Table, Delete Row, Move Column…). While we might be interested in each of those commands, often we want to understand use of the feature at a higher level. This requires feature usage data—counters that we build into the feature to answer specific, predicted questions:

  • How many files contain a Table—what is the impact of improving the Table feature?
  • How big is the average Table—should we focus on improving large or small Tables?
  • What are the most frequently used Table styles—what design choices should we provide?
  • What other features are used in files containing Tables—how can we improve the interaction of different document parts?

If you want an interesting exercise, think of an idea you have for improving Office. First, use Send-A-Smile to tell us about it. :) Then, think about what information you would need to know about how that feature is used in order to create and validate your design for all of the users that will be impacted. That’s your feature usage data. And here’s a tip we use with all of our teams: sketch out the graph of the data which you expect to see once implemented—this helps you think about the form of the data to collect and sets your expectations for the final data to help you spot errors in your initial assumptions.

This is just a sampling to give you an idea of the breadth and depth of typical usage data we use across our engineering stages, from initial ‘what-ifs’ to late stage ‘Beta use is showing we have to change this.’

The Story of Paste and Usage Data

Let’s look at an example. Usage data tells us that Paste is the most frequent editing operation (for instance, it’s used almost two times more frequently than Bold). We also know from our other research methods (field visits, usability labs, focus groups, Send-A-Smile, etc.) that sometimes reformatting pasted content can be painful. We feel it ourselves—it can be hard to get the new content to look right.

As we began working on Office 2010, we wanted to improve the Paste experience. In order to do that we needed to understand more about how Paste is used. We looked to the usage data to provide some context. Below is an example of some of the ways usage data informed the engineering of what would eventually become the Paste Options gallery, which Mirko explains in his post on Live Preview Paste.

Question

Data used

Results

How many users would this impact?

Command Usage:

  • % of users using it
  • % of sessions in which it’s used
  • # of times users used it in total
  • # of times users used it per session

Paste is used by almost every user in almost every session of Word. In fact, it occurs more frequently per session than other related commands…like Cut or Copy.

How are Paste and Paste Recovery being used?

Feature Usage:

  • Paste. Knowing we would always want to understand more about how people paste, we had instrumented the Paste feature to count the number of times it was used in different contexts.
  • Paste Recovery. This is the little on-object UI (OOUI) widget that appears after you paste. Analyzing its use helps us understand the final format of many pastes.

clip_image006

clip_image008

The result was the chart above. Note how frequently content is pasted from a different document or different application—those sources probably have different formatting, explaining the need for the additional formatting options provided by Paste Recovery.

How do users get to the commands?

Command Usage:

  • # of times it is accessed from the ribbon, from its shortcut (Ctrl-V), from the right-mouse menu, from the Quick Access Toolbar, etc.

Predictably in Word, the most frequent is Ctrl-V by a wide margin, with the right-mouse menu next.

We used this data to explore 1) improving the ease of access, the contents, and the interaction model of the Paste Recovery widget for keyboard users; 2) putting the Paste options directly in the right-mouse menu, for those users who are accustomed to pasting via the context menu; and 3) we also thought about how we can expose the improved Paste experience in the Ribbon. After iterating on several designs in the usability lab, you can see the result in the Office 2010 Ribbon, right-mouse context menu, and the Paste Recovery OOUI widget:

clip_image009

What do people do after they paste?

Command Usage:

  • List of the most frequent commands which follow Paste.

Other than continuing to type, the most frequent commands relate to formatting (changing font size, color, appearance). The next most frequent are another Paste or an Undo.

We used this data to understand what options needed to be available in the Paste Recovery options. It also clearly signaled that people are not satisfied with Paste results since they were often making “manual” formatting changes (or undoing the action) immediately after pasting.

Which Paste Recovery options are most used? Do we have the right defaults?

Feature Usage:

  • % of times each Paste Recovery option is chosen.
  • % of times different Paste Recovery options are evaluated but not chosen.

clip_image011

The result was the chart above. It shows how frequently each Paste Recovery option is used. KeepSourceFormatting is the default and comparing that to the number of times Paste is used versus number of times Paste Recovery is used, we have the right default. Note that the other options in the Paste Recovery widget are arranged in order by frequency:

clip_image013

Armed with the usage data above, we explored different designs, iterated on them, and arrived at the current design that Mirko explains in his post. And now, as we receive usage data from the Office 2010 Beta releases, we continue to analyze the usage of this and the other significant improvements we made this version.

So, I hope this discussion has given you a sense of how we use usage data to improve the Office user experience. It’s been a game-changer for us…thank you for providing us the data.

Tim Briggs, UX Researcher, Office Design Group

Comments
  • i have 2 points:

    1. Paste Recovery, i turn this off as soon as possible, just annoys me whenever i paste something in excel.

    2. i completely overhauled the ribbons in excel and outlook. do you get this data?

  • why did Microsoft spend so much time on the ribbon? And if paste is so popular, why did Microsoft make something of a mess of improving it in Office 2010?

  • I was wondering if there is an Office 2010 UI Guidelines document similar to the one created for licensing of the 2007 Ribbon?  Will similar licensing requirements be allowed for developing applications using the 2010 style of ribbon?

  • [This is mostly my standard MS Office and UI rant which I like to get off my chest, except the final paragraph, which is on topic]

    When I first saw the Ribbon I thought it was awesome. There had been so many menus and toolstrips before, but then I used it and didn't like the organisation. Personally I slightly prefer the iWork inspector (which I suppose is sort of like a floating ribbon). Sometimes the iWork commands could be better organised, but I prefer that I can have it sitting to the right of the document so that my focus is on the document and not an huge bar across the top.

    I'll admit I've hardly used 2007, but I'd like to say how much I loved the Task Pane when it appeared with 2002. A non-modal interface for editing styles and selecting clipart, etc. was a revelation. I've sort of fallen out of love with it with 2003, but that's because of the clipboard history appearing when I don't expect it or the Shared Workspace pane appearing after the document loads. Those are annoying because they suddenly obscure part of the document. Though the autorecovery pane appearing is welcome! Or maybe Visual Studio's interface spoiled me with all its panes able to be docked, floated or hidden (and those preferences able to be exported to another machine). I also think it's great that the little OOUI widgets (I never knew they had a name before) are used in both VS and Office, though SHIFT+ALT+F10 is a little awkward on an Apple keyboard. Visual Studio: now there's something Apple could learn from! Anyway, I just think there's a load of stuff from the ribbon that should go to the Task Pane since with bigger and wider screen, vertical space for the document is usually more important than horizontal.

    Random question: can Word styles be used in Powerpoint yet?

    Finally a couple of real questions about this particular topic. Have you found that exposing popular, but hidden commands has increased their usage? Or vice versa: has a popular command decreased in usage because another was made more prominent? Does evidence-based UX design have limitations in being data driven, especially since the majority of data are generally from copies in the wild? Do you have to take a visionary lead sometimes? Do you ever try to re-create workflows totally from scratch, not using existing user interface widgets?

  • John Wildes: Yes, Office is working on an update to the royalty free license and user interface guidelines that will include the Office 2010 improvements.  This will be made available about the same time the product is released.  Thanks for your interest!

  • Again a classic example of how little MS understands "real" users and How DDE can lead to disasters

    If I want to do a paste special Values, then formats and column widths I have to repeat the paste special operation 3 times.

    If instead of radio buttons you had given check boxes, i could have done this in one operation

  • So many users can complaining about the File button which requires accurate mouse cursor placement as opposed to the 2007 Orb that followed Fitt's law (just move the cursor quickly to the top-left-most area of the screen and click). Coming out of Backstage view once you enter is another thing I had trouble figuring out. The team needs to make sure entering and coming out of Backstage view is as accessible as possible. Clicking on the title bar for example should close Backstage view, so should pressing Backspace key which is the first key users are likely to hit instead of Esc. The only rectangular little File button should NOT be the way to enter and exit Backstage view. Change this accessibility annoyance before RTM. With the 2007 orb, people could click outside the menu too to exit it. Since Backstage view occupies the entire workspace area, it is very annoying to position the mouse again over File to exit Backstage view. This is a basic usability thing. Hope MS understands before RTM.

  • Just imagine if the Windows Start menu occupied the whole screen when clicked except the taskbar and imagine if the only way to close the Start menu was clicking Start orb again!

  • Lastly, in Paint and WordPad, clicking Alt+F opens the File button equivalent but pressing Alt makes the menu go away. Now we must press one key combo (Alt+F) to enter backstage view, then hit another key (Esc) to exit it. Remember Office team, usability reigns supreme. Usability is what made Windows XP so successful.

  • Or at least make the right mouse button close Backstage view. Currently the right mouse button is unused in Backstage view, how about clicking it anywhere closes Backstage view. It is really getting difficult to enter and exit Backstage view. It is a step back from 2007 orb in terms of *usability* at the expense of packing in more features, extensiblity and eye-candy.

  • I have sent many smilies and frowns. Hope these are taken care in later editions.

    As this blog talks about Paste, I would like to mention that the Options (OOUI) is very handy when I take an Excel table/Chart to Powerpoint.

  • Question:

    Technical question (probably best suited for a programming forum): Does the VSTO API expose any methods to capture Command Usage Data (example below)?  

    I’m able to recreate a thesaurus with a macro and I’m also able to create a custom button event in VSTO that calls and captures the user’s word selection from a custom list, but I’m not able to capture the existing event when a user’s right click context menu Synonym selection event occurs.  

    Do you have any suggestions or know if the option to capture this data is available to developers (outside of MS HQ)?

  • How about all of the companies that are still using Windows XP or maybe vista with Office 2003 because

    1. it works

    2. it costs money and lots of lost time and productivity to upgrade to a product of questionable added value

    3. the ribbon is stupid, but completely removing the classic ui as an option makes my blood boil.  why in the heck would you completely remove the menus, which were compact, non-intrusive, fairly well organized, constant accross systems and screen size, and most importantly - familiar; and replace it with something that is none of these things, imposes a huge learning curve (that is worst for your core longtime users), and takes more mousing and searching for once simple operations.  The fact that most of my key commands no longer work (including the ability to access the classic menus by keyboard using 3rd party addins to the ribbon) is sand in my eyes.  I wish I could use openoffice at work now, but we're locked into and even like some of the custom VBA and spreadsheets that rely on office.

    I haven't tried 2010 tech preview yet, so i apologize for my rant if this has been addressed, but as a recent (forced) switcher to multiple apps with ribbons, i have nothing but anger and pain in my heart the last couple months

  • Rangood, a way in which you can accomplish this is to rely on the context menu extensibility model (http://msdn.microsoft.com/en-us/library/ee691832(office.14).aspx) - you should hide the built-in control whose usage you want to track and add a custom control that exactly looks and works like the hidden built-in control.  In addition to replicating the functionality of the built-in button (via CommandBars.ExecuteMso method) that the custom control mimics, you can also record and collect click data in the OnAction callback for the custom control.  If you decide to this, you should make sure to have customer consent, though; your customers should be able to opt-out of this type of usage data collection and should certainly be aware of your planned utilization of that data.

  • Just a question... when will you re-introduce menus and customizable toolbars? For now I'll stay on Office 2003 or an "open Office"

    Usability: fullfil expecations, productivity (e. g. using shortcuts), readability (assistive technologies), an consistent UI (e.g. classic windows -> classic office), customization (let the user decide if he/she want to use gimmicks (ribbon) or a productive UI (menus/ toolbars)...

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment