The Magic of Background Removal

Hi, I’m Tucker Hatfield and I’m a Program Manager on the Office Graphics team.

Pictures are great – worth 1,000 words they say – so it’s a great idea to use them to spice up a document or add some flair to a presentation. The problem is that they usually end up being self-contained rectangles in the middle of things, and they don’t really flow into the content. You can put borders or effects on them to make them look more artistic, but up until now the only way to isolate part of the picture was to go into an expensive photo editing package and learn the cumbersome process of selecting and removing portions of the image.

Background Removal is a new feature in Word, Excel, PowerPoint and Outlook that makes this process quick and easy for any picture. Unlike similar tools, the Office Background Removal tool doesn’t just select color ranges or trim to a border you draw. Background Removal uses new capabilities and algorithms from the Microsoft Research in Cambridge, UK to achieve better results automatically with very little effort or fine tuning from the user.

So, how does it work?

Even though I can’t explain the deepest secrets of how the code works, I can show you how to use it effectively. Let’s start with this picture and assume that we want to remove the background and keep only the flower.

Clicking the Remove Background button in Picture Tools will start the process. First off, Background Removal tries to figure out what portion of your picture is the foreground, the portion to keep, and which is the background, the portion to remove. The first step in this process is the marquee selection area that gets drawn when you first start Background Removal. When you first start the tool, you’ll see the marquee and portions of the image are overlaid with magenta. Everything marked with magenta is what Background Removal has marked as the background. The normally colored portions are foreground, and will be kept.

You’ll probably notice that the marquee is inset slightly by default. Why is that? Well, it’s rare that the subject of a photo fills the picture completely, and insetting the marquee slightly makes it easier for Background removal to figure out what is the foreground and what is the background. In general, the less background included inside the marquee the more accurate Background Removal will be.

As you can see above, if the goal is to isolate the flower, the default marquee size doesn’t really get the desired result. As it stands, the result would look like this:

To further refine what we get, we’ll need to adjust the shape and size of the marquee. The important rule to remember is that you want the marquee to contain everything you want to keep. It’s okay if there are portions of what’s in the marquee that you don’t want to keep – the magic will do its best to figure out what to keep and what to ignore – but nothing that sticks very far outside of the marquee will be kept, so it’s important to make sure everything you want is inside. Let’s size the marquee so that it is just slightly bigger than the flower. The marquee is sized just the same as any shape or image, by grabbing the handles and resizing or dragging the whole shape to a new location.

Background removal figures out what you wanted and isolates the flower, which results in everything but the flower being removed.

Since the system Background Removal uses to isolate foreground objects from the background isn’t simply based on color choices or contrast values, it can extract even similarly-colored objects from the background.

Or you can even choose to keep something other than the obvious…

Of course, no matter how good the logic is that’s trying to figure out what the foreground of the picture is, there will always be some cases where simply adjusting the marquee can’t figure out what should be kept and what should be discarded. For those cases Background Removal has some simple tools to mark up and refine your selection. We’ll talk about how to use those tools remove the background from an image that presents a problem in a future post. For now, let me close with a couple of quick illustrations of how you might use that flower we removed in the first example.