Posted by Microsoft Research

Microsoft researchers’ innovations add a virtual keyboard and voice replies to Microsoft Band, along with machine-learning enhancements to canned responses.

When Microsoft Band debuted in October 2014, it was more than a wearable fitness device. In addition to valuable health-related features such as a 24-hour heart-rate monitor, Guided Workouts, and built-in GPS tracking, it allowed users to preview their emails and texts, get calendar alerts, get actionable insights from Microsoft Health, and connect to the Cortana digital assistant.

Users caught on quickly—and immediately clamored for more productivity features, including the ability to respond quickly and naturally to urgent messages. Thanks to cutting-edge technologies incubated by Microsoft researchers, a variety of new features have been introduced for Windows Phone 8.1 users in the latest update to Microsoft Band, which was released on Monday. They include an innovative virtual keyboard designed specifically for the device and new voice reply functionality powered by Cortana.

With these enhancements, the band now packs indispensable productivity features—which users can rely on not just while exercising but whenever they don’t have the time, attention, or ability to engage with their smartphones.

“A wearable device has very easy access to your attention,” says Tim Paek, who led the team of Microsoft researchers that spearheaded the new productivity features. “If you get an urgent message and you need to respond right away, it can be very inconvenient to rummage around for your phone. With the device on your wrist, you can reply easily and discreetly.”

That brings us to the two new productivity features for Microsoft Band: Voice Replies and the Virtual Keyboard.

Speech recognition is made possible by Cortana, the personal assistant for Windows Phone, on the back end. As you speak your reply, the band displays the words it detects, and you can edit that text using a tiny yet highly accurate QWERTY keyboard displayed on the 11-by-33-millimeter screen—about the size of an address label.

“It’s the smallest keyboard around, so we designed the layout to leverage the entire screen real estate for tapping on the characters you want,” says Paek. “We also made it easy to tap and edit words. We’re definitely upping the ante here on input for wearables,” he says.

The typing experience builds on core technologies that Microsoft researchers developed for typing on the latest Windows Phone, including Word Flow. Word Flow technology helped Windows Phone set the Guinness world records for both sighted and blindfolded texting on a touch-screen device—the sighted record being set in 2014 by a high school student in Seattle.

See also: Microsoft Brings World’s Fastest Texting to Windows Phone 8.1

“After we beat the blindfolded texting record, we wondered how else we could use the machine-learning technology we developed,” says Paek. “Fortunately, Microsoft Band was being developed, so we decided to take on the challenge of enabling typing on a fitness band, which most people didn’t think you could do.”

The Microsoft Research and Microsoft Band teams collaborated on the user experience of the Virtual Keyboard. As they iterated on the design, they began to realize that text input was not only possible but could also be very fast and accurate.

“When Microsoft researchers introduced a keyboard this fast and accurate, we definitely were surprised,” says Jason Grieves, the Microsoft Band program manager lead for productivity. “But once we started designing, building, user testing, and iterating, we started to see that what we thought was impossible was real.

The Microsoft Research and Microsoft Band team that developed the new productivity features: Asela Gunawardana, Senior Researcher, Microsoft Research; Darren Gehring, Principal Test Manager, Microsoft Research; Dmitry Rudchenko, Senior Developer, Microsoft Research; Dan Ostrowski, Design Lead, Microsoft Band; Jason Grieves, PM Lead, Microsoft Band; Amish Patel, Experience Director, Microsoft Band; Tim Paek, Research Manager, Microsoft Research; Devlin Bentley, Developer Lead, Microsoft Band; Ankur Sharma, Developer, Microsoft Band; Vishwas Kulkarni, Senior Developer, Microsoft Research.

“It reminds me of the story of Green Eggs and Ham by Dr. Seuss,” he adds. “You don’t think you’ll like it until you try it, and once you do, you can’t believe you lived without it.”

In addition to Voice Replies and the Virtual Keyboard, the Microsoft researchers also applied machine learning to enhance the canned response feature already available on Microsoft Band. With this new functionality, the band can analyze all sorts of communications and offer suggested replies to a range of messages, from Yes or No questions to ones where the best response is “I’ll be there soon.”

So say you’re in a meeting and your spouse sends a text message asking whether you want Chinese food or Mexican food for dinner. Microsoft Band recognizes that the message involves two choices. It offers you “Chinese” and “Mexican” auto-reply buttons, which you can just tap. The interaction is over in about two seconds.

This kind of “intelligent auto-reply” is machine learning in action. Microsoft’s research investments in machine learning have been paying off for customers, with these new features of Microsoft Band being just the latest to be delivered.

Previously, your only option was to reply with canned messages. “Now we offer multiple ways to respond,” says Paek. “If you’re offered an appropriate auto-reply, you can tap the choice and you’re done. If that’s not what you want, you can use speech or the keyboard to create your own custom response.”

Working on the new productivity features has deepened the collaboration between the Microsoft Band team and Microsoft researchers, who were behind the band’s original productivity features as well as a range of smart fitness features including smart counting—automatically sensing and counting your exercises as you do them—and 24-hour heart-rate monitoring.

See also: Microsoft Band, the first wearable powered by Microsoft Health, keeps fitness and productivity insights a glance away

“The partnership between Microsoft Research and Microsoft Band has been one of the best examples of a new culture at Microsoft,” says Grieves. “Both teams understand how important productivity is to Microsoft Band, and when two very different and passionate teams share a similar objective, great things will come.”

The teams share a conviction that wearable devices can help people become healthier and more productive. “My first son was born just four weeks ago, and the reason I love working on Microsoft Band is knowing I’m working on technology that will make his life and his world a better place,” says Grieves.

“Technology traditionally focuses on enabling you to understand and communicate with the world,” he adds. “But Microsoft Band is also about understanding you so you, in turn, can live a healthier and more productive life.”