by admin on March 31, 2006 02:00pm

Microsoft’s Open Source Software Lab is an ambitious research project. Located on the company’s main campus, the lab houses more than 300 servers, which collectively run more than 15 versions of UNIX and 50 Linux distributions. It boasts a team of senior-level programmers and system administrators, some of whom were architects of popular Linux distributions or authors of well-regarded books. In short, the lab is one of a few such facilities in the world dedicated to open source research.

The driving force behind the lab is Bill Hilf, General Manager of Platform Strategy at Microsoft. Hilf joined the company in 2004 after working at IBM, where he was instrumental in driving IBM's Linux technical strategy for its emerging and competitive markets organization. Prior to his stint at IBM, Hilf was VP of Engineering at E-Toys, where he helped build the company's e-commerce infrastructure.

When Hilf speaks about the lab and his involvement, the usual response he gets is, “At Microsoft? Why run Open Source in a mixed environment at Microsoft?” While theories abound—ranging from “Microsoft is working on its own Linux implementation” to “Microsoft is considering porting Windows to Linux”—the truth is far simpler. The lab provides Microsoft with deeper insight into the world of open source software, and it helps the company improve how Microsoft products work with open source software.

“Contrary to the belief that Microsoft is anti-open source, the reality is not so black-and-white,” says Hilf. “Most customers don’t live in an either/or world, nor do they choose a technology based on its development model. Instead, they choose a technology based on its ability to serve a business need or solve a particular problem. By running open source software in a Windows environment, we’re learning how those technologies can work better together so that our customers can benefit from a broader range of choices.”

One of the issues being addressed in the lab is how Microsoft management tools can do a better job in heterogeneous environments. For example, for customers who are using Microsoft Systems Management Server or Microsoft Operations Manager and need to manage a Linux or UNIX server, the lab can provide input on which third-party technologies can enable that scenario.

Another example is the testing it has done with Windows Server 2003 R2, which includes a variety of UNIX-based services like Network File System (NFS) and Network Information Service (NIS). Collectively called the Subsystem for UNIX-based Applications, the lab tested those services extensively to see how well they could interoperate with open source software in a data center environment.

Practicing the Art of Coopetitition
Although testing interoperability between Microsoft products and open source software is one of the lab’s primary roles, it’s not the only one. The lab also helps Microsoft to build better products through a deeper understanding of open source software.

“Licensing restrictions permitting we analyze and benchmark open source software in areas where Microsoft competes or has an interest,” says Hilf. “We share those results with other teams at Microsoft, who use the data to determine how we can improve our own products.”

One recent example is the work the lab did for the Microsoft Windows Compute Cluster Server 2003, which the company announced in late 2005 as part of its entrance into the high performance computing (HPC) market. Today, that market is largely dominated by Linux.

“When the product team first began building Compute Cluster Server, they asked us to find the best HPC solution from an open source perspective,” says Hilf. “We built a large, clustered system using Linux and did extensive benchmarking, then we wiped out that installation and ran the same tests for Compute Cluster Server. The data we collected will help us to deliver a more compelling product.”

“Both Windows and open source software will continue to be around for years to come, so it’s important that we test and analyze interoperability with open source software even if we may sometimes compete with some of this software – this is the real world where mixed environments exist,” says Hilf. “Coopetition – cooperating and competing – is part of the real world. Customers exist in the real world so we focus on what they care about, not what people philosophize about.”

Although there are many different ways that Microsoft could gain that desired knowledge about open source software, Hilf believes that one of the most effective ways is through a hands-on approach in which his team must address the same challenges as customers who run open source software in real-world scenarios.

“Deeply understanding a technology without actually using it would be like trying to deeply understand a foreign country without spending any time there,” says Hilf. “Listening to Berlitz language CDs or reading travel guides might help familiarize you with a foreign culture before you visit, but you’ll remain a tourist until you’ve lived there for a while.” Rather than function as a third-party trying to understand the open source phenomenon by looking in from the outside, the Microsoft Open Source Software lab is immersing itself deep into this space, relying on hands-on experience and hiring the necessary technical expertise to generate fact-based, unbiased information.

“We’re out to find the science that proves or disproves the statements made about open source software, so that we don’t need to guess or draw abstract conclusions,” says Hilf. “By being a center of knowledge and competency, we’re able to provide hard facts to Microsoft product teams when they ask questions on the state of management for open source software or the state of a certain open source application.”

A Piece of Fiber and a Hole in the Wall
When Hilf was asked to build a lab and hire a team of researchers, he had no idea that he would literally be starting from scratch. Microsoft runs everything on Windows, yet Hilf had to make the lab a resemble a real-life, open source environment, meaning that he would probably get limited help from Microsoft’s IT group.

“During one of my first days on the job, I stood in an empty room while some IT guys threaded a network cable through a hole in the ceiling,” said Hilf. “I was still standing there, staring at the piece of fiber, when one of the guys came downstairs and said, ‘That's it—you're on your own now.’ Other than that cable, we literally had to build the lab from scratch.”

Hilf’s first step was to hire the lab’s staff, which would be its most important asset. The lab employs a mix of employees and contractors, all of whom have been senior developers or systems administrators in the open source community. Some have been chief architects or technical leads for Linux distributions, such as Daniel Robbins, the founder of Gentoo Linux, who worked in the lab from June 2005 to December 2005.

Other lab staffers have deep UNIX experience and are authors of UNIX books or tools. The lab also boasts open source software security experts, embedded developers, virtualization and clustering experts, and developers with strong backgrounds in GTK+, GNOME, KDE, and Localization.

After hiring a staff, Hilf began to assemble a vast array of different hardware, software, and applications. The lab contains more than 300 servers from vendors including Dell, Hewlett-Packard, IBM, Microtel, Penguin, Pogo, and Sun. The lab’s software is even more diverse, with some 15 versions of UNIX and 50 distributions of Linux—including many lesser-known ones like Asianux, CentOS and NetBSD.

“We run dozens of different versions of Linux to test open-source interoperability in a multitude of scenarios,” says Hilf. “And because we do everything on our own, from running our own network and security services to patching and updating, our environment mimics those of real customers.” If a Microsoft product makes it through this lab, it will probably survive in 90 percent of the UNIX, Linux and open source customer environments out there.”

An Open Source Bubble in a Sea of Microsoft
One of the more interesting and unexpected dynamics at the lab is that this very large open source and UNIX shop is surrounded by the world’s largest all-Microsoft IT environment, which includes Windows-based security services, Internet proxies, mail services, and other IT infrastructure elements.

“Customers frequently ask us how we manage open source software inside such a Microsoft-centric IT environment,” says Hilf. “They want to know how we get the platforms to work together, how we handle software deployment, and what kind of tools we use. We’ve had to figure out ways to interoperate not just within the lab, which itself is incredibly complex and diverse—but also between the lab and the rest of Microsoft.”

The lab’s breadth of management tools parallels its diverse servers, operating systems, and other applications. For software management and distribution, the lab uses a combination of Microsoft Systems Management Server and mainstream open source software, solutions, and services, such as Vintela VMX, Kickstart, Red Carpet, Portage, and Red Hat Network. To remotely manage the lab’s more than 300 systems, lab staffers use SSH, VNC, X-Windows Tunneling and Windows Terminal Services.

While it’s highly unlikely that any single customer would run such a diverse range of technologies, mimicking a broad range of scenarios allows lab personnel to better understand the challenges that customers face and hopefully play a role in remedying those issues.

“Running such a diverse range of technologies within a Windows-centric IT infrastructure has allowed us to test interoperability on a daily basis,” says Hilf. “In the process, we’ve learned some very interesting things. Some are simple, like how to access the Internet from Linux machines behind a Microsoft ISA proxy. Others are more complex, like how to set up highly mixed storage and backup systems .

Building and Testing Interoperability at the Lab
One of the lab’s more interesting discoveries came about while testing the interoperability of management tools—specifically, in extending Microsoft Systems Management Server (SMS) so that it can be used to manage UNIX, Linux, and even Apple systems.

“SMS was built to use an open protocol—called OpenWBEM—to communicate with other software that runs on non-Microsoft systems,” says Hilf. “By using that capability to extend SMS using Vintela Management Extensions (VMX), we’re able to manage all of our servers and desktops through a single interface..”

Another useful lesson in how commercial third-party software can be used to extend Microsoft products for use in a non-Windows environment involved Microsoft partners Centrify and Vintela, whose solutions the lab used to integrate its UNIX and open source systems with Microsoft’s Active Directory directory service, which provides identity, user access, and policy management services.

“The combination of Active Directory and Centrify Direct Control gives us a really powerful single authentication solution across a highly mixed environment. Although there are other ways to do this, we’ve had good success with this solution,” says Hilf.

The lab also played a role in helping test support for Linux in Microsoft Virtual Server 2005, Service Pack 1, which can virtualize Linux and Sun Solaris operating systems on servers running Windows. “We ran all 50 Linux distributions as guest operating systems on a single machine running Virtual Server,” says Hilf. “It worked out very well because we didn’t need a separate server to test each version of Linux.”

In “A Look Inside Microsoft’s Open Source Software Lab – Part 2,” (available here the week of 4/10/2006), we examine some of the lab’s other areas of focus, including work to facilitate skills transfer between Windows-based and UNIX/open source environments. We also look at research the lab is doing to better understand related aspects of the open source software where it may be headed next, including a look at the community development model and other key trends.