April, 2008

by Sam Ramji on April 29, 2008 05:59pm

I have the privilege of interacting almost every day with technical and business experts who are creating the future of software—including both core engineering teams at Microsoft and thought leaders across a broad spectrum of open source communities. Especially in the last few months, I’ve been able to take more time to articulate where I think this is going – such as writing how open source has influenced Windows Server 2008 and participating in Infoworld’s roundtable on the state of open source.

I think that many people are seeing that the interrelationship between Microsoft and open source is being changed fundamentally (and for mutual benefit).

Today, Bob Muglia and Brad Anderson announced that System Center will have the ability to deliver automated management across heterogeneous IT environments, such as UNIX and Linux. What I see as a best practice for commercial and community engagement with open source technology plays a big part in this.

Specifically, Microsoft will deliver an agent infrastructure and management packs (MPs) for monitoring Linux and UNIX platforms for System Center Operations Manager 2007. Early partners like Xandros and Quest are delivering cross-platform MPs for MySQL and Apache, and Oracle, respectively. Microsoft and Novell are collaborating on the SUSE Linux Enterprise MP.

The agent infrastructure Microsoft is building to interoperate with UNIX and Linux is built leveraging industry standards and open source such as WS-Management and OpenPegasus. Pegasus is an open-source implementation of the DMTF CIM and WBEM standards coded in C++, designed to be portable, and licensed under an MIT license, and work is underway to integrate with the newly DMTF ratified WS-Management standard. Pegasus already ships as part of major Linux and UNIX distros.

It simply makes great technical and business sense to cooperate with the OpenPegasus community to build upon an industry-standards based, cross-platform technology. Just as important, however, is preserving the virtuous cycle of contribution, benefit, and subsequent contribution: Microsoft is joining the OpenPegasus Steering Committee. The agent technology—being built will be contributed back to the community under the Microsoft Public License (MS-PL), an OSI approved open source license.

I greatly appreciate Allen Brown's positive comments (Allen is the President and CEO for The Open Group) and the support and education we’ve received from the sponsors and maintainers of Pegasus. He said:

“We are pleased to have Microsoft join the OpenPegasus Steering Committee and welcome their commitment as a positive step for the global open source development community. Since The Open Group initiated the OpenPegasus project seven years ago, it has been deployed across a wide range of IT platforms worldwide. We look forward to Microsoft’s active participation in the continuing development of the project.”

Today’s announcement and the business and technical decisions made by the System Center team are a great example of the fact that commercial innovation, industry partnerships, and open source participation can all work together to make the whole greater than the sum of its parts. My enthusiasm and excitement—and my applause for the System Center team, partners like Xandros, Quest, and Novell, and the OpenPegasus community—is tempered solely by my conviction this is not the only or last example of the best of Microsoft, partners, and open source growing together. This is a great day – and there are more great days to come.

by Bryan Kirschner on April 25, 2008 06:12pm

The same week that Brad Smith (Microsoft’s General Counsel) keynoted at the Open Source Business Conference (OSBC), we held our first Microsoft-wide Open Source Day (which Jamie and Mario both blogged about). We all noticed folks expressed interest in what was presented and discussed at Open Source Day, so we’re going to try to share what we said and what we learned with the Port25 community.

I was one of the presenters and facilitators for Q&A and panel discussions. The thrust of my presentation was ensuring everyone understands why Microsoft cares about open source, how Microsoft engages with open source communities, and what this means to you as a Microsoft employee (see the succinct, if not terribly exciting, slide below…).

If you read Port25, and you know about Codeplex, and Microsoft.com/opensource, and you’ve seen the cool stuff the open source heroes are doing, you probably have a good idea about the former two items. It’s the “you” part for which Open Source Day represents a major milestone, and that’s what I’ll talk about here.

Here’s why Open Source Day—kicking off a continuing dialogue with hundreds of Microsoft employees, and, by extension, the people they work for, who work with them, who they have lunch with etc, etc—is a transformative event. There have been pioneers working on open source and Microsoft “growing together.” At the risk of oversimplifying, I’ll put them into two groups: people like me, Sam Ramji, and Bill Hilf, and other members of the Port25 team, for example, whose full-time job is thinking about open source in one way or another. With no disrespect to what is one heck of an open source all star team (there are many I could highlight, but we’ve got some nice videos for Tom and Hank…), it’s still a relatively small number of people relative to 80,000 plus employees around the world.

The next group is larger and its people who just did smart things: I’m oversimplifying, but open source wasn’t necessarily their one-and-only-job, or even specifically in their job description. There’s the Shared Source team, the Codeplex team, and bunches of people across the company (once again I’ll stick to the latest videos…Jim, John…shoot, it’s killing me to leave people out…Rob, Sara…). One of the best parts of being in “Group 1” is just discovering what “Group 2” is doing…far more often than not, just because it made sense for their product or team…and for open source partners or communities.

But we’re still not fully tapping “Group 3.”…. until now. From KDE to BSD, Office to MSR, there are a lot of people at Microsoft who came in to the company with experience and passion for some form of participation in open source. And there are lots of people who, in the course of their work and lives, wonder “why can’t my team / group / product do <insert idea about growing together with open source.” I know because Open Source Day broadened out the discussion to more people than ever before—and with more clarity about what is not only possible—but encouraged by the company. Open Source Day is a pivotal point in freeing the many, many smart passionate developers and program managers and others from any lingering concern that engagement with open source is something they need to worry about “more than” doing something else. (Yes, anything still has to make sense for customers, partners, and shareholders…like everything else--but open source engagement and strategy is a “first-class citizen.”)

These statements weren’t made specifically “for” Open Source Day. Brad’s comments are from his OSBC keynote. Ray’s are from his talk at the Microsoft MVP Summit. But they sum this up better than anything I had in my slides:

Ray Ozzie in response to a question about Microsoft and Open Source (read the whole thing here):

Well, my position toward Open Source generally is that it's a part of the environment. It's very useful for developers to be able to get the source code to certain things, to modify them. …Microsoft fundamentally as a whole has changed dramatically as a result of Open Source in terms of as people have been using it more and more, the nature of interoperability between our systems and other systems has increased. …Open Source is a reality. We have a software business that is based on proprietary software. We tactically or strategically, depending on how you look at it, will take certain aspects of what we do, and we'll Open Source them where we believe there is a real benefit to the community and to the nature of the growth of that technology in Open Sourcing it. …e live in a world together with Open Source, and we have to make it possible for you to build solutions and for customers to build solutions that incorporate aspects of both.

Brad Smith at OSBC (read the whole thing here):

…Before I say anything else, I do want to say this: We at Microsoft respect and appreciate the important role that open source software plays in our industry. We respect and we appreciate the passion and the great contribution that open source developers make in our industry. We respect and we appreciate the important role that open source software plays for our customers, customers who almost always have heterogeneous computer networks with software and hardware and services that, as you all well know, come from multiple vendors. That is not what you have always heard from us, and I recognize that. But I did want to start by saying that…

And the beauty of Open Source Day is this: Brad Smith and Ray Ozzie are pretty darn important, but, basically, they are in “Group 2.” What they’ve really done is catalyze and energize the hundreds—thousands—of people in Group 3 to take Microsoft and open source into a new era. I just hope those of us in “Group 1” have the energy to keep up!

by Frank Chism on April 15, 2008 05:40pm

It’s multicore time. Do you know where your parallelism is?
Do you know where your parallelism is? Well you better find it! This is because you must or your users will not see the doubling of performance that they have seen for the last seven or eight generations of microprocessors. Why? Well the chip makers ran out of physics, clocks just couldn’t go a lot faster every few months the way they used to and more cores, even if they double or triple the performance of the processor do not lead to a faster feeling system if all the applications are serial. So, is your application serial? It better not be.

Two examples from my own life demonstrate the user and buyer frustration we are about to hit head on. First my sister, a long time Mac user, bought a new dual socket dual core Mac. On paper this system was easily three to five times “more powerful” then her previous system, a dual processor Mac. Her first complaint was, “It doesn’t feel faster. It certainly doesn’t feel ‘wicked fast’ compared to my five year old system.” I pointed out to her that what she had was more power, not more ‘faster’. I told her to try doing more than one thing at once. It took awhile for her to get the hang of this, but one day recently she called and started raving about how ‘powerful’ her new system was compared to her old one. What she had learned how to do was to get three or four long running tasks to run in the background while continuing to do her interactive day to day tasks without any noticeable degradation in interactive performance!

As a second example, for Christmas I built my wife a new dual core system because she was complaining about her system being too slow and crashing all the time. It was a thing of beauty with twice the CPU, twice the memory, and five times the capacity and twice the bandwidth of her old disks. The processor was two generations newer with a 20% faster clock and over four times the memory bandwidth. I was ready for a big hug and profuse thanks for my intelligence and craftsmanship. Boy was I in for a disappointment. What I got was, “It isn’t any faster.” I am a victim of the core war!

The lights are on, but there’s no parallel home.
Well my personal tragedy was nothing compared to what will happen to any software developer or corporation that depends on their fortune or fame to flow from people using their product and liking the experience. That is, it will be tragic for those who do not start to think parallel for every step of every phase of every program that they write and design in parallel not try to graft it in afterwards. Just as with security, and network awareness, you don’t get good results trying to add in parallel later. You must start in parallel and continue in parallel and only go serial as a last drastic measure and then only for the shortest possible time. Sadly, even in my own backyard of High Performance Computing the number of programs that have tried to graft in parallel rather than design it in is appalling. What they end up with is often what I call the ‘Silence of the Lames’. That is, a lame parallel port that doesn’t scale or doesn’t speed up their program nearly as much as is possible.

Now you might say that this only works for scientific programs with massive amounts of data. Well, sorry Charlie, if your product is something as ‘serial’ as a document processing program like Microsoft Word, you had better be thinking how you can use those extra cores to improve the user experience. Why? Well as I said in the beginning: Code parallel or die!

For the want of a nail, the shoe was lost; for the want of a shoe the horse was lost; and for the want of a horse the rider was lost, being overtaken and slain by the enemy, all for the want of care about a horseshoe nail.  - Benjamin Franklin

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my ‘Parallel Imperative’ and what you can personally do about it.
So, never stop studying and I’ll blog at you later.

- Frank

by Frank Chism on April 14, 2008 03:22pm

“What we have here is a failure to communicate.”
- Luke in “Cool Hand Luke”

A fist full of cores in a server rich environment.
If you haven’t noticed that the world has changed, you had better wake up and smell the coffee. The era of killer apps written in serial code is ending. Like the norm in species extinction, the reason many ISVs will not survive this change is that the environment in which we develop code has changed rapidly and not all species can adapt to this new world. The change was preceded by a hint of ‘The Shape of Things to Come’ in the Beowulf and compute cluster revolution. This revolution created the first generation of commodity clusters based on microprocessor nodes connected by commodity networks. An evolutionary message from this generation is that if you are building a cluster you really ought to use server class nodes. You _can_ build a cluster from desktop parts, but the higher you scale your node count the more you want the convenience and reliability of server parts. I remember seeing a look of outrage on the face of an Intel marketing person when I commented that I thought their then brand new Itanium Tiger platform was a really nice part for building a cluster. He thought of the new platform as a High Performance Server, not as a ‘part’. Well get used to it. We don’t build computers out of discreet components like transistors any more (which is good for him because Intel makes really big Integrated Circuits, not transistors) and we have just moved up to the point where the level of integration is the server, not the processor.

And Then There Were Cores
So into this environment where servers are parts and many high performance computing applications were written to a distributed memory communications aware parallel programming model, the microprocessor has changed in the most radical way I can remember. Our equilibrium has been punctuated (see: "Punctuated equilibria: an alternative to phyletic gradualism" (1972) pp 82-115 in "Models in paleobiology", edited by Schopf, TJM Freeman, Cooper & Co, San Francisco. Eldredge, N. & Gould, S.J.) and we can expect rapid change to a whole new kind of parallelism. The change I refer to is the introduction and near ubiquity of multiple core processors. For years software developers have been able to count on dramatic increases in CPU speed and processing power on a Moore’s Law schedule. But, Moore did not say that the processing speed and power of CPUs would double every eighteen months, he said that the number of gates on a single chip would double every eighteen months. So, all was good for a long time because that doubling of gates meant smaller _and_ faster gates and greater architectural complexity and thus ‘faster’ CPUs. So by about 2005, just about every good idea for a better computer processor architecture ever thought of was incorporated into the major microprocessors on the market. So, when 2006 rolled around the obvious thing to do was to use the next doubling of gates to double the number of ‘processors’ on a microprocessor. Dual core was here and the trend will continue for at least a couple of doublings. Who knows what strange beasties will emerge when we get to the point where ‘many core’ processors become ‘too many core’ processors. Until then, though we can be sure: Multicore killed the serial star....

More on this tomorrow.

by jcannon on April 14, 2008 03:14pm

John Lam brings word of a Ruby meet-up at the MVP Summit happening today at 2PM at the International Meeting Place at the Washington State Convention Center in Seattle.

Quote John: "I’m going to be at the Convention Center from around 1:30 onwards. There are a lot of central public meeting places at the convention center. From where I sit at my desk this morning, it looks like the “International Meeting Place” on the second floor will do just fine. I’ll hang out there and I’ll be happy to demo / talk about IronRuby, OpenSource and whatever else *you* want to talk about. Follow me on twitter (john_lam) if you want up to the minute updates on where we’ll be just in case this location doesn’t work out."

More details here

by Garrett Serack on April 10, 2008 04:20pm

Back in January, I invited the Apache Software Foundation to attend the Windows Server 2008 Application Compatibility Labs, here on our campus in Redmond.

The week that Apache was here, was so valuable for both groups--the product groups got to see and understand what some of the issues were that some of the Apache projects have run into, and the Apache folks were able to get their hands on the developers who built the system.

Myself and Bill Rowe had hammered out some details before I actually sent the invitation out. Along with posting it on some of the Apache Mailing Lists, I also posted the invitation on my own blog so that others could see what we're up to. And, as to be expected, there was a wide variety of comments posted--both positive, and ... less positive.

My favorite though, was: "Microsoft should go to Apache developers and see if Windows Server 2008 works correctly with Apache, not the other way around."

In some ways, that would have been somewhat impractical--when the Apache folks visited us, they had the opportunity to meet with engineers and program managers from many different groups, in addition to getting access to the hardware in the lab and the expertise of the folks who run that.  For us to pick up the 20 or so people from the product groups that they actually met with, and drag them all out to all the locations where Apache developers are--which is pretty much everywhere--would not have been possible

Still, I felt it would be more than valuable for me to go ApacheCon, so that I had the opportunity to meet with Apache developers where they roam. When Bill was in Redmond, he invited me to the Apache Hackathon--the couple of days at the beginning of the conference that developers could hang out and code.  So, a snappy 10hr flight later, here I am at ApacheCon in Amsterdam.

The Apache Foundation is an interesting community--or rather community of communities.  It's not just one project (the http server is what most people think when they hear Apache), but literally dozens of top level projects, and a whole bunch more in the 'incubator' (where baby projects are cultivated until it is clear that it will have ongoing support and development).  The hackathon is just a large room with tables where folks can come in, sit down open their laptops and start coding. It's actually a lot quieter than I imagined it would be.  Naturally, the folks in communities tend to gravitate together and discuss their projects.

As I'm not really on any project, I've been bouncing around chatting up different groups, getting their perspective of their own little chunk of Apache.  Most of the people I've talked to aren't surprised at all that I'm here--which is definitely a change from conferences a year ago--and are excited to hear about our efforts.

Now, for the funny thing.  I booked my hotel a few weeks back, using the internal travel system here at Microsoft.  The hotel that the conference is at was booked, so I looked for one nearby.  Unfortunately, the tool doesn't let me search for hotels near another hotel, and I didn't know what else was close that I could search near (and my inability to read Dutch didn't help), so I used the tool to show me where the hotels were, I'd switch to http://local.live.com and see how close it was, and if it was close, I'd switch to the other tool to check out the availability, and there was not much available. ... I guess I was distracted while I was doing it, and I ended up booking a hotel right next to the airport, which is in no way close to the conference, and so I spent the night in that hotel--and called the wonderful travel support folks who found me a hotel where I needed to be, and I moved there the next morning. Lesson learned: next time I travel to the Netherlands, I'm asking Hank to find me a hotel.

by MJM on April 04, 2008 04:34pm

Jamie blogged about Open Source Day earlier this week. I’d like to provide a little more context about what the day was and why we did it.

About 6 months ago, our team was discussing the fact that, while many external people understand what Microsoft is trying to do in the open source space, many folks inside Microsoft still aren't fully aware.  As a result, we continuously discover people across Microsoft who are interested in (or working on) ways for their product or program and open source to grow together but believe they have to 'go it alone.'

To help those folks, we decided to put together an informational event to explain to key segments of our internal community what the company is doing with open source and why. Little did we know that what was originally expected to be a 2-3 hour discussion in a conference room with a few people would turn into a full-day event with 12 presenters, a 400-person, full multi-media presentation hall and one heck of a catering bill.

When I first announced our intent to inform employees about Microsoft’s source code strategy, I received an amazing response. Although my team spends its days thinking about how Microsoft participate in the open source communities, we sometimes forget how many parts of the company touch on open source issues. From product groups to marketing groups to our legal division, it seems like everyone has questions or ideas about how Microsoft should engage with open source.

Ultimately, since this was our first internal event, we decided to go with the 30,000’ view and bring in a series of speakers to talk about what we’re doing with source code and why we’re doing it. We left a lot of details to later discussions. The point of the day was to get employees thinking and talking about open source in new and challenging ways. 

To make sure the day wasn’t a Microsoft echo chamber, we invited Raven Zachary from the 451 Group to provide external context for our discussion. He got the day off to a stimulating start by providing an entertaining and informative overview of the business of OSS and how Microsoft fits in. While Raven pointed out some of the good things we’re doing, he focused particularly on what we’re not doing so well and how we can do better. He also provided the audience with some great insight into how Microsoft is viewed by many in the OSS community. He showed one slide in particular that really made me think:

This is a map from a site that is regularly updated to reflect how the “software wars” against Microsoft are progressing. I think the image is great, and not just because it’s entertaining. It reminds me that everything we do as a company is being watched and analyzed by smart and insightful -- even if sometimes a bit obsessive :) --- people. That means we have to do “real” things, and we have to be transparent about why we’re doing them. That doesn’t mean that everything we do has to please everyone; that will never happen. We just have to be sure there’s a reason for what we do and we that we share that reason openly and frankly.

After Raven set the stage, the audience heard from a number of folks including Sam Ramji, the Sr. Director of Platform Strategy, and Bryan Kirschner, the Director of Community Strategy who explained our open source strategy from a business and community perspective respectively.

Probably the most provocative of the morning’s keynote speakers was Tony Hey, the Corporate VP of External Research. Tony’s background is in academia, a place where Microsoft has had some challenges (as this slide from his presentation vividly illustrates):

Tony gave an inspirational talk about how Microsoft should be willing to work with open source in the academic arena. Tony is a straight shooter, and he emphasized that, when you strip away the ideology and propaganda, our goal is to provide the best experience for our customers. To do that requires flexibility in how we approach software development and licensing. As a company, we should expand our horizons and harness the best of all the various approaches to create the most efficient solutions to academics’ technological challenges.

The rest of the day involved talks by various groups who participate in code sharing at Microsoft, including the Shared Source Program and CodePlex (before anyone complains, neither of those groups were billed as purely “open source,” although there are certainly open source aspects to both.) We also discussed OSS legal issues and had a couple of panels involving folks here at Microsoft who work on open source projects or programs, including Rob Mensching, the developer of WiX, John Lam, the developer of IronRuby, Shawn Burke, the developer of the ASP.NET AJAX Control Toolkit and Tom Hanrahan, the director of the Microsoft-Novell Interoperability Lab in Cambridge, Mass.

At the end of the day, we had a reception for the hardy souls who stuck around through the snow. I was very pleasantly surprised by the reaction of the audience throughout the day and at the reception. There were tough questions and great stories from across the company. We knew there was a lot of interest in the company regarding open source, but we were surprised by the depth and breadth of that interest. This bodes well for the company because that interest will drive more and more open source activity inside and outside the company as we move forward. Who knows, maybe we’ll get Steve Ballmer to talk at Open Source Day 2009.

by jcannon on April 01, 2008 08:16pm

Last Friday, March 28th, 2008 was Microsoft's first company-wide Open Source Day. This was a significant event and milestone in the company's move towards openness and was made freely available to any employee interested in attending. Packed full of panels and presentations by leaders in open source technology and programs at Microsoft, including a few industry folks, the discussion was open and uninhibited (subtitled : "Everything you wanted to know about open source and Microsoft, but were afraid to ask.")

The event itself was held at the Microsoft Conference Center on the Redmond campus....the same facilities we use for executive discussions with visiting customers and dignitaries. The space itself represents our largest and most sophisticated meeting center- and it was used at capacity to host a cross-section of business, marketing, legal and engineering employees throughout the day. Who would imagine this was happening only four floors below the same building complex that houses Bill Gates and Steve Ballmer's offices....

Mario is blogging more deeply on this later in the week. However, I wanted to share some pictures I snapped during the event which I thought those outside the company might find interesting.

The above slide, titled, "Why go open?" was presented by Raven Zachary of the 451Group. Raven, the first speaker of the day, was presenting his perspective on the trends and issues facing the open source industry. This slide in particular focused on the "why" of going open....You may not be able to read the fine details, but it illustrates primary reasons why software companies choose open source business models. In order: competitive forces, first-mover advantage, customer demand, improved distribution, commercial barriers, community model and internal resource constraints.

Not the first topic one might expect Microsoft to lead the day's discussion with. But this was a different day and it lead with bold statements to challenge the wisdom of this crowd.

This picture is my favorite. For those unfamiliar with the weather of the Pacific Northwest, you know that snow is seldom seen, especially this late in the season. However, Friday saw a squall come through Redmond...and while not sticking, certainly causing a whiteout and dangerous road conditions. 

Why is this my favorite? When someone came up & asked what was going on in the Conference Center, pointing to the Open Source poster on the wall, I said, "Didn't you hear? Hell has frozen over?" :)

Till next time,
- Jamie

by jcannon on April 01, 2008 02:26pm

We wanted to thank the Open Source Initiative (OSI) for the commitment and willingess to work together - and today, making public the Microsoft Open Source Initiative: The Microsoft Open Source Initiative (OSI) is a non-profit corporation formed to educate about and advocate for the benefits of open source and to build bridges among different constituencies in the open-source community. Quote the latest OSI Board Blog: "In the OSI board election held at the end of March, we've [Microsoft] gotten a controlling majority of board seats."

You can find more information at http://www.opensource.org and sballmer's first blog post.