Follow Us on Twitter
by jcannon on October 10, 2008 06:11pm
When I began my journey with technology, it was with a passion for the web. Living off a friend’s T1 line, I was hacking together HTML when Mosaic was the only show in town. I’m returning to that love of the web next week where I’ll be moving to a new position in Product Management for Windows Live social networking.
To the community: I can’t tell you how much I’ve enjoyed learning, listening and working with you. I’ll take your wisdom with me & promise to carry the “open” flag with me wherever I go within Microsoft. You’re in good hands: effective immediately, Peter Galli, will be taking over as Open Source Community Manager on Port 25. Things have been quiet on Port 25, and Peter has great plans to shake things up …but more on that later.
See you around, -Jamie
by Brett Shoemaker on February 01, 2008 08:51pm
As an open source business strategy lead here at Microsoft, I am particularly interested in community reaction following acquisition waves like the one we have seen recently (Sun/MySQL, Nokia/Trolltech, SpringSource/Covalent, etc.). While I am interested in reaction to each announcement individually, I find those that attempt to extrapolate what the event says about the broader OSS landscape especially interesting. This time around, one question that keeps surfacing is whether open source companies have sold out. Put differently, does selling mean selling out? My answer is no.
First, let me quickly point out the obvious. This recent wave of open source acquisitions is nothing new. Over the last 3 years, we have seen a number of open source companies sell to traditional ones (e.g., Zimbra to Yahoo, XenSource to Citrix, Gluecode to IBM). There is also a continuum of “ownership” and participation at the project level as well from company-driven to community-driven projects (e.g., from IBM’s influence over Geronimo to Zend’s PHP involvement to community-driven projects on Sourceforge or CodePlex). And, there is a continuum of opinion on it.
When I hear the question raised of whether open source companies are selling out, my reaction is “Why should OSS companies be held to a different standard than that of traditional ones?” What I mean is that I expect companies, whether open source or not, to do what is in the best interest of their customers and provides the best opportunity for future growth. The question should not be are OSS companies selling out, but rather are OSS companies selling to the right companies and in what ways will it further the company’s purpose.
Furthermore, the approach that an OSS company takes—IPO, acquisition, VC backing, or go-at-it-alone—doesn’t particularly matter. Today, we see more acquisitions and not IPOs because these traditional companies place higher valuations on these OSS companies than the market does. While the market focuses more on revenues, these traditional companies price in other variables (competitive impact, benefits to existing complementary offerings, etc.).
Does this acquisition trend mean that the terms open and closed source will no longer be relevant in the future? Maybe. Maybe not. To me, it’s minutia compared to the overall trend. I expect to continue to see convergence between the traditional and open source business models, and I expect to see Microsoft and other traditionally proprietary companies’ involvement continue to grow, as it is in the best interest of customers, partners, and shareholders. The heterogeneity of the technology landscape will continue to grow and consist of multiple source approaches so as to deliver the most value to customers. So, for me, this wave of acquisitions is nothing more than the next logical step on that path, and I’m excited to be a part of figuring out those next steps.
by Peter Galli on May 20, 2009 03:05pm
Tony Hey, the corporate vice president of Microsoft Research's External Research group, used the Open Repositories Conference to announce today the public availability of Zentity and the second version of the Article Authoring Add-in for Word 2007, both of which will be released as open source.
Zentity, previously called Research-Output Repository Platform and code-named Famulus, is a platform that allows institutions to store all of their digital scholarship: papers, lecture, presentations, videos-anything that might be collected by the university as part of the digital output of their researchers and scholars.
Over the past nine months two betas for this have been released, which refreshed the user interfaces and added new controls, and complement the services provided in the package.
The second version of the Article Authoring Add-in for Word 2007 includes new functionality, including the ability to upload directly into a repository - Microsoft's or those of others - via the SWORD [Simple Web Operation for Repository Deposit] protocol.
Support for authoring Object Reuse and Exchange resource maps within the Word environment has also been added, as well as the ability to perform literature searches and to import the bibliographic information in Word with one click, which makes it very simple to quickly add citations into a paper.
A key element of the Microsoft External Research vision is to support the scholarly communications lifecycle with software and services so that data and information flow in a coordinated and seamless fashion.
With regard to the plan to open source these tools, Lee Dirks, the director of the Education and Scholarly Communication team, said that "first and foremost, we're releasing the binaries, but soon thereafter, we'll release both of these as open source. Once they are available, our big push over the next 12 to 18 months will be to build a worldwide community around these assets."
You can find a lot more information on these announcements here.
These moves also follow the March release by Microsoft and the Creative Commons of an add-in for Microsoft Word 2007 that enables authors to easily insert scientific hyperlinks or ontologies as semantic annotations to their documents and research papers.
Microsoft is also making the source code available for the Creative Commons Add-in for Word 2007 free of charge to open source communities on CodePlex through the OSI-approved Microsoft Public License, which lets developers tailor it for specific industries using domain-specific language.
by Bryan Kirschner on November 24, 2008 02:47pm
I was recently at Harvard for two events. The first, which I'll talk about in this blog, was part of the Berkman Center for Internet and Society at Harvard lunch series.
Mario Madden and I were invited to speak at a session called "Microsoft and Open Source: Opportunity or Threat?" You can watch the whole thing online at the link - and David Weinberger liveblogged as well.
The focus of the whole thing was, to quote Karim Lakhani, our host, a "vigorous discussion." So we had about 15 minutes to give an up-front presentation about our thoughts on the "opportunity or threat" issue.
The rest of the time was open discussion. So I do recommend checking out the webcast-it's tough to do the discussion justice second hand. I will call out a couple things that won't show up in the recording.
First, Harvard really is an important source of expertise on open source. There's a whole bunch of research that's certainly been valuable to me (on developer and corporate motivations, for example). There are also people like Margo Seltzer (the former CTO of Sleepycat) who I got to meet at the second event, which I'll talk about in my next blog.
Second, the whole OSS Lab at Microsoft community has emphasized the importance of dialogue for as long as we've been around. This event drove that home once again. Some folks followed up verbally or in mail to semi-apologize for it being a bit of a challenging environment for us.
But I didn't think it was challenging: if a question is difficult to answer because someone is working hard at making it difficult for me to answer, I'm not too keen on that. But if a question is difficult to answer because the answer is something we haven't thought about (and maybe should) or it's just a tough problem...if the questioner is willing to help me be smarter about figuring out a good answer, well, bring it on, as they say.
David Weinberger actually raised a point like this when we talked about Microsoft-released projects and contributions(from his blog):
Q: [David] Are 500 contributions a lot? Compared to the number of patents? Products? A: [Bryan] We'll measure success when every product group considers open source. Q: [Karim] IBM says they have 1,000 developers working on Linux, etc. Do you have any number you can point to that's similar? A: No.
I added we don't have a KLOC or person hours target...should we?
Third, just for the record, here, I said think open source and Microsoft represents a mutual opportunity (...check out the podcast for all the reasons why.) But that brings me to the one thing that most sticks in my mind. A CS professor who attended told us she waited until the recording was finished because she didn't want to be rude-but that to her, we were talking about our open source strategy as if it was something new and innovative.
But from her perspective, she said she's been doing software development a long time, and this sounds just like what Microsoft did in the late 1980's, when being open to developers is what made early Microsoft products interesting to her as a developer. So (to paraphrase): not to be rude, but why do you think this is cool?
This was funny because (as I replied) I absolutely agree with her. Our open source strategy took a lot of learning about how open source has changed the landscape, and what it has brought that's new and different, but the fundamental principle remains the same: openness to third-party developers is a powerful and enduring principle.
And it is part of Microsoft's DNA, as we sometimes say ("...the engineering relationship is getting back to the way it used to be in 1994-1997, which is a great relief to us," [Jeremy] Allison, said recently about Samba and Microsoft).
At one time, Microsoft was perceived to be a leader in openness through free SDKs and extensive APIs, active developer communities, published object models (wow, now you can call the Excel object model from the Powershell scripting language...) , and more.
For a number of (in my opinion) remediable reasons, from the time open source started to capture the popular imagination till today, Microsoft has not been perceived as a leader. But I don't see any reason why we can't reach the point where the best things Microsoft has brought to users and developers and the best things open source has brought to users and developers will be decidedly better together. I think there are some arguable examples already (XNA is high on my list: traditional coding contests plus easy paths to write and sell games, plus a growing open source community).
The other event at Harvard was a business focused Open Source CEO Summit...which I'll talk about in my next blog.
by jcannon on August 29, 2006 01:58pm
Last week, O'Reilly Media posted portions of two conversations that took place at the O'Reilly Radar Executive Briefing. One between Tim O'Reilly and Brian Behlendorf about lessons from Apache and CollabNet, and the onter between Bill Hilf and Danese Cooper of Intel about Open Source at Microsoft. O'Reilly was kind enough to allow us to re-post these discussions on Port 25 - we're hoping you enjoy the lively, and frank discussion as much as we did. Details below...
From O'Reilly: Distributing the Future August 21, 2006: "Open Source at Microsoft" Total running time: 33:40
Production Notes The initial montage is from Tim O'Reilly, recorded at OSCON '04 in a phone interview with Doug Kaye of IT Conversations, and used with permission. "The future is here, it's just not evenly distributed yet" is a quote from author William Gibson that Tim used with attribution.
Credits include special thanks to David Battino for composing and performing the theme music. David can be found at Batmosphere.com, and he also edits O'Reilly's Digital Audio site. David provided a lot of help and feedback getting this program launched. We used Soundtrack Pro, Bias Peak, and Audio Hijack Pro to put it together.
Daniel H. Steinberg is a developer, a longtime technical writer, and currently spends most of his time podcasting for O'Reilly.
Attachment: DistributingFuture.mp3
by admin on June 13, 2006 03:29pm
FreedomHEC trip report On Friday the 26th of April I attended the FreedomHEC Un-Conference (Yeah I am late with posting it). This was a two day conference which was held on the 26th and 27th. I only attended the first day. The FreedomHEC Unconference was billed as:
The hardware unconference where you'll learn how easy it is to make your hardware compatible with free, open source operating systems such as Linux, and available to new markets such as servers, next-generation entertainment devices, and more. Get answers on everything from kernel data structures to the fine points of licensing. Discover how participating in the Linux process is fast and simple, how the development process works, and where to get started.
The hardware unconference where you'll learn how easy it is to make your hardware compatible with free, open source operating systems such as Linux, and available to new markets such as servers, next-generation entertainment devices, and more.
Get answers on everything from kernel data structures to the fine points of licensing. Discover how participating in the Linux process is fast and simple, how the development process works, and where to get started.
It did not completely achieve this goal, but was very helpful to people who have never done device driver development for the Linux Kernel.
It was set up by Don Marti, and was attended by approximately 35 people of which 8 or so were current kernel maintainers. I think the enthusiasts outnumbered those people who would be writing company device drivers by 2 to 1.
The conference started with a suggestion session as to what people wanted to see so that a calendar could be created. As a result the first day calendar became this;
1. SYSFS OVERVIEW
2. LINUX SOCIAL ENGINEERING
3. SCSI Q AND A
4. ROCKETS
5. GETTING DRIVERS INTO THE KERNEL Q&A
Below are more details on the conference. My apologies if this at times is confusing, but I am working off my notes here...
This was an overview given by Greg Kroah-Hartman, who is the current Linux PCI tree maintainer (Among others he also does sysfs, kobject, debugfs and kref code). He works for SuSE Labs at Novell.
He presented a high level walkthrough of the sysfs subsystem (the /sys area) This is only available in the 2.6 kernels and is still evolving.
Sysfs (/sys) is a RAM file system, so anything created in there by humans after boot will be lost after any subsequent boots.
-sysfs shows all of the devices (virtual and real) and their inter-connectedness
In the past /proc has been used but /proc should be used for processes and not device drivers. /proc is the older method of creating device driver configuration files.
One of the reasons that /sys exists is to standardize configurations of various device drivers. /proc has been used/misused in the past for configuration representations/files/atributes, the data under this file system are different from programmer to programmer and device to device and are very hard to interpret unless you know the format.
/sys changes all that. It provides a standard method of defining and using device driver attributes. It is based off the principle of one value per file. And this value can only be a simple value, no histograms or large binary configurations (debugfs is specifically for this purpose!). Still people seem to break these rules at times.
Do a tree under /sys to see what the structure is, and cat the files for the values.
Power management used to turn things on or off, is not fully working yet. USB is getting there for power management; the rest, not yet. Put in a USB Pen drive for example and see /sys/block change real time (/sys/block/sda). If you unplug a device, udev will take care of cleaning up. You do not have to un-mount or do anything else. (If you do that to a drive you are writing to, of course you are on your own) You can also mount things by label easily.
If a change is made to a device attribute in /sys, an event is triggered that programs like hal can get real time. General guideline, if you write a user space program/driver, tie it into hal.
Also, /dev is now a RAM file system in 2.6 (Some distro’s might not have implemented this)
If you want to add proprietary drivers, they will live in /lib/usdev/devices, this is when they are not sysfs aware. This location is persistent.
This presentation was given by Randy Dunlop, A past USB and Kernel Janitor and maintainer.
The presentation was given to give people an overview of do’s and don’ts when they start submitting code to the kernel. A lot of the information given was common sense but a lot of people do not follow the rules. (e.g Rules so Obvious that they are not followed)
The presentation was very much in Bullet form and so are my notes on this, so they might not flow as well as they could.
SOCIAL ENGINEERING
(Meritocracy) Development.
Massive amounts of open communication via email etc.
LINUX CULTURE
LINUX DEVELOPMENT VALUES
THINGS TO AVOID
WHEN NEW INFRASTRUCUTRE IS NEEDED
DRIVERS FOR NEW HARDWARE
NEW DRIVER DEV
COMMUNICATIONS
MAILING LIST etiquette
GENERAL
In all the presentation was a crash course for people who have never done collaborative development, and tried to prepare people what is ahead and what kind of commitment will be needed to be part of the Linux kernel development.
Led by James Bottomley SCSI Subsystem maintainer.
I do not have many notes on this part because not much was said to keep notes on, a few things that I did write are as follows (Second item is of great interest)
If you want to get an idea of how SCSI drivers work, check out the 53c700 which is an excellent driver to learn from. It is older but continuously maintained.
There is kind of a new concept called a Target driver. Target drivers make it appear to the outside world as if it is a driver (A virtual driver more or less), but instead connects to the actual physical device drivers. Allowing, for example, to turn a Linux box into a RAID device. Then you can talk to the target driver as if it is one device. Not many of them out there yet, but there is one from IBM which James did not think was a very good. And there are a few on the way. There are two people (names escape me currently) that are actively developing them. In my opinion this is a pretty interesting concept that allows you to do a whole bunch of cool things. (e.g Network routers, Cheap raid devices etc)
You can (should?) write SCSI drivers in user space.
A lighter presentation was given by two students from Portland state. They are using Open source hardware and software to build rockets. Some interesting reads can be found on that project here:
http://psas.pdx.edu
This was pretty much done by Greg.
Some source code control software of choice of the Kernel Maintainers
Testing Kernel code is hard, but crashme is used by the maintainers be used for (stress) testing kernel stuff.
Greg was very adamant that they take any device driver. No matter what HW it uses, as long as you or somebody else is willing to maintain it they will take it. Old/New it does not matter. Even if there is only one user for it they will take it.
That’s all folks.
by admin on March 31, 2006 05:00pm
Welcome to Kishi’s Korner (for the record, I did not come up with this name.)
I wanted to take a moment and use my first blog entry to introduce myself. My full name is Harvinderpal Singh Malhotra and somehow “Kishi” was chosen as a nickname for me when I was growing up (don’t ask, like the name "Kishi's Korner", it’s a long story). Okay, so I am the Project Manager for the Open Source Software Lab. I’ve spent the bulk of my career in IT Operations as an Infrastructure Architect with Fortune 100’s such as Pfizer, United Health Care etc. and since 2003 I’d been with MSN Operations. While at MSN I was responsible for re-engineering the server deployment process and providing key infrastructure services including Active Directory, DNS, WINS, SecurID, TermServ, Domain Registrations, SMTP, and AntiVirus for the MSN Operations IS team, among other things.
While much of my life has been spent living IT Operations, one of my true passions is research and writing. That’s why I’m thrilled to be involved with the Open Source Software Lab and Port25. I look forward to sharing our work with you and more importantly learning from you on how we can improve our methodologies and perhaps come up with new projects that we haven’t thought of. This is a unique opportunity for myself and the community and I look forward to being a part of it.
If you are so inclined, please take a moment to shoot me some thoughts on things you’d like to see us work on in our lab. Moving forward I’ll be having some of my team of Open Source, UNIX, and Linux engineers share what they are working on. If you have comments on our methodology or suggestions on how we’ve scoped a particular project, please let me know.
OK, enough with the introductions, let’s talk projects! I look forward to hearing from you.
by Bryan Kirschner on April 28, 2009 03:37pm
The first time I went to a LinuxWorld conference as a Microsoft employee, a guy passing by me saw "Microsoft" on my name badge and stopped. "Microsoft? What are you guys doing here?" he said. "I loved Microsoft. You put my kids through college."
As it turns out, he owned a small IT business during the late ‘80s and early 90s, which thrived building applications during the headiest days of the "PC revolution."
The last time I went to an OSBC as a Microsoft employee, I MC'd the third annual Open Source ISV "Day 0" event hosted by Microsoft. I told that story in my opening remarks. At the reception at the end of the day, one of the attendees came up to me and said: "You know, I'm one of those guys who's been doing technology for 30 years. And today's event felt like Microsoft in the early 90s. It's the first time I've gotten that from Microsoft in a long time."
It seemed a very fitting way to bracket one of the most challenging but also rewarding periods of my career: one that had its roots and the fertile soil for its success in my friends and former bosses Bill Hilf and Sam Ramji. They created space for me, the latitude to go out and figure out a way forward for Microsoft and open source, by first listening to customers, developers, and sys admins face-to-face.
That opportunity culminated in my becoming the first person in the company (but not the last!) to hold the title "Director of Open Source Strategy" and shipping the first company-wide statement of policy and position on open source.
But, by this time, you've probably figured out something's changed. I've moved on become Vice President for Corporate Strategies at Greenberg, Quinlan Rosner Research.
There are a few things I have always gotten excited about: technology is one. Politics is another. Learning new things is a third. These add to a strong desire to spend all of my time playing MMORPGs. But since that isn't economically viable, they fortunately also add to up a consistent interest in understanding interesting, often controversial, convoluted, and conflict-ridden-situations and figuring new ways forward.
I did this in the public sector, working on community policing, where I sprinkled in some work on political positioning, messaging, and communications. And then I brought that background to Microsoft ten years ago.
Greenberg Quinlan Rosner connects all the dots in a new and exciting way. The founder, Stan Greenberg, is widely known for being the pollster and strategist for Bill Clinton, Tony Blair, and Nelson Mandela. GQRR has a big political consulting practice, and a smaller (but expanding) corporate consulting practice. Continuing and accelerating the growth of the latter is my new job.
I've been around Port 25 since its very beginning. Pre-beginning, actually. I owe a huge debt to everyone inside Microsoft but, even more importantly, outside Microsoft who helped make it what it is today.
My new boss, Jeremy Rosner, was the subject of a movie called "Our Brand is Crisis." Port25 will always be with me as a powerful and tangible part of a big shift from "Microsoft and open source" looking more like a "brand" that equals "crisis" to one that looks more like...well, like Port25. Which is what it should be.
So...thanks. I certainly still expect to be engaged on issues of openness and technology.You can now find me at Greenberg Quinlan Rosner.
by MichaelF on February 15, 2007 06:29pm
Two updates this week but I also have some bigger news to report:
Starting February 10th, any registered user can start their own Codeplex Project! Check out the details here: http://www.codeplex.com/Wiki/View.aspx?ProjectName=CodePlex
by MichaelF on September 15, 2006 03:37pm
Today Channel 9 posted an interview they did with Bill to discuss a number of topics, or to use Charles' words:
"...to talk about, what else, open source software and Microsoft's position on it. Sure, we have shared source, etc, but what is Microsoft doing in the open source software space? Why do we have an open source lab, what going on there, and what was Mozilla doing there recently? Bill and team have a lot of respect for Channel 9 and created an off-shoot that targets the open source community, called Port 25 (http://blogs.technet.com/b/port25/). What is Port 25 and why?"
It's an interesting interview and you can see it here:
http://channel9.msdn.com/showpost.aspx?postid=235278
-michael
by admin on May 17, 2006 02:05pm
VC Summit 2006
Last week I attended the Microsoft VC Summit at our Silicon Valley campus. Before Microsoft and IBM, I helped to build four start-ups, three in the Bay area, so spending a day with a couple hundred VC folks talking about industry trends and business models and in general networking with some great people, was a lot of fun. I did a Q&A onstage with Scott Sandell from NEA on Microsoft, Open Source, our strategy and the relationships to the venture capital community. One of the questions that we spent a good deal of time discussing was the impact of open source software to the venture community. It was an interesting discussion about defining and measuring a successful open source software company. In my opinion, many of these companies are either evolving or starting out with business models that incorporate open source ‘components’ with commercial components (Greenplum is a good example of this), largely because selling support and services for non-differentiated commodity software is not proving to be a sustainable revenue generating model for most of the commercial OSS companies.
Steven Weber’s ‘The Success of Open Source’ discusses this in detail, and Stephen Walli and Matt Asay have been blog-debating this recently as well. I’m interested in business models and I’m interested in analyzing the history of business models. I think one aspect that is often left out of this discussion is that some of these OSS companies have been around for a while, so there is a reasonable history to look back at and measure. Red Hat and MySQL were both founded around 1995 (if memory serves me), and many others can be tracked back six, seven plus years as well. So the question that was discussed at the VC Summit, as well as just this week in the blogosphere, is what qualifies a commercially successful OSS company, and (importantly for investors) how do they rank comparatively to other commercial software companies that VCs may be considering as a potential portfolio company? These venture specific conversations were, of course, very much focused in benchmarking revenue and profitability. I talk a lot about the evolution of commercial and open source models and I think this type of analysis will influence the evolution as vendors, customers, and the investment community start to take a realistic look at the pros and cons of the model. For more information on the VC summit, Don Dodge has a good summary here.
by jcannon on July 06, 2006 05:42pm
Linux Format reported on Port 25 recently with the tagline “Reports of snowballs seen in hell as Microsoft offers to work with Linux developers,” which I thought was funny. It’s apparently getting even colder down there as we’ve now announced an open source project that adds support for ODF to Microsoft Word 2007 ("Microsoft Expands Document Interoperability").
A few months ago I started working with Jean Paoli, whose leadership on Interoperability at Microsoft is steadily moving product teams toward the goal of consistently delivering high-quality interop. Brian Jones notes this in his blog but doesn’t call out Jean by name. You can be sure that you’ll see more of Jean’s handiwork in the coming months and years.
During the time I’ve worked with him I’ve been greatly encouraged by his commitment to openness in documentation and in implementation. The Open XML Translator project is a great example of this – it’s an open source project hosted on Sourceforge.
I couldn’t help but hop over to Slashdot and check out the reactions to the news – and as usual there was a mixture of the rational and irrational, hope and fear, insight and suspicion of conspiracy. It’s worth making one point over and over.
The Open XML Translator is an Open Source project. The Open XML Translator is an Open Source project. The Open XML Translator is an Open Source project.
By definition it can’t conceal its implementation, is open to experimentation, modification, and commercialization (it uses a BSD license), and is owned by the community.
If you think it needs improvement, then improve it. If you think it doesn’t matter, ignore it. But above all, really think about it and what it means that we’ve taken this step before reacting reflexively.
This is actually something new and different.
by Peter Galli on December 15, 2008 04:34pm
The feature complete Release Candidate for Eclipse4SL, an open source, feature-rich RIA application development environment for Microsoft Silverlight in Eclipse, has been released, and can be downloaded here.
This new and important milestone includes a number of new compelling features, including full implementation of the XAML Editor with auto preview; resource and project management; wizards for adding Silverlight project items, including support for JavaScript; and extensibility.
Guidance will also be given in the coming weeks on building Silverlight applications accessing Java Web Services using REST, SOAP and other standards
As planning is ongoing, the next few weeks will be focused on the final sprint to Spring 2009, which includes support for the Mac and an improved C# experience.
by MichaelF on January 29, 2007 04:15pm
In December, Jamie posted a call for questions in the spirit of Festivus one of our favorite secular non-mainstream holidays (aye, we be talkin like pirates on September 19th too matey). Here is the result, or at least the first part of it.
We didn't get to all of the questions in this first pass, but we will be posting the continuation of this conversation early next week. Let us know what you think, if you enjoyed this we'll be happy to do it more regularly.
by Frank Chism on October 20, 2006 03:06pm
"`When I use a word,' Humpty Dumpty said, in rather a scornful tone, `it means just what I choose it to mean -- neither more nor less.' "
Through
the
Looking
Glass
Lewis
Carroll
Where’s the glory? I work in the cluster business. I can tell you that all too often I have felt like Alice trying to hold a conversation with Humpty Dumpty in Looking Glass Land. This usually occurs when I’m talking to someone new to cluster computing or someone who comes from a different tread of the industry than I do. My roots are in a thread that used number crunching to mean serious floating point arithmetic done by Fortran programs to simulate physical processes. Of course, some of the support routines and tools and even the operating system might be written on C, but Fortran ruled. Imagine my surprise when I found there was a ‘Number Crunchers Users Group’ in Seattle and they got together to discuss using spreadsheets. “Now where’s the glory in that?” I thought to myself.
Time marches on, but technology runs as fast as it can just to stay in one place. Fortunately for me, the object oriented police have provided me with just the right jargon to describe my predicament. Just consider that in any modern object oriented language it is possible that + can mean any number of things. Humpty would be proud. In OOP + means just exactly what the developer chooses it to mean. This is called overloading an operator. That may be OK for a compiler, but what about me? When I use cluster I am thinking of something that descended from the original Beowulf. No, not the King of the Geats. I mean the seminal work of those oft sung NASA nerds who put together the first Beowulf compute clusters. When I say nerds, I am here to praise cluster creators, not heap dirt on them or their work. After all, they ain’t dead yet.
For example, I work for a company that has several cluster offerings. There’s failover clusters, and load balancing scale out clusters, and my baby compute clusters. Now that’s overloading. You can usually tell what kind of cluster we mean by the type of work we talk about feeding it. If you had one type of cluster in mind and I had another and we kept talking long enough we’d either figure out the root cause of the confusion or dismiss our conversational partner as an idiot.
But wait. It gets worse. Within my own little compute centric world, two new terms have come into common usage. They are farm and grid. So how do I tell a farm from a cluster if both are eating compute intensive programs? And worse yet, how is a cluster or a farm related (or not) to a grid? I was recently told by a co-worker to not tell our customer that he had a cluster, because as far as he was concerned it was a grid. This is proof that technical correctness is not nearly as important as political correctness. As in politics, so in life.
I can’t claim to have invented farms, but I can certainly claim to be one of the first of the render farmers. I was working at an early Computer Generated Images (CGI) site that was falling behind schedule for a major (OK, it was a big deal to us) Hollywood movie. If we were to finish in time for the planned release, we needed to get our CGI effects generated at just about twice the rate we were running at on our current machine. Fortunately the little ol’ mainframe we were using, a Cray-1, had just been superseded by the Cray X/MP, which had two CPUs instead of one and each CPU was about 50% faster than the Cray-1 CPU. In an example of embarrassingly parallel render farming, we ran odd numbered frames on one thread, even numbered frames on another and ran a third thread to collate the frames and send them to the camera.
I can’t be blamed for grid at all. Well yes, some of the computers my company sold were ‘on the grid’, but I never thought of the grid as anything other than a route for users to do cool things with our machines. In fact I wasn’t sure that grid was anything other than a buzz word used to get NSF funding. Now, thanks to the efforts of the hardworking and unpaid volunteers at Wikipedia, I have at least one fixed mark to guide my wondering barking.
If a cluster on the grid failed over and no one was there to farm it, would it make any sense? So, can we all agree on one set of definitions for clusters (several flavors to be sure), farms and grids? If not, I’m sure I’ll hear from the more assertive of the Port 25 readers and perhaps we can reach a group consensus and I can start quoting the group mind of an entire community in defense of my own use of these terms without sounding too much like Humpty Dumpty making up meanings as I see fit.
Cluster: Making more than one computer behave as a single resource.
Failover or High Availability Cluster: A cluster specifically designed to perform functions in a manner that makes the service it provides continuous, even in the event of individual computer failures.
Load balancing or Scale out Cluster: Generally a high availability cluster that in addition to offering resiliency against individual computer failures also offers addition ability to deliver more of the intended service.
Compute Cluster: A cluster that is built as a single unit and treated as a single system and tuned to perform compute intensive tasks either as a capacity engine, that is to run lots of single node jobs or many low scale parallel jobs, or a capability engine, that is to run much bigger parallel jobs than a single node can accommodate.
Compute Farm: A cluster that uses a collection of computers, generally in a centralized location, to run many similar jobs in parallel for improved time to completion of a particular process. This is very similar to a Compute Cluster in capacity mode but the farm is not necessarily built to look like a single system.
Compute Grid: A heterogeneous farm that is spread out across a wider network or even the Internet but more importantly that is controlled by and conforms to the standards, concepts, and tools originating in the Global Toolkit. It can be used in both capacity and capability mode but is generally a distributed collection of resources, not a single system.
I tried to turn the handle but—
That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my proposed definitions and how they can be improved. Other items on my blog-fodder list are ‘The Parallel Imperative’ and ‘What the Heck is Parallel I/O Anyway?’
So, never stop studying and I’ll blog at you later. - Frank