I can't help but observe the "discussion" underway with respect to spreadsheet interoperability that Rob Weir has started. Essentially Rob is complaining that Microsoft didn't implement the formula namespace of OpenOffice.
For the chair of the committee to post vitriol like this about the implementation of his own format raises a number of very concerning problems.
I'd like everyone reading the post to know that Rob was invited to participate in the DII events leading up to the SP2 release, and offered the opportunity to test the beta software specifically for the purpose of providing feedback on the implementation. Normally the chair of group of the standard being implemented would jump at the chance. Rob didn't, electing instead to wait for the shipping version and then claim that it is somehow deficient to other ODF implementations that he has deemed suitable for his purposes.
Does it make sense to have a chair for the ODF TC whose apparent mission is to create a caste system for ODF implementers? Do we really think Rob, who debates whether the tough (and publicly vetted) implementation decisions of his constituents are "malice" or "incompetence?" – is this the hallmark of a leader in the standards community striving for innovation using open technologies? Is this the characteristic that OASIS wants to promote in the development of technology standards? In Rob, do we really have a person capable of operating in a vendor-neutral forum? If departments within 18 various governments really do use ODF as their standard, should we be comfortable with an ODF TC chair that is trying very hard to discredit and divide its supporters?
Is it time for Rob to step down as chair? I think so.
I'm not saying Microsoft (or anyone) should be the chair instead, but I am saying that Rob is unfit as a leader given his inability to separate his personal venom from his role as a leader in driving the standard forward. It seems like a better approach to empower people on the ODF TC who have a long-term view of the need to enable interoperability, and to move those with more short-term vendor-oriented agendas to the side.
John Head is on point with this post. eWeek seems to be fine with SP2.
As far as I can see, the only thing that Rob is really demonstrating here is that the "grossly inadequate" formula support of ODF (those are the words of David Wheeler, leader of OpenFormula, read on for details) is causing problems with vendors implementing the standard. He instead resorts to scoring implementations based on a percentage of common ground, rather than conformance to something written on paper. This gives Rob the freedom he needs to define his own criteria for what ODF implementation is, and who is doing it according to his rules.
Rob seems to be positioning himself as the final arbiter on what is "good" ODF vs. "bad" ODF. OASIS? specification? – Unimportant when Rob Weir can arbitrarily define criteria for what he thinks is good. He's in a position where only he will declare his own ODF preferences as the blessed implementation. It seems that neither the ODF TC nor the spec matter anymore. It seems that ODF is being run by an individual.
Current ODF standards do not support formulas no matter how much Rob wishes it to be so. Implementations of ODF spreadsheets are application-dependent. ODF 1.2 is not an approved standard. OpenFormula is not an approved standard. While it may be that both are on a path to standardization in the future, today they are not. This is a situation that has been known to the ODF TC for more than 4 years, yet no solution based on an approved standard (other than Open XML) has been found. These are all indisputable facts.
In his post, Rob proposes using "legacy OO namespaces" (also declaring OpenOffice as the "current convention"). Rob's suggestion to use "legacy OO namespaces" is a reference to a vendor's product and indicates favoritism to a particular implementation. The defender of "precise, repeatable, common" seems to be abandoning that hill, hoping instead to claim for his own the dialog that Microsoft has been conducting for a long time: Interoperability requires the participation of many, and will not be defined by a standard alone. Doug covers that pretty well I think.
The irony isn't lost at all. This is the same guy who went to such a length to chastise Open XML for its undefined list styles and compatibility settings. For some reason his expectations of Open XML seem to be somewhat higher than they are for the committee he chairs. For some reason, it is ok for Rob to patch glaring holes in ODF as "current convention" and then complain vigorously about alleged dependence on Microsoft Office for implementing Open XML. This is shameful, hypocritical and warrants corrective action.
It wouldn't be such a huge deal if the tone were constructive or aimed at improving the situation. It seems he is only interested in distancing himself from scenarios where ODF can be used successfully with Microsoft Office (as well as the DII discussions where that implementation was discussed in detail during its development. Funny that he didn't show up there to share this feedback.)
Rob's conclusion on the cause of that problem:
"I was taught to never assume malice where incompetence would be the simpler explanation. But the degree of incompetence needed to explain SP2's poor ODF support boggles the mind and leads me to further uncharitable thoughts. So I must stop here"
Let's just remember that it was the ODF TC which deemed formulas "out of scope," and after 4 years, still have no solution for standardizing the definition of "Sum = 2+2." Rob says "Everyone knows what =A1+A2 means." Really Rob? What does it mean if A1 contains 1, and A2 contains "two"? Would it surprise you to learn that Excel and OpenOffice produce different answers in that case? Which one is correct? This question and a thousand more like it is why formula interoperability is hard work, and not at all the trivial matter Rob claims it is.
During the original discussion within the ODF TC, not everyone agreed with the omission of formulas from the spec… David Wheeler seemed to be pretty clear when commented on this on February 7th, 2005:
This previous comment scares me: "There are from our point of view also no interoperability issues, because the namespace prefix mechanism we have specified unambiguously specifies what syntax and semantics are used for a formula". Here's how I read that: "Every implementation must reverse engineer all other implementations' namespaces (they're not in the spec, so everyone's free to invent their own private incompatible namespaces). Then, every implementation must implement all the syntax and semantics of all other implementations' namespaces for formulas, if they wish to achive interoperability. And oh, by the way, your implementation might not implement the namespace for the document you're trying to load, so you may lose all the formulas."
I'm sure that's not what was meant, but that's how it reads to me. I hope that helps explain why I think that the current formula information in the OpenOffice specification is grossly inadequate."
So… maybe it's too easy, but "I was taught to never assume malice where incompetence would be the simpler explanation." David Wheeler saw this coming over 4 years ago, and yet, OpenFormula is not a standard today, and ODF has no definition for spreadsheet formulas. Rob tries to excuse his way around this in his post, but these comments are made by the committee that he chairs. I'll leave it to you, then, to decide between "malice" or "incompetence" of the poster who would elect to throw his own committee under the bus to get hits on his blog… or fail to take this very good advice.
By the way, it is worth noting the response to this stern (and very accurate) prediction.
"Hi David,Thanks for the concerned comments and all the considerable effort you have put into solving this problem. You're challenging us all to go where none have dared tread before. So go ahead and lead the way. You have the TC's attention. We are listening. As you grind out the grit of your proposal, please keep in mind that we have to fit proposed solutions into the politic of work that has already been done. A politic that represents years of work that is just now on it's way to ratification at OASIS, and beyond to ISO. Keep in mind also that the ISO certification comes at the request of the European Union. Time is of the essence. Ratification perhaps trumps perfection. At least for the moment."
This comment was from Gary Edwards, (he of "cracks in the foundation" / OpenDocument Foundation fame) who eventually left the TC and shuttered the OpenDocument Foundation. I seem to remember some dialog from Rob about Open XML being "rushed" through standardization. Funny how those things come back to haunt you.
I'm very discouraged by Rob's post. As far as I can tell, rob is playing a shell game where only his definition will be good enough for supporting ODF, and that definition will change to whatever Microsoft isn't doing.
This is far from constructive. This is not a way to foster interoperability and industry dialog. This is not a leader for people to follow.
James, try this: http://www.microsoft.com/downloads/details.aspx?FamilyID=e7a23d42-0835-440f-9400-badfe9672b21&DisplayLang=en
and this. http://www.documentinteropinitiative.org/OASISODF1.1/reference.aspx
All available today.
Just one last reply... I think when you say "debug" that really underscores the misunderstanding here. "bug" is the wrong way to characterize the issue. Because formula support is missing from the spec, it's really about design. That's what DII is intended to discuss... design. (seems like you are familiar enough with software development to see the distinction?)
We, the public, read between the lines when the results of the behind the scenes political maneuvering about the ODF debacle came to light in Massachusetts. We read the smear campaign against Peter Quinn in the Boston Globe which did not include any chance for him to respond before it was published, with the resulting furor causing him to quit his job.
We, the public, watched the personal smear campaigns occur when Microsoft ran roughshod over the EMCA/ISO standards body during the OOXML "campaign", totally trashing the reputation of that world wide institution.
We, the public, have seen time, after time, after time, after time, inside and outside of court documents, where Microsoft has performed the "Embrace, Extend, Extinguish" maneuver on defacto standards pertaining to interoperability.
And now we see you, Mr. Knowlton, calling for the head of Rob Weir.
I'm afraid that the reputation of your corporation precedes you Mr. Knowlton, and it carries little credibility when you seek to disparage someone.
My suggestion would be that if you truly desired that the users of Microsoft Office obtain an enjoyable ODF "experience" (and how I hate that word), that you abstain from blogging and expend that energy toward verifying that Office will inter-operates with the other implementations of ODF in the world.
Let's face it. Gray Knowlton is a Group Product Manager from Microsoft which has a long track record of subverting standards. And while Gray is certainly entitled to his opinion, I find it hard to separate those opinions from his role of being the Group Product Manager for the Microsoft Office system.
Given the number of times MS has burned people they should be treated differently than other companies. If someone constantly hit me over the head with a bat after saying they would not. I'm not going to trust them, no matter how much they swear up and down they aren't gonna do it again.
No, I don't trust Microsoft, nor do I trust any of their employees. My number one reason for not wanting to upgrade to Vista or Windows 7 is trust, I don't trust Microsoft. The reason I discourage all the people I work with from using OOXML is trust, I do not trust MS in any way shape or form.
"fool me once shame on you, fool me twice shame on me"
Somebody has to speak the truth. Given the facts that Rob Weir reported as the outcome of his analysis of the interoperability of several spreadsheet programs -- Are you disputing that these are the facts? -- it is difficult if not impossible to conclude anything other than that Microsoft is playing the game of "comply (well, actually, not quite) but sabotage".
You want to see Weir replaced by somebody "independent". Sorry, but we know only too well what "independent" means in Microsoft-speak, from internal documents leaked over the years: a paid agent who can be passed off as independent. There's a _lot_ of ground to be made up in credibility before your proposal can be seen as anything over than a good old power-play.
"Just one last reply... I think when you say "debug" that really underscores the misunderstanding here. "bug" is the wrong way to characterize the issue. Because formula support is missing from the spec, it's really about design. That's what DII is intended to discuss... design. (seems like you are familiar enough with software development to see the distinction?)"
@Gray: What I fail to understand is how this interoperability forum ended up with no spreadsheet formulae from Microsoft Office being readable by any other application? If the specification has an ambiguity or missing feature in it, isn't the best approach to analyse how existing implementations deal with it, and use that rather than creating something that is incompatible with every existing implementation?
As the Microsoft-sponsored CleverAge plugin for Microsoft Office 2007 can swap ODF spreadsheets with other programs, how is it possible to justify Office 2007 SP2 not being able to deal with these same documents? That is a regression - a bug.
If changing the handling of spreadsheet formulae in this manner was a design decision, and it wasn't caught by the DII process, then I consider that a major flaw in the process. If Microsoft has not internally tested their ODF implementation against documents from other ODF implementations, then that is a flaw in Microsoft's test processes, in my opinion.
And yes, I am familiar with software development, and I make sure to test my software against real world implementations as well as specifications as I know the two are all too often different. Any failures would be classed as either a bug or bad design - and something would have to be done to make it work.
Take it on the chin, Gary.
It's plain that Microsoft messed up, all it would have taken is an email to an ODF representative saying; "Hey, this part of the spec isn't documented, how would you do it?"
Criticize Rob all you like for his manner in pointing it out, but somebody was going to sooner or later. Don't try to pretend Microsoft didn't just release a XXXX poor product.
It surprises me that the Openoffice developers can figure out many details of Microsoft's closed formats [this requires a lot of hard work and desire for interoperability], yet Microsoft can't be bothered to attempt interoperability in such an important area with the most popular open source competing product (ie, whose source code they have)? They have the software for testing purposes and a liberal license to go with it. They have the source code. What were they waiting for? And this is even more shocking when you consider that Openoffice was not the lone product where Microsoft failed in this important area. Openoffice compatibility would have simply been the easiest to acquire.
It's difficult to believe Microsoft had an interest in interoperability when you consider this decision they made. Maybe they were trying to make a statement: that interoperability with important open source products (whose blueprints they have) is not something they value whenever the spec does not mandate it.
Further, iirc, Rob Weir pointed out that Microsoft had access to implementations that already worked more sensibly (ie, with greater degree of interoperability). Why did they decide not to leverage that interop work? So people drop interop aids onto Microsoft's laps, it almost seems, and Microsoft doesn't take it up because the spec does not mandate it?
Incompetence or bad will? .. or is Microsoft running out of money and trimmed down their interop division significantly?
>> "..You're challenging us all to go where none have dared tread before. So go ahead and lead the way. You have the TC's attention. We are listening. As you grind out the grit of your proposal, please keep in mind that we have to fit proposed solutions into the politic of work that has already been done. A politic that represents years of work that is just now on it's way to ratification at OASIS, and beyond to ISO. Keep in mind also that the ISO certification comes at the request of the European Union. Time is of the essence. Ratification perhaps trumps perfection. At least for the moment."
>> ...I seem to remember some dialog from Rob about Open XML being "rushed" through standardization. Funny how those things come back to haunt you.
In what way did the author of this posting think this was evidence favorable to Microsoft?
When we compare what transpired, I suppose Rob Weir has a higher standard for the less-than-perfect than does Microsoft.
While Microsoft was not worried about putting all sorts of half-baked information into a rushed OOXML standards process, Rob Weir apparently did want to maintain a high level of quality for ODF (which, iirc, was shown to the public for a much longer period of time than was ooxml before becoming standards).
It would not surprise me one bit to learn that someone from Microsoft in Rob's shoes wouldn't have hesitated to throw in a half-baked formula spec.
BTW, this happened years ago. Since them there has been a lot of formula work that has taken place.
The results speak for themselves, as Rob pointed out. Microsoft's offering did a horrible job at interoperability. I'm still not sure I know what was Microsoft's excuse. I can say this much, if Microsoft would be as generous with the Openoffice community as the Openoffice community has been with Microsoft (in revealing the entire blueprints of their product), then I am sure it is very likely that the Openoffice community will even do some of the interop work on Microsoft's behalf and for no charge. [I have to wonder if Microsoft is not having resource issues.]
The formula namespace mechanism is part of ODF: it is there for all to see in IS26300 s6.7.6. One would have to expect that when the ODF TC decided on it, they would have done so in the full knowledge that an implementer could decide to skip the prefix or to barf.
And that someone using software from implementers who decided to barf, would have formulas that their application rejects: that is just a ramification of the feature existing, a temporary inconvenience until developers catch up. It is a decision the implementers made for us; and indeed, they probably realized when they were doing it that it would block interoperability from people using the namespace prefix in the way the standard requires.
The use of the namespace prefix is a 'should' in IS26300, s 6.7.6
The ball is in the other implementers' court(s), either to skip all namespaces prefixes or to add a line to cope with Microsoft's prefix. If they don't want to cope with any prefix, they need to advertise that they are unconforming in this area.
And if they never want to support any prefixes, they should get ODF 1.1 changed at ODF TC to remove it (or a defect report raised for inclusion in the ODF 1.2 work): it would be paradoxical if this problem were actually caused by ODF having *too much* specification of formula ;-)
If I can defend Rob's position a little, he does state that this is an interoperability issue, rather than a standards compatibility issue: the argument of being incompatible in the name of standards is a tenuous one, especially if you have an extreme short-term view. And this issue has been around for years, so I don't think Microsoft can claim any ignorance that it wouldn't play well with other applications, which does not go well with the Guiding Principles (such as "Preserve User Intent"). And I am also not very convinced by the idea that therefore MS would have to reverse engineer all the formula languages of the other implementations: it has its own well-documented formula language and deciding to wait for Open Formula for interchange is a perfectly respectable position.
At its heart, I think the deeper problem is that starting the formula with a namespace prefix is a bad markup choice by the standard to require: you don't fiddle with well-known notations like that. There should be a separate attribute for it, scoped to the table. Messing up well-known notations is a ratty thing to do, and I think it is positively bad because it creates this kind of consequence for implementers and users. And in XML is it completely unncessary because attributes are cheap and easy.
The "just one more thing" syndrome that ruins standard technlogies, when there is always one more wrinkle or one more thing to go wrong unexpectedly.
Well, just for the sake of completeness. Your pointer to http://www.openxmlcommunity.org/applications.aspx doesn't really help.
As far as I can see that list doesn't make any difference between ECMA376 and IS29500.
And don't tell me there are no significant differences between the two. I also note that defect handling in IS29500 is becoming quite a problem and that some BRM "decisions" are actually damaging IS29500 so there is a need for corection that seemingly is handled outside of the normal processes.
So could you please point me to a list of applications that implement IS29500 and NOT ECMA376? If an application implements both that is fine ofcourse.
And please also show me where I can find a test suite to check IS29500 documents for compliance.
Anyway, ultimately it is up to OASIS to announce their POV on the compliance of SP2's ODF implementation. The practical tests I have seen are however not really promising.
Instead of bitching around, let's all move to productive mode and work on the technical problems by solving them to create true interoperability based on open standards.
Jan Wildeboer (a Red Hat guy)
I just wondered whether it was worth pointing out - the question here doesn't really seem to be about a defining standard in this case. Yes, a bar for which to aim for implementing formulas would have left no room for intuitive leaps or personal interpretations - and such a standard appears to be in the making.
What seems painfully obvious to me however is that a large number of competing implementations appear to have achieved interoperability *despite the absence* of such definition. One of these implementations is even an existing plugin for MS Office. So it is demonstrably achievable.
Given that the implementation of Office SP2 must therefore have been capable of achieving the same, it appears that Microsoft have taken the deliberate step of specifically opting not to interoperate.
So what I would like to know is, on what grounds is this action defensible? Are Microsoft claiming that other companies possess skills of clarivoyancy or communications that they themselves are sadly not privy to? Or perhaps that other companies are more competent at making the software that has been Microsoft's bread and butter for nearly two decade now?
Because to a mildly technical reader such as myself, I'm afraid it appears that Microsoft has absolutely no interest in interoperability. The gesture towards compliance (which itself is being called into question now, I believe) was a result of ongoing legal action against - and public scrutiny of - Microsoft.
If there is a reasonable explanation then I am amenable to persuasion. Unfortunately, the benefit of the doubt is something I'm not prepared to extend to a company like Microsoft.
``Rob says "Everyone knows what =A1+A2 means." Really Rob? What does it mean if A1 contains 1, and A2 contains "two"? Would it surprise you to learn that Excel and OpenOffice produce different answers in that case?''
I've tried and honestly both give me an error (admittedly not the same one). MS Office is Office Excel 2003 (11.5612.5606) and OpenOffice is OpenOffice.org 3.0.1 OOO300m15 (Build:9379). MS Office is an Italian version (I've tried "two" and "due"=italian for "two"), OO.org has French locale (I've tried "two", "due", "deux"). In all cases I've tried all the forms I could think of: two, "two", =two and ="two".
As a professional programmer, I have to say I'm disappointed to see how you respond publicly to what Rob posted. I know that Rob was using words like malice and incompetence, but I don't think you can ignore his call to move forward in the very next paragraph (that you didn't mention here). He also didn't say that you Gary are guilty of malice, or that you are incompetent. He seemed to imply that the way Microsoft implemented ODF support in Microsoft Office 2007 SP2, was either the result of malice or incompetence. So since the attack wasn't on you personally, I fail to see why you attack him personally. If you are worried that because he is chair for the ODF TC, it might look like he is speaking in official function (I didn't get that impression), then you can always attack the ODF TC which deemed formulas "out of scope". You can then rightfully claim that they should have known that this would cause problems down the road. Instead you choose a personal attack in reply to an attack on Microsoft's implementation of ODF in Microsoft Office 2007 SP2. Therefore it looks like you are speaking in Microsoft's name, and the way you launched a personal attack just makes Microsoft look bad. I think you gave an emotional response, and you should in the future think twice before you post something that reflects bad on your employer.
That being said: Rob did some tests. He did those in his own time. Microsoft's implementation in Microsoft Office 2007 SP2 didn't work well with the files produced by other programs that use ODF. Files produced by Microsoft Office 2007 SP2 didn't work well in the other programs. That's what he saw. That's what he mentioned.
You say he speaks for one vendor?
He seems to have demonstrated rather clearly that there are problems with interoperability with the current version of the standard. You agree with that. What you don't agree on is how to address the problem.
If I wanted to work with .xls files, would I look at how everybody implemented them, or at how Microsoft Excel did it?
So when I want to work with odf files, where do I look? I'd first look at the specs, and if it's not in there, i'd look at how it was implemented in OpenOffice.
You argue that this means I would favor one vendor (OpenOffice), whilst I would say that I'm looking there because in the ODF world, that is the most widely used program, and therefore I want to be sure I can at least make my software interoperate with that one. I think that makes perfect sense.
You are correct in saying that the standard is lacking in this department, but you are in my opinion incorrect in saying that therefore the best solution is to invent your own way to implement this. The best solution is the one that works with the type of ODF file most customers will have, and since most of these files will be produced by OpenOffice for now, that is what one needs to be able to read and write. Since Microsoft is a de facto standard when it comes to Office use, a correctly working implementation of ODF the OpenOffice way, would make Microsoft Office the program that produced the most ODF files (that can be correctly used in other software that reads and writes ODF). This would make Microsoft Office the place to look at if you want to implement a compatible program. This to me would seem to be the correct way to do it, and it would prove to everybody that Microsoft's words about interoperability aren't empty.
Secondly, your remarks "solutions that require reverse engineering of proprietary implementations to achieve 'practical interoperability' are not the answer", sounds very empty to me as a programmer. The fact is that if you want to see how OpenOffice implemented a certain feature, the source code is available for free, so there is no need to "reverse engineer" how they did it.
Anyways, that is my opinion as a professional programmer.
My only advice for the day: If people post something that makes you hyperventilate, remember to breathe before you post.
@ChrisR - "Given the number of times MS has burned people they should be treated differently than other companies."
I see. So then we're really not talking about interoperability and standards, are we? Folks who work with standards (particularly those who consume them as a software requirement / request) are disappointed to learn about those who call on ODF as an anti-Microsoft rally cry, instead of a document format Standard freely implementable by anyone.
He's a Microsoft employee, too.