More Background on the SBS Community Survey and answers to your questions

re: More Background on the SBS Community Survey and answers to your questions

David Schrag — Fri, 13 Jun 2008 07:36:00 GMT

Kevin, I'm afraid you've entered the world of "arguing with Schrag." As many people will tell you, I'm like that little lap dog that sinks his teeth into your pants leg and won't let go no matter how hard you try to shake him off.

"There is [no] limitation on how many times you can stuff the ballot box."

-- I'll admit that it's hard to imagine anyone trying to answer this survey more than once or twice, but the fact that it's possible for someone to write a script that would create hundreds or thousands of responses automatically makes all of the numerical data suspect. I would do this just to prove a point, but sadly I lack the programming skills.

"The Windows SBS Community is whatever you define as the SBS Community."

-- So what is the point of generating an annual satisfaction rating of something that no one has defined or can define? I don't see any reason to ask this question year after year.

"We intentionally made the questions ambiguous."

-- I don't think that's a generally accepted best practice in survey design. See the section "Keep your focus" at http://www.ccs.uottawa.ca/webmaster/survey/best-practices.html. See also the "Overall Considerations" on page 6 of http://wp.bitpipe.com/resource/org_1027448665_672/Online_Service.pdf.

"If this statement were true, than the average sat numbers would be skewed year over year."

-- If WHICH statement were true? But that's beside the point -- I believe the SAT numbers *are* skewed year over year. When I took the SATs, they had to perform some pretty complicated statistical tricks to convert the raw responses into the 200-800 point scale that we're so familiar with. I know they've changed the scoring since then, but I'm pretty sure that there's no simple relationship between the number of questions answered correctly and the official score.

"What is wrong with [question 5]?"

-- 1) As stated above, if there is no common understanding of what the community is, how can there be a meaningful rating of satisfaction?

2) The fact that the overall satisfaction rating can be virtually unchanged while the individual component ratings swing back and forth is further evidence that the overall satisfaction rating doesn't tell you anything interesting.

"Search Engines is one of the top identified Community Resources under questions 15 - 18."

-- How is that possible? At https://connect.microsoft.com/SBSCommunity/Survey/Survey.aspx?SurveyID=6295, search engines are not included as a Community Resource. The choices are:

* Windows SBS Newsgroups

* Microsoft owned blogs

* Third party sites / web forums / blogs

* Microsoft Product Support Services

* Microsoft.com Support knowledge base

* Windows SBS 2003 on Microsoft TechNet (offline and online)

* Windows SBS Training / Webcasts / Events / Chats)

* The Windows Small Business Server page on www.microsoft.com

* Windows SBS User Groups

Maybe search engines appeared as a choice in prior years, but not this year.

"When you do not encourage people to provide us feedback it is directly coorelated to low responses."

-- I stated a hypothesis about why the response rate was low, and you were able to disprove the hypothesis with data. That's how this is *supposed* to work. As for my lack of encouragement being related to a low response rate worldwide, my dear man, you give me far too much credit.

"What we are tying to find out is are some people not even going to TechNet. If somoeone marks it as a 1 or 2 and then says in their comments, "I don't even use TechNet anymore. Every time I have read the content, it has bbeen useless / stale/ not detailed enough/ etc...." this is uesful data to us."

-- But if you ask people to give you a rating without requiring them to tell you whether they used the resource, you are not getting useful data. Suppose someone gives TechNet a 1 or 2 but says nothing at all in the comments about TechNet. Do you interpret that to mean that they have used TechNet recently and have found it lacking, or that they were so turned off by their experiences with TechNet a year or two ago that they don't bother to look there anymore? And if you don't know how to interpret the answer, how do you know whether to spend your efforts on making further improvements to the content on TechNet or on marketing the improvements you've already made to bring people back into the fold? And if you were going to do both anyway, what's the point of doing the survey?

"This Survey has has a direct impact on the way that we engage with our community and ultimately has made life easier and not harder for the community."

-- But the examples you just gave were about interactions with the MVPs and the user groups. You didn't do a worldwide satisfaction survey to find out if there were problems with Vista SP1 and WSUS, nor should you have. Furthermore, any dissatisfaction with how patch releases were handled in the past was dissatisfaction with Microsoft's *products* and *procedures*; it had nothing to do with the *community.*

Last year you said "What is amazing is that if you look at the answers to the two questions below and combine the results, over 50% of the respondents listed third party sites/ web forums / blogs as their number one or two resource. All of the MS owned sites were very low comparitively. Now, this is not to say that the respondents do not value these MS Sites and resources, just that they do not rate them as their number one or two resources. I think this is very eye opening."

-- So once your eyes were open, what did you do about this (if anything)? What could you do? What are the responses telling you? Do these results concern you? Should they? Would you necessarily be doing a better job if Microsoft resources got more number 1's and 2's? Maybe it just means that the kind of information people seek most often is the kind that does not lend itself to the Microsoft resources. You have a lot of data here, but you have no information.

Two years ago you promised some significant changes to the SBS community page.

-- Why not ask specifically what people think about the changes to the page? Or ask "what is the URL for the SBS Community Page?" and see how many people know the answer? (I sure don't. And you want some irony? I just Googled [sbs community page] and the first hit was www.microsoft.com/windowsserver2003/sbs/community/default.mspx. What do you get when you go to that link? You guessed it: "We are sorry, the page you requested cannot be found." The number two hit on Google gave me the right URL: http://www.microsoft.com/technet/community/en-us/windowssbs/default.mspx.) I note, by the way, that at the bottom of the page there is a usefulness rating survey. Does anyone ever fill that out? If so, what kind of scores are you getting?

I don't mean to knock your intentions to get feedback. Feedback is great, and I'm thankful that you're interested in getting our opinions. But if all the good stuff is coming from the open-ended comments, then you should (a) toss the other questions because they are a waste of time and (b) pay as much attention to the open-ended comments that are spread around in blogs, blog comments, Yahoo groups, and numerous other venues throughout the year. A snapshot of comments collected in June does not deserve any special attention.

Your friendly neighborhood sourpuss,

David "There He Goes Again" Schrag

re: More Background on the SBS Community Survey and answers to your questions

Kevin Beares — Thu, 19 Jun 2008 20:51:21 GMT

Hey David,

I decided to just let you keep your opinions and I will keep mine. Not that I do not think that your opinions do not have merit, I just don't agree with all of them and that is what makes community so cool. We can all have our own opinions. I am getting what I need frin this survey and from our convesation. I have almost 600 responses so far. I will leave it open for the rest of the month and then spend July going over the results. I promise to share my conclusions on what he heard from the data.

I will leave you with these thoughts David.

From reading your comments, it seems that you put a lot of value in numbers, metrics, etc.. I value something different. I value the human apsect of research. I majored in Social Science research at the U of Md at College Park. I don't spend a lot of time getting hung up on questions being perfect. I focus on looking at the data and spending the time trying to understand what it may or may not say. I make conclusions based on the data and make adjustments to the way we engage with the community. If I get it right or wrong, we may or may not see a change in how people rate things or what they say with their comments. I can only continue to keep looking at the data. If you think that this survey is the only data point, you do not know me or even have a clue as to what I do as a community lead. All I get is data, every day. This is just one way to collect a bunch in a single time period. What is interesting is the coorelation of the feedback in the survey and the thousands of data points I hear all year long.

The bottom line in my role is simply this. Do something and don't just be complacent with what you have done up til now. So, I will continue to run the survey and make tweeks that I think are necessary, but as long as the data gives me something to act on, I will.

Remember, community is not something you can control. It is something that you can merely try to have some influence on. It is based on relationships, not tools and websites. The tools and websites are merely ways to facilitate a conversation with the community and the community with itself.

I hope that I can have some influence in making the SBS community a better place for people to hang out, to get answers, to develop relationships, improve their technical skill, and hopefully pay it forward.

Thanks for listening.

Kevin

re: More Background on the SBS Community Survey and answers to your questions

David Schrag — Fri, 20 Jun 2008 03:22:06 GMT

FWIW, Kevin, I majored in psychology as an undergrad, so I have an appreciation for social science research methods. Also, my wife got her doctoral degree in education with a focus on qualititative research, so I'm not unfamiliar with its benefits (and drawbacks).

What I DON'T value is numbers, metrics, etc., that have no meaning. You are familiar, I assume, with the concept of false precision. (For those who are not, one example of false precision is saying that a political candidate has a favorability rating of 64.462% ... with a margin of error of +/- 3%.) The problem I have with many of the questions in your survey is related to the problem of false precision. Because of the questions' ambiguity, lack of inter-rater reliability, and other methodological flaws, the responses and subsequent analyses may lead you to believe things that are not "true" -- to the extent that any set of opinions can be said to have "truth."

All I'm saying is that if what you value is qualitative data, then don't bother to collect the quantitative data. And if you do care about quantitative data, then do what it takes to collect it properly.

And, says my wife, even when doing qualititative research you can't have ambiguous or otherwise flawed questions. Poor questions generate poor data, no matter what kind of data you're collecting.

David

re: More Background on the SBS Community Survey and answers to your questions

Kevin Beares — Fri, 20 Jun 2008 20:28:51 GMT

David,

I think you continue to miss my point about the data. While false precision could be a factor in the survey, it does not necessarily mean that it is. This is why the survey is set up to gather both quantitative and qualitative data. If they two do not jive than the quantitative date is thrown out. If there is a correlation than the data is looked at more closely.

I would argue that the questions, while slightly ambiguous are not that ambiguous and inter-rater reliability is a bunch of hooey. While every rater has a different scale, it is still important. What I am measuring are people’s own perceptions of how they rank something. If someone ranks something a 1 (No Value), that is valid data. If a given questions hovers around a rating of 1 or 2, I look for supporting comments. If there are no comments to support it or to give me to act on, I can do nothing about it even if it I think it is valid, but the number is certainly a good indicator to me that I should be looking for comments that would probably be primarily negative regarding the low rated resource.

Seriously, we could go on for weeks on this. We will not agree on this. I think I have said my peace and you have as well.

Let’s move on to other topics.

How about telling me what you think the SBS product team could do to improve your experience in the community? Do you think we should move to using Web Forums? Do you think we should have two entry points to the same discussion? In other words, you can either access the same conversations via a newsgroup client or a web forum UI via Internet Explorer (Or whatever Web Browser you prefer).

Send me your comments, ask me to blog about something specific.

Thanks again for your passion and your feedback,

Kevin Beares

Community Lead

Windows Server Solutions Group (WSSG)

re: More Background on the SBS Community Survey and answers to your questions

David Schrag — Sun, 22 Jun 2008 03:02:50 GMT

Happy to continue the conversation, but this time on my turf: http://davidschrag.com/schlog/334/what-i-want-from-the-sbs-community.

re: More Background on the SBS Community Survey and answers to your questions

Kevin Beares — Mon, 23 Jun 2008 05:37:26 GMT

Thanks, but no thanks David. I have run out of steam. I have a lot of data mining in my future and a long awaited vacation in one week.

I was hoping that you would take my questions seriously and not answer my questions with questions. If you have some value that you think you can add to the conversation, I am all ears. Until then. have a great summer and hopefully, you are spending some time trying out SBS 2008.

Take care,

Kevin