I figured a new blog post would be better than continuing a thread of comments on the previous blog post. Where is the love? The SBS Community Survey is floundering with very few responses.
Results from previous surveys have been posted on this blog. I probably could have done a better job of talking about the action items I took from the last year's survey results. My posts, may have not been to the depth that some of you are looking for, but I have talked about the survey and it's purpose on numerous occassions. Here are some old posts that shed some light on what things the SBS team got out of previous surverys: So, what have I learned so far....., SBS Community Survey Update and Commitments for the SBS Community Lead, I have made some preliminary changes to the SBS Community Page, In response to your feedback, I create a pretty cool little search tool...., The Survey is closing down at the end of the month,
I will try to do a better job this year talking about what action items we decide to take based on this years feedback. Looks like from Viajys comments, there are issues with the SBSC program. To that point. If you have more feedback you want to provide on the SBSC program please take the survey again and keep submitting additional feedback. Because this survey is anonymous, there is not limitation on how many times you can stuff the ballot box. Think of this as Dancing with the Stars or American Idol. I promise we will categorize and priotize the feedback and get this feedback to the people who can effect the most change.
Answers / Responses to your questions:
Question/ comment: "The survey refers several times to the "Windows SBS Community" without defining what that term means. Is it limited to the SBS Community Resources listed in the survey? Is that a generally accepted definition? It would be interesting to know how many people could even list what the SBS Community Resources are without any hints. How many people would consider PSS to be part of the Community?"
Answer/reply: The Windows SBS Community is whatever you define as the SBS Community. I am not in the position to define what our community is. So, no it is not limited to the SBS Community Resources listed in the survey. So, this is not a generally accepted definition or an attempt at a definition. It is what I have defined along with numerous other Community Lead across Microsoft as Categories of Community Resources that could at least be bucketed, rated, and commented on. Think of them as conversation starters. These areas/resources are typically what I point people on the product team at as an area where the community is having a conversation either directly with us, with each other, or us with them. In terms of TechNet and Blogs, it is more one way than two way, but they are generally accepted WEB 2.0 Apps that allow feedback to flow back to us.
Question/ comment: "- The concept of "value" is somewhat vague. Each resource listed can potentially be of great value or zero value, depending on when and how it is used. Some resources are better for general product and sales information, some are better for ongoing education and training, some are better for troubleshooting, etc. If one of the resources gets higher scores than another, it doesn't necessarily mean that the "losing" resource is no good. - Expanding on that last point, it might have been better if you had said "here is a list of 10 questions that someone working with SBS might have. For each question, how likely would you be to consult each of these community resources in search of an answer?" If, hypothetically speaking, you found that there was a 90% chance that people puzzled by an error message would go to third-party web sites but only a 30% chance that they'd go to the MS KB, then you'd know something was very wrong with the KB."
Answer/reply: Maybe you missed my blog post, but my major focus is to get the product team engaged with the community. The survey is a good indicator/validation engine of where I should direct the product team to focus the most attention. I do not take a low score on a specific community resource as a vote to do away with a given resource. I look at the verbatims as well and see if there is a coorelation. Sometimes, we conclude that the community finds little value in the resource, but because it is just that we have not given it enough attention and that they are upset that we have not spent more time keeping the resource up to date.Applying a rating allows us to look for trends year over year. This is a generally accepted practice. The survey folks along with a lot of other Community Leads at Microsoft helped me design the survey and the questions. We intentionally made the questions ambiguous and we also used a very common rating system of 1 to 5 that we use in most other MSFT surveys.
Question/ comment: Your rationale for not changing the questions from year to year makes perfect sense IF the basic survey design is sound. But the basic survey design here is fundamentally questionable because you're not getting a random sample. You're only getting motivated respondents, and you really have no way of knowing which way that will skew the answers. And even if it were a random sample, the collection of the same meaningless data year after year does not make it meaningful. (I'm thinking of question 5 in particular.)
Answer/reply: I will break up my response to this in two parts.
Question/ comment: In my experience, many SBSers turn to search engines when looking for answers. Those search engines may spit back answers from several different community resources at once. Am I really supposed to remember how many times a search engine brought me to TechNet as opposed to a KB article as opposed to something at www.microsoft.com as opposed to a newsgroup post, etc.?
Answer/reply: Search Engines is one of the top identified Community Resources under questions 15 - 18. I understand that a lot of people do not make the distinction on what was the final site that they landed on. What they say is that they use a search engine and RSS feeds to get their information. I make special note of this. This is why I do a lot of work to make sure our team does not spin up a thousand blogs and stay with one so that the amount of information is pooled in one place on the web and search engines discover our blog more often when people are searching for something on our product. Check out what happens to teams who have 10's of blogs. They are not easily discovered and their community suffers because no one can find the answer that they so eloquently posted on their very little traffic blog site.
To your point "and it may be why your response rate is low. Perhaps there are hundreds or thousands of people who have reviewed the survey questions and have decided that it's not worth answering" completely and utterly false. The hit count to the survey is almost an exact match to the amount of responses. The problem is that people are simply not going to the survey. So, when you do not encourage people to provide us feedback it is directly coorelated to low responses, not poor survey design.
Question/ comment: I commend you for trying to get feedback on various resources, but why are you using this tool? If I recall correctly, CSS and the monitored newsgroups have their own satisfaction surveys that immediately follow the incident. If I were the CSS manager, I'd put a lot more stock in comments and ratings from people who had just finished receiving support than from people who may or may not have had any interaction with CSS in the last year. (Just because you tell people to base their ratings on the last 12 months doesn't mean they will.) As for gathering data on which MS community resources are preferred by the community, surely you have a more objective way of measuring that. For example, I assume you know exactly how many people attend each SBS webcast, event, chat, etc., and I know that most of these events ask participants for satisfaction ratings at the end. That must be a more valid indicator of popularity and satisfaction than what you're getting from the survey. I also assume that you know how many hits and inbound links are associated with each KB article, TechNet article, MS blog, SBS page, etc. Again, this objective data ought to be given more weight than the 5-point-scale ratings of some self-selected survey respondents.
Answer/reply: We chose to steer away from asking targeted and leading questions. TechNet, CSS with Newsgroups, Webcasts, etc. already has it's own survey engine to see if an article hit the mark. What we are tying to find out is are some people not even going to TechNet. If somoeone marks it as a 1 or 2 and then says in their comments, "I don't even use TechNet anymore. Every time I have read the content, it has bbeen useless / stale/ not detailed enough/ etc...." this is uesful data to us. It tells us that we are not retaining our customers and we need to do a better job to make better content. I am trying to catch the feedback that TechNet is not getting.
Susan quoted me here and I will put it in again, "Because of the previous surveys, we have made some adjustments in our community engagement. To name a few; we really worked on putting more focus on the Official SBS Blog. Our Sustaining Engineering team runs mini betas with our MVPs before KB's and Bug Fixes are released via Windows Update. We try to get more involvement in our techbetas from our User Groups. The list does go on."
Do you know why Vista Sp1 is not available as an automatic update via WSUS yet? Because of the MVPs participating in our early SE Betas and the relationship that they have developed with our SE team. WSUS had a bug that caused any Cab file over 500 megs to fail to pass a WinVerifyTrust and subsequently, WSUS will continue to try to download the Cab. The fix was created, but not available except if you called CSS. The SBS MVPs helped us push the WinSE team to force this fix as a critical fix on WSUS for 2 months prior to making SP1 available to avoid anyone having this endless download loop condition from occurring. So, actually, this Survey has has a direct impact on the way that we engage with our community and ultimately has made life easier and not harder for the community. SP1 is potentially going to be available via WSUS this month based on our final assessment of the uptake of the critical fix.
Trust me, we do look at all the responses and take action based on the feedback. This is my job. I love my job and I love being a community lead.
Thanks for all of your feedback,
Kevin BearesCommunity Lead - WSSG