A few days ago we invited y’all to submit questions about certification development. Steve Maier asked “How much do you look at the comments that people put into the exam?” This is a great question and one that gives me the opportunity to tell you lots of good stuff. First I have to explain the difference between beta exam comments and live exam comments.
Most exams that we create go through a phase that we call beta. The beta exam is available for a limited time on an invitation-only basis, which allows us to collect data about each item and determine which items distinguish a qualified candidate from an unqualified candidate. After the beta phase is complete, we review the data about each item and decide which items we are going to keep for the live exam (more details on this process will be provided in a future blog). Scores for the beta candidates are then determined based on the items that we decide to keep.
Just like during a live exam, the candidates who take the beta exam can submit comments about each item. We read every comment submitted during the beta phase of exam development. These comments are one of the data points that we consider when we are deciding whether to keep or kill an item during this phase. If candidate comments indicate a technical flaw, multiple correct answers, no correct answer, or lack of clarity in an item, we will review that item and its associated comments with subject matter experts. These beta comments are SUPER IMPORTANT to help us understand what’s going on with an item, and we depend on you submitting them. You should also be aware that we have no way of knowing when you mark an item for review but don’t have time to provide feedback on it. Because we have to limit the time allotted for comments for security reasons, be sure to comment on the items that you believe have the most critical issues before you comment on items that might have minor issues.
[Tangential side note: Folks who are invited to take the beta version get to take the exam for free. Yes, that’s right, I said free. Click here to find out more about how to participate.]
Unfortunately, we cannot review all the comments that you submit on live exams. There are simply too many. We do, however, read a sampling of candidate comments from each exam as part of our annual review process.
At this point I’ll bet you’re saying, “How am I supposed to submit feedback about a live exam if you don’t read my comments?” That’s easy. If you have feedback about an exam, send an e-mail to mcphelp@microsoft.com. If you are concerned that a specific item is technically inaccurate, you can submit an Item Challenge, and we’ll look into it.
Well, that about covers it. Funny how such a short question can lead to such a long answer!
If you have other questions about exam development, add them here as comments, and we’ll work the answers into our posts over the next few months.