Monday, Jun 23, 2008
Improving qualitative metrics: adding purpose to your scales
Add clear, specific purpose to your rating scale help and your ratings actually mean something. Specify: why are you measuring this?
Rate this 1-5. What does that mean? We know 1 is most likely bad and 5 is most likely good, but what are we rating? Why?
Flat scales don’t tell you much more than someone thought something. They don’t tell you what they thought.
For heuristic evaluations, it’s nice to assign a score to each item you evaluate. At Comcast Interactive, we had this long discussion about heuristic evaluations. Should we use a word scale (good, ok, bad)? Should we use a production-focused scale (blocker, needs work, launchable)? Should we use numbers (1-3)?
The problem with good, ok, bad, and 1, 2, 3 is that they’re both flat scales. They don’t tell you much other than the IA+Usability team evaluated the product.
The production-focused scale communicates more clearly. In this context, the purpose of the heuristic evaluation is to highlight the really bad and what needs fixing so that business can launch the product. A blocker blocks launch, so the severity of that problem is pretty clear.
So, for any scale, if you determine why you rate something, the purpose, then you will have ratings that support what you’re trying to do.
Working in a group, it’s critical everyone answer the same question.
Let’s say three people perform a heuristic evaluation of the same product. One answers “is this usable?” The second answers “is this a good experience?” The third, “is this launchable?”
You can’t compare and discuss differences in answers because you don’t know what the differences mean. You can’t even agree on the similarities. You don’t know what they mean either. If all three answer the same question—is this usable—then you definitely understand differences in the answers are worthy of discussion.
Talk About "Improving qualitative metrics: adding purpose to your scales"
Register or Log In to comment