Tuesday, Jun 24, 2008

Improving qualitative metrics: adding comparisons

Metrics & Validation by Austin Govella

Improving qualitative metrics: adding comparisons

Clarify the "compared to what", and all of a sudden your whole team is now answering the same question.

2 Comments

You’ve clarified the purpose to your scale, specifying what question you want people to answer, but there’s still an unanswered question floating around.

Let’s say you’re on Amazon, and you’re rating a book. You answer the question “would I want to read more books like this?” You say yes, but in your head you’re running a comparison.

Do you want to read more books like this? Compared to what? Cookbooks? Stephen King? Naked Lunch? Don’t Make Me Think?

If you’re in a group and everyone is trying to answer “should we launch this”, differences in individual experience—their unique “compared to whats”—make it almost impossible for everyone to discuss the same issues.

At CIM, we have an ace team of people who have worked on good and great products from all over the country. Asking someone from AOL “would you launch this” reveals a different answer compared to someone from a series of small engineering-focused start-ups.

To make sure everyone answers the same “compared to what” question, specify what you’re comparing to.

On Fancast, you have the ability to search for actors, tv shows, and movies. If we ask the team “would you launch our search feature”, everyone needs to know who we’re comparing to.

For Fancast, we might compare our search to IMDB, Fandango, or Yahoo Movies. The final comparison is really less relevant than having everyone answer the same question.

“Compared to IMDB’s search, would you launch Fancast’s search function?” That’s better than “based on your myriad, varied, and wildly different backgrounds, would you launch Fancast’s search function?”

Talk About "Improving qualitative metrics: adding comparisons"

Register or Log In to comment

Jon Moore

Jon Moore said:

An interesting question we often ask is: “would you launch this compared to what we currently have in production?”

Sometimes it’s enough to just be better than yourself. :)

Wed, Jun 25, 2008 at 12:03 AM

Austin Govella

Austin Govella said:

I’ve been mulling this over (for a month), but I disagree. Sometimes better than yourself is all you have to launch, but it’s never enough, especially for commercial products.

Jeff Patton has some good analysis as to why this is true on his blog, Agile Product Design (12 emerging best practices for adding UX work to Agile development).

Good UX depends on how the experience works in context, and these very important contexts include competing experiences from both direct product competitors as well as analgous experiences. (E.g. Flickr manages photos but offers an analog you could use for file management.)

But as long as everyone answers the same question, then you’re already way ahead. :-)

(Isn’t this why the user story specificies the user and the why?)

Tue, Jul 29, 2008 at 02:34 AM

Register or Log In to comment

Hot topic? Subscribe to a feed of this post's comments.