Measuring UX from Thinking and Making

UX Health Check presentation at SXSW 2011

Austin Govella — Mon, 14 Mar 2011 01:00:15 GMT

This morning I spoke alongside Fred Beecher and Lou Rosenfeld in a workshop at SXSW Interactive, "UX Research and Web Analytics: X-Ray Insights". Fred showed several case studies where UX and analytics worked together (or against each other in one case). Lou demonstrated both the value of search engine analytics, and how easy it was to begin analyzing your search engine logs today to make useful immediate changes.

I introduced the UX Health Check to the SXSW audience. Unlike previous talks, this was more of an "about the health check" and less of a "how to". (For more of a how to, check out the IA Summit UXHC presentation Livia Labate and I did.)

I uploaded today's Health Check slides to Slideshare and embedded them below for convenience.

The User Experience Health Check

As always, if you have any questions, please let me know. I'd be happy to help you implement the health check -- in part, or in full -- at your organization.

UX Research & Web Analytics: X-Ray Insights

Austin Govella — Tue, 15 Feb 2011 12:17:00 GMT

Do you want your website to meet both your business goals and your users' needs? Understanding what people do on your site gives you the x-ray insight crucial to effective business decisions.

Fred Beecher was kind enough to invite Lou Rosenfeld and I to present a workshop talk at this year's South by Southwest Interactive. We'll provide a practical framework for squaring quantitative evidence and qualitative insights, and we'll see concrete examples from search log and click path analysis. We'll also learn how you can continually measure the quality of a site's user experience.

The talk, "UX Research & Web Analytics: X-Ray Insights", runs Sunday, March 13 at 11:00 AM. (From the talk's page, you can add it to your personal SXSW schedule.)

In my part of the talk, I'll introduce the UX Health Check and show how you can measure the user experience when you don't have analytics to analyze. And, I'll also use click-path analysis to show how you can use the Health Check alongside traditional analytics to measure continuous improvement in your designs.

Lou will be presenting on finding user insights in your search logs, previewing material from his forthcoming book on search analytics.

Take this social media study

Austin Govella — Tue, 07 Sep 2010 15:37:24 GMT

Javier Martin (@mantruc) has launched a study examining your use of social media tools like Facebook and Twitter and email and IM. Support a kindly researcher and take the survey. It'll take about twenty minutes of your time, and you are eligible to win one of two $100 Amazon gift cards, but really, you should take it to help further our understanding of each other. The survey is up onlne: "http://bit.ly/SMstudy":http://bit.ly/SMstudy

UX Health Check presentation from Redux DC

Austin Govella — Mon, 06 Jul 2009 10:13:00 GMT

Livia Labate presented the UX Health Check at RedUX DC last month.

Livia's created a website for the UX Health Check where you can keep up with developments, news, how-tos, and tips. And she's posted the video presentation and slides from RedUX DC.

For those of you too lazy to make with the clicky to a new site, you can make with clicky right here to watch Livia's presentation

The User Experience Health Check, quantitative metrics for the qualitative user experience

Austin Govella — Fri, 27 Mar 2009 10:27:00 GMT

Livia Labate and I presented on the User Experience Health Check at the 2009 IA Summit in Memphis, TN this last weekend.

The UX Health Check is a technique for UX professionals to express the quality of a product or service’s experience and convey it in terms that are familiar to business stakeholders, creating a common ground for discussion and exploration.

I developed the original concept while working with Livia Labate and team at Comcast Interactive Media. It emerged from a real need from a product team looking to convey how the user experience improvements were evolving over time in a way they could show progress to executives.

Livia has started a site at uxhealthcheck.com where you can view the slides, as well as the poster we presented.

We'll be adding more and more Health Check related content to the site over the next several days and weeks (including templates and presentation audio).

Presentation - Faceted seach best practices - Wed, Feb 4

Austin Govella — Wed, 21 Jan 2009 11:23:00 GMT

I'll be presenting with two others Wednesday, February 4, as part of the Taxonomy Community of Practice series of teleconferences presented by Earley and Associates. I'll spend 20 minutes looking at ways to identify and then prioritize user mental models as a way for directing and improving the search experience. For the finale, I'll demonstrate how you can combine these methods to benchmark and the evaluate your search experience over time. The call runs from 1-2pm, EST. Registration is $50. Check the website "for more information and to register":http://www.earley.com/_February2009.asp.

Here's the description from the website:

Faceted search depends on well organized taxonomies. However, in some cases, the organization has not done the upfront classification work needed to fully leverage facets for search and guided navigation. The approach is “let’s just use what we have and let the engine index our content”. That approach may provide some benefit, (if the bar is low enough to begin with) but how can you get the most from investments in tools and technologies and fine tune your taxonomy to produce a better user experience? In this session we’ll review practices around developing taxonomies as they are specifically applied to faceted search. We’ll discuss do’s and don’ts and show you how to get more from faceted search and create an intuitive user interface that will improve usability and result in increased conversions.

The User Experience Health Check

I'm making a firm effort this year to spread the word about the UX Health Check as far and as wide as possible. The Taxonomy Community of Practice Call is this year's first volley. I'll post more information about this process and method here, but look for articles and conference appearances as the year winds on.

However, if your interest is piqued or you're dying to know more now, drop me a line ("austin@grafofini.com":mailto:austin@grafofini.com). I'm looking for as much practice with the elevator pitch and description as I can get.

Weekly usability tests, Nielsen talks about Tivo

Austin Govella — Wed, 30 Jul 2008 03:05:38 GMT

Jakob Nielsen's latest Alertbox shares how Tivo did 12 usability tests in 12 weeks and offers some tips and guidelines for doing the same at your organization. I've actually been hearing about this a lot lately: design teams running weekly or bi-weekly usability tests. It'd be nice to have more companies share their experiences, the how and why and such. At CIM we did something similar in concert with a project using the scrum process. While this is all cool, the article summary really stuck in my craw: "frequent and regular testing keeps the design usability focused". Why would you want to keep your design usability focused? Nielsen probably doesn't mean only focused on usability, but it sure comes off that way. It just feels so 90s. If we all know the design should be focused on the experience what kind of frequent tests can you run to keep your design experience focused? Well? If you were running some kind of bi-weekly validation to keep your team experience focused, what would it look like? Hint: There's a comment box right down... there. Fess up.

Improving qualitative metrics: adding comparisons

Austin Govella — Tue, 24 Jun 2008 17:45:00 GMT

You've clarified the purpose to your scale, specifying what question you want people to answer, but there's still an unanswered question floating around. Let's say you're on Amazon, and you're rating a book. You answer the question "would I want to read more books like this?" You say yes, but in your head you're running a comparison. Do you want to read more books like this? Compared to what? Cookbooks? Stephen King? Naked Lunch? Don't Make Me Think? If you're in a group and everyone is trying to answer "should we launch this", differences in individual experience -- their unique "compared to whats" -- make it almost impossible for everyone to discuss the same issues. At CIM, we have an ace team of people who have worked on good and great products from all over the country. Asking someone from AOL "would you launch this" reveals a different answer compared to someone from a series of small engineering-focused start-ups. To make sure everyone answers the same "compared to what" question, specify what you're comparing to. On Fancast, you have the ability to search for actors, tv shows, and movies. If we ask the team "would you launch our search feature", everyone needs to know who we're comparing to. For Fancast, we might compare our search to IMDB, Fandango, or Yahoo Movies. The final comparison is really less relevant than having everyone answer the same question. "Compared to IMDB's search, would you launch Fancast's search function?" That's better than "based on your myriad, varied, and wildly different backgrounds, would you launch Fancast's search function?"

Improving qualitative metrics: adding purpose to your scales

Austin Govella — Mon, 23 Jun 2008 21:10:09 GMT

Rate this 1-5. What does that mean? We know 1 is most likely bad and 5 is most likely good, but what are we rating? Why? Flat scales don't tell you much more than someone thought something. They don't tell you what they thought. For heuristic evaluations, it's nice to assign a score to each item you evaluate. At Comcast Interactive, we had this long discussion about heuristic evaluations. Should we use a word scale (good, ok, bad)? Should we use a production-focused scale (blocker, needs work, launchable)? Should we use numbers (1-3)? The problem with good, ok, bad, and 1, 2, 3 is that they're both flat scales. They don't tell you much other than the IA+Usability team evaluated the product. The production-focused scale communicates more clearly. In this context, the purpose of the heuristic evaluation is to highlight the really bad and what needs fixing *so that business can launch the product*. A blocker blocks launch, so the severity of that problem is pretty clear. So, for any scale, if you determine why you rate something, the purpose, then you will have ratings that support what you're trying to do. Working in a group, it's critical everyone answer the same question. Let's say three people perform a heuristic evaluation of the same product. One answers "is this usable?" The second answers "is this a good experience?" The third, "is this launchable?" You can't compare and discuss differences in answers because you don't know what the differences mean. You can't even agree on the similarities. You don't know what they mean either. If all three answer the same question -- is this usable -- then you definitely understand differences in the answers are worthy of discussion.

The U.X. health check, a Summit side conversation

Austin Govella — Thu, 10 Apr 2008 11:57:00 GMT

At Comcast Interactive, we've been working on a new method for evaluating our work. What's emerged, we've been calling a UX Health Check, and I think it might have a lot of potential.

So, while we're milling about the Summit with all you great minds, I'd like to talk to you about the Health Check and get your opinions.

What does it do?

The health check is a method for quantifying the quality of a user experience.

It can measure experience at one point in time, or be used in a repeated way to track the quality of an experience over time.
It can be used for any product or service.
It can measure very detailed feature-sets, or very generalized notions of service.

Where'd it come from?

The health check was created to help product management communicate progress and improvement to executives. Instead of measuring design, it measures how well your users interact with your experience.

What are the outputs?

A measurement of where you are now
A measurement of how your experience has improved or not over time
A recommendation for product managers on where the biggest problems and opportunities are

Interesting?

If this sounds interesting, catch me at the Summit, and I'd love to talk to explain how it works and get your feedback.

Unique visitors is a dumb metric

Austin Govella — Mon, 24 Mar 2008 10:13:00 GMT

Lots of organizations try to measure success by measuring unique visitors. Typically, these groups have some sort of advertising-based business strategy where they believe the more eyeballs they reach, the better they’re doing.

But that’s not true.

For sites supported by ad revenue, unique visitors tell you how big your check will be that month. If every visitor is worth $1 in ad revenue, and you have 1 million visitors this month, then your check will be $1 million dollars.

What’s missing, what’s not being measured is whether or not your business model is successful. “But I have a million dollars”, you say. But having a million dollars isn’t your business model.

Let’s say this month, you have 1 million visitors, they all earn you $1 in ad revenue, and you get your check for $1 million dollars. And let’s say your site sucks, so none of them came back, and they told their friends to never go. Next month you have no visitors.

Each visitor still earns $1. And you go to the bank to deposit your check for $0.

What’s missing, what’s not being measured is the behavior your organization relies on to make money. There’s something all those unique visitors do, and it's the doing that's necessary for the success of your business.

In marketing terms, unique visitors measures how many potential customers you have acquired. However, in order to make money, you have to convert those potential customers into actual customers.

Broad metrics like unique visitors and total page views give you broad aggregate data that’s just about useless. What you really want is a behavior-based metric that measures the behaviors that earn you money.

A lot of sites driven by ad revenue need the same behavior in order to be successful. A user learns about the site, they visit the site, and then they return to the site again and again. The visitor's return visits are what really drive ad revenue. Instead of measuring unique visitors, a better measure of your site’s success is return visitors.

Performance metrics should always measure the behaviors your organization relies on to survive. Your metrics should always be behavior-based. Aggregate metrics like unique visitors and total page views don't tell you much of anything.

Evaluating experience design

Austin Govella — Mon, 21 Feb 2005 17:31:00 GMT

I edited this post to clarify how the facets would be used based on a conversation with Paula Thornton.

One of the most limiting factors in Design isn’t the splintering of specialist groups, nor the emergence of specialised vocabularies. We lack common language for discussing Design, for communicating and evaluating the creation of experience.

We all contribute to the overall user experience, but we don’t have a clear definition for what we do: what is experience? We have fuzzy definitions. Several models are emerging (I’m working on one as well), but we still lack an objective means for evaluating experience design. We may not understand it, but we know what it looks like, what it feels like, its general shape. I think we can use several facets to evaluate the resulting user experience:

Personal: How well does the experience relate to the individual user? A conversation with your best friend compared to talking to the clerk at the DMV.
Desirable: How much do the users desire the experience? How much do they want to experience it? How much do they need it? A triple heart bypass versus having your ears pierced.
Enjoyable: How much do users enjoy the experience? A chore versus something you enjoy doing.
Accessible: How accessible is the experience for the user? How understandable, comprehendable, physically available? Climbing Mount Everest versus climbing the curb.
Negotiable: How able is the user to negotiate the experience to better communicate with them? How customizable?

Theoretically, you could survey a group of users to evaluate a given experience in much the same way psychological surveys are performed. A numeric value can then be given to the experience in question. Two quick examples illustrate how you can evaluate different kinds of experiences using these facets:

Heart surgery: A very personal experience and very desirable (if you want to live), but not enjoyable.
Solving a rubix cube: A very personal experience, and very negotiable, but not very accessible. Anyone can try, and almost everyone can manipulate the cube, but very few can solve them.

Evaluating successful experiences

A successful experience evaluates differently for different purposes. Successful sales and education require the almost perfect transmission of mental models. The better sales or education evaluate across all five facets, the more effective the sales and education will be.

Compare learning on your own to learning how to program your VCR, and you can see why so many people have learned how to do the latter. I think one can say that communicative experiences, experiences where the primary goal is to communicate, should evaluate highly on all five facets in order to be successful.

Other experiences evaluate differently for success. The success of jail time’s deterrence requires it be neither desirable, nor enjoyable. And we work at making it undesirable: restricted freedoms; small, overcrowded quarters; and a culture of violence and racism. But, in the eyes of someone trapped in a recidivist culture, jail time loses many of its deterrent features.

None of these facets are mutually exclusive. If we assign values for a given experience, there’s an interaction among these values, but no zero-sum interplay. Improvement along one facet may improve or worsen an experience’s value for another facet. And a high value in one facet might suggest high values in another, but this is not always the case.

The more enjoyable sex becomes, the more desirable the experience will be. But desirablity won’t always correlate with enjoyment. A divorce may be very desirable, but it’d be foolish to suggest it’s an enjoyable experience.

Most importantly, just as no two users ever have the same experience, these evaluations are highly subjective and can only measure the users perception of a future experience (their expectation), or it can measure their perception of the past.

Secondly, though we assign values for an experience for each of these facets, we’re not making value assessments. A successful experience does not necessarily achieve a good end. Nazi propaganda had great design, fulfilled goals, successfully transmitted mental models: it’s a stunning portfolio piece. Nazi propaganda was a successful experience even though the result was far from “good.” Tobacco advertising is another example.

Also, “accessibility” has everything and nothing to do with web accessibility. It has to do with how able I am to participate in the experience. For example, when communicating with my cat, I can enter the same room, and even pet her. She’s physically accessible, but verbal communication isn’t possible. I don’t speak Meow. Who knows what she’s saying, or what she’s thinking. We could say the same thing about my girlfriend.

Finally, these facets for experience should work independent of device or medium. Most of us work on the web, but experience happens with everything, so evaluation methods should work any where. We should use the same method to evaluate using a watch as we use for reading a novel, having sex, or ordering books from Amazon. Experience is independent of devices or objects. It happens in the head, so we need to evaluate the way a given experience interacts with what some have come to call a user’s infospace.

Other facets and models

Splitting experience into conveniently digestable bits is nothing new, and I can’t say I’ve even come close to examining every other model (check out “Forlizzi”), but of the few I’ve seen that work independent of device and medium, they inevitably conflate the mechanics of interaction with the experience that results from the interaction. I think this muddles things.

Probably the most well-known model using “facets” and “expereince” in the title is Peter Morville’s “Facets of user experience.” For the most part, the five facets I’ve mentioned here overlap with several of Peter’s seven. For example, that an experience be usable or findable, I would evaluate as accessible and negotiable. However, the center colum for Peter’s UX honeycomb attempts to evaluate the “value” of an experience. For the web and for communicating with clients, I think the honeycomb works great. For other experiences, though, I don’t think usefulness or credibility have anything to do with an experience’s value.

But my partner at Grafofini, Alex, suggests that valuability and credibility represent additional facets we should add to the list. I’m not so sure credibility can’t be reduced and evaluated using the other facets. And isn’t value an abstract conglomeration of how an experience evaluates against all five of these facets for experience?

I’m not sure I have the answer just yet, but my gut says no.