Validation from Thinking and Making

Weekly usability tests, Nielsen talks about Tivo

Austin Govella — Wed, 30 Jul 2008 03:05:38 GMT

Jakob Nielsen's latest Alertbox shares how Tivo did 12 usability tests in 12 weeks and offers some tips and guidelines for doing the same at your organization. I've actually been hearing about this a lot lately: design teams running weekly or bi-weekly usability tests. It'd be nice to have more companies share their experiences, the how and why and such. At CIM we did something similar in concert with a project using the scrum process. While this is all cool, the article summary really stuck in my craw: "frequent and regular testing keeps the design usability focused". Why would you want to keep your design usability focused? Nielsen probably doesn't mean only focused on usability, but it sure comes off that way. It just feels so 90s. If we all know the design should be focused on the experience what kind of frequent tests can you run to keep your design experience focused? Well? If you were running some kind of bi-weekly validation to keep your team experience focused, what would it look like? Hint: There's a comment box right down... there. Fess up.

Improving qualitative metrics: adding comparisons

Austin Govella — Tue, 24 Jun 2008 17:45:00 GMT

You've clarified the purpose to your scale, specifying what question you want people to answer, but there's still an unanswered question floating around. Let's say you're on Amazon, and you're rating a book. You answer the question "would I want to read more books like this?" You say yes, but in your head you're running a comparison. Do you want to read more books like this? Compared to what? Cookbooks? Stephen King? Naked Lunch? Don't Make Me Think? If you're in a group and everyone is trying to answer "should we launch this", differences in individual experience -- their unique "compared to whats" -- make it almost impossible for everyone to discuss the same issues. At CIM, we have an ace team of people who have worked on good and great products from all over the country. Asking someone from AOL "would you launch this" reveals a different answer compared to someone from a series of small engineering-focused start-ups. To make sure everyone answers the same "compared to what" question, specify what you're comparing to. On Fancast, you have the ability to search for actors, tv shows, and movies. If we ask the team "would you launch our search feature", everyone needs to know who we're comparing to. For Fancast, we might compare our search to IMDB, Fandango, or Yahoo Movies. The final comparison is really less relevant than having everyone answer the same question. "Compared to IMDB's search, would you launch Fancast's search function?" That's better than "based on your myriad, varied, and wildly different backgrounds, would you launch Fancast's search function?"

Improving qualitative metrics: adding purpose to your scales

Austin Govella — Mon, 23 Jun 2008 21:10:09 GMT

Rate this 1-5. What does that mean? We know 1 is most likely bad and 5 is most likely good, but what are we rating? Why? Flat scales don't tell you much more than someone thought something. They don't tell you what they thought. For heuristic evaluations, it's nice to assign a score to each item you evaluate. At Comcast Interactive, we had this long discussion about heuristic evaluations. Should we use a word scale (good, ok, bad)? Should we use a production-focused scale (blocker, needs work, launchable)? Should we use numbers (1-3)? The problem with good, ok, bad, and 1, 2, 3 is that they're both flat scales. They don't tell you much other than the IA+Usability team evaluated the product. The production-focused scale communicates more clearly. In this context, the purpose of the heuristic evaluation is to highlight the really bad and what needs fixing *so that business can launch the product*. A blocker blocks launch, so the severity of that problem is pretty clear. So, for any scale, if you determine why you rate something, the purpose, then you will have ratings that support what you're trying to do. Working in a group, it's critical everyone answer the same question. Let's say three people perform a heuristic evaluation of the same product. One answers "is this usable?" The second answers "is this a good experience?" The third, "is this launchable?" You can't compare and discuss differences in answers because you don't know what the differences mean. You can't even agree on the similarities. You don't know what they mean either. If all three answer the same question -- is this usable -- then you definitely understand differences in the answers are worthy of discussion.

A sample 'Only' statement for the I.A. Institute

Austin Govella — Thu, 12 Jun 2008 10:13:00 GMT

To kind of go through how this works, I thought we'd work through an example using the IA Institute. Now, I'm not picking on the IAI. I love them. I am a reasonably vigorous part of them. But when I was sitting in the annual meeting, I though to myself, "man, do these guys need an only statement". So here goes our fictional exercise at crafting an Only statement for the IAI.

We'll start with their tagline (“The IAI supports individuals and organizations specializing in the design and construction of shared information environments”) and convert that into an only statement:

WHAT	The only organization
HOW	that develops and supports a community
WHO	for information architects
WHERE	anywhere in the world
WHY	who want to design information spaces
WHEN	in a world of ubiquitous data, access, and connection

Well, that's not totally true. We still can't agree on what an information architect is, so let's change that the UX professionals:

WHAT	The only organization
HOW	that develops and supports a community
WHO	for ~~information architects~~ user experience professionals
WHERE	anywhere in the world
WHY	who want to design information spaces
WHEN	in a world of ubiquitous data, access, and connection

And do we really develop new communities, or do we support existing communities? Let's tweak that, and change the text so we don't use world twice:

WHAT	The only organization
HOW	that ~~develops and~~ supports a community
WHO	for user experience professionals
WHERE	~~anywhere in the world~~ around the globe
WHY	who want to design information spaces
WHEN	in a world of ubiquitous data, access, and connection

So, that's it. Now we have an Only statement that describes who we are and what we do. It's a nice enough exercise, but Only statement works best as a way to validate design decisions.

Using the Only statement

Creating the Only statement packs all of your meaning together. Once everything's packed, you can unpack the meaning to understand more about the project's core essence.

So, in the magic world of our example, we've reached the final version of our Only statement, and it reveals an interesting fact about the organization:

It's not necessarily a professional organization, and not necessarily supported by membership.
It's all about the community of practice, and not necessarily the practice. (Props for the 'Blurt!)

That definitely gives us some things to think about.

Based on the Only statement, we might reassess the services the organization provides. For our fictional version of the IAI, we might decide a community needs several things:

Jobs (an ecology of stuff to do and people to do it; not necessarily paid work.)
Discussions (email lists, forums, distributed conversations)
Multiple languages (translations)
Events (meetings, conferences, f2f conversations)
Localized news, events, discussions, jobs (Politicians always say "everything is local".)
Discussions with other communities (elevator pitches/mobile widgets, evangelization)
Mentors and mentees

And since we're framing things up, maybe we organize community needs into two chunks:

knowledge sharing (our list of community needs from above)
community memory (best practices, tutorials, case studies, library, books, links)

Already, we have an understanding of what the IAI is and what it's not. We have a framework for deciding what kinds of activities it should support, and those it shouldn't. Essentially, we've defined a strategy we can follow for years.

And that, I think, is the magic of the Only statement: that it can help guide product and design strategy. But does it have to? Next, I'll talk about at how an Only statement does and doesn't interact with strategy.

The 'Only' statement: focus on your project's key goals

Austin Govella — Thu, 05 Jun 2008 19:33:00 GMT

The myriad reasons mission statements suck has more to do with who put them together and why. Any time you explain your team's shared vision in bite-size morsels anyone can consume, that's what we call "a win". Although some mission statements explain your vision, they rarely explain why, or provide a convincing how. Tthe why and the how are what make your vision a signpost your team can strive for.

Introducing the Only statement

A couple of years ago, "Marty Neumeier":http://www.neutronllc.com/ released a follow-up to the "Brand Gap":http://www.amazon.com/gp/product/0321348109?ie=UTF8&tag=thinkingandma-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0321348109 called "Zag":http://www.amazon.com/gp/product/0321426770?ie=UTF8&tag=thinkingandma-20&link_code=as3&camp=211189&creative=373489&creativeASIN=0321426770. It's a great book. Although it might seem like a book about brand strategy, I thought it was more of an introduction to business analysis. (I posted a brief "review of Zag":http://www.thinkingandmaking.com/view/the-best-innovation in January.)

In Zag, Neumeier describes a great technique, the "only" statement (starting on page 65). An Only statement is like a mission statement except it focuses on what makes you _unique_. A mission statement might answer "what do we want to do?" The only statement answers "what do we do best?"

How it works

The concept starts simply enough. Complete this sentence: "You are the only [blank] that [blank]."

The first [blank] is for your category, and the second is for what makes you unique. An example explains it better. Neumeier creates an Only statement for a fictional wine bar as an example: "Our brand is the ONLY chain of wine bars that builds community around wine education".

Neumeier unpacks the magic:

Even with this simple statement you can see that there are three unique attributes that will set this brand apart: It's a chain instead of a one-off; it's about community, not just customers; and it's built on education, not just enjoyment.

The Only statement as an exercise

The more detailed version of the Only statement exercise has you answer six questions:

WHAT	is your category?
HOW	are you different?
WHO	are your customers?
WHERE	are they located?
WHEN	do they need you?
WHY	are you important?

For the wine bar, Neumeier provides these answers:

WHAT	The ONLY chain of wine bars
HOW	that builds community around education
WHO	for men and women of drinking age
WHERE	in cities and progressive towns in the US
WHY	who want to learn more about wine
WHEN\|	in an era of cultural awakening

As Neumeirer explains, answering these questions describes your category and how you're different (the WHAT and HOW). It also describes who your audience is and where they are, as well as focuses "on a need state" (the WHY) as well as an underlying trend (the WHEN).

Using Only statements to validate design decisions

Neumeier's goal is to help organizations find radical differentiation, so the Only statements focuses on your unique selling point. If you can focus your team on your project's Only-ness, then feature decisions get easier.

When you want to add a new feature, run it by your Only statement. Does the new feature match up with your WHAT, HOW, WHO, WHERE, WHY, and WHEN? If you're choosing between two features, which one is better? (Maybe neither?)

Using Only statements for shared vision

An Only statement is a really good way to focus a team on the project's constraints (the WHAT, WHO, WHERE, and WHY), as well as on its strengths (the HOW and WHEN). This kind of focus is especially important on teams where shared vision drives the quality of the work (like "an agile team":http://www.thinkingandmaking.com/view/agile-ux-six).

It's equally important to note the difference between sharing a vision and shared vision. Sharing a vision is when Kennedy says we'll have a man on the moon in x years and everyone agrees: yes, we will try to put a man on the moon in x years. Shared vision is more like a shared worldview. When Kennedy shares the vision that we'll have a man on the moon in x years, everyone believes, yes, we *can* -- and we _should_ -- have a man on the moon in x years. Only statements help communicate a worldview that a team can share.

Only statements in the wild?

As I was putting this together, I realized it's a little abstract, so I'll try to post an example using a real project. However, if you have an example we could use, let me know!

The U.X. health check, a Summit side conversation

Austin Govella — Thu, 10 Apr 2008 11:57:00 GMT

At Comcast Interactive, we've been working on a new method for evaluating our work. What's emerged, we've been calling a UX Health Check, and I think it might have a lot of potential.

So, while we're milling about the Summit with all you great minds, I'd like to talk to you about the Health Check and get your opinions.

What does it do?

The health check is a method for quantifying the quality of a user experience.

It can measure experience at one point in time, or be used in a repeated way to track the quality of an experience over time.
It can be used for any product or service.
It can measure very detailed feature-sets, or very generalized notions of service.

Where'd it come from?

The health check was created to help product management communicate progress and improvement to executives. Instead of measuring design, it measures how well your users interact with your experience.

What are the outputs?

A measurement of where you are now
A measurement of how your experience has improved or not over time
A recommendation for product managers on where the biggest problems and opportunities are

Interesting?

If this sounds interesting, catch me at the Summit, and I'd love to talk to explain how it works and get your feedback.

Unique visitors is a dumb metric

Austin Govella — Mon, 24 Mar 2008 10:13:00 GMT

Lots of organizations try to measure success by measuring unique visitors. Typically, these groups have some sort of advertising-based business strategy where they believe the more eyeballs they reach, the better they’re doing.

But that’s not true.

For sites supported by ad revenue, unique visitors tell you how big your check will be that month. If every visitor is worth $1 in ad revenue, and you have 1 million visitors this month, then your check will be $1 million dollars.

What’s missing, what’s not being measured is whether or not your business model is successful. “But I have a million dollars”, you say. But having a million dollars isn’t your business model.

Let’s say this month, you have 1 million visitors, they all earn you $1 in ad revenue, and you get your check for $1 million dollars. And let’s say your site sucks, so none of them came back, and they told their friends to never go. Next month you have no visitors.

Each visitor still earns $1. And you go to the bank to deposit your check for $0.

What’s missing, what’s not being measured is the behavior your organization relies on to make money. There’s something all those unique visitors do, and it's the doing that's necessary for the success of your business.

In marketing terms, unique visitors measures how many potential customers you have acquired. However, in order to make money, you have to convert those potential customers into actual customers.

Broad metrics like unique visitors and total page views give you broad aggregate data that’s just about useless. What you really want is a behavior-based metric that measures the behaviors that earn you money.

A lot of sites driven by ad revenue need the same behavior in order to be successful. A user learns about the site, they visit the site, and then they return to the site again and again. The visitor's return visits are what really drive ad revenue. Instead of measuring unique visitors, a better measure of your site’s success is return visitors.

Performance metrics should always measure the behaviors your organization relies on to survive. Your metrics should always be behavior-based. Aggregate metrics like unique visitors and total page views don't tell you much of anything.

Evaluating experience design

Austin Govella — Mon, 21 Feb 2005 17:31:00 GMT

I edited this post to clarify how the facets would be used based on a conversation with Paula Thornton.

One of the most limiting factors in Design isn’t the splintering of specialist groups, nor the emergence of specialised vocabularies. We lack common language for discussing Design, for communicating and evaluating the creation of experience.

We all contribute to the overall user experience, but we don’t have a clear definition for what we do: what is experience? We have fuzzy definitions. Several models are emerging (I’m working on one as well), but we still lack an objective means for evaluating experience design. We may not understand it, but we know what it looks like, what it feels like, its general shape. I think we can use several facets to evaluate the resulting user experience:

Personal: How well does the experience relate to the individual user? A conversation with your best friend compared to talking to the clerk at the DMV.
Desirable: How much do the users desire the experience? How much do they want to experience it? How much do they need it? A triple heart bypass versus having your ears pierced.
Enjoyable: How much do users enjoy the experience? A chore versus something you enjoy doing.
Accessible: How accessible is the experience for the user? How understandable, comprehendable, physically available? Climbing Mount Everest versus climbing the curb.
Negotiable: How able is the user to negotiate the experience to better communicate with them? How customizable?

Theoretically, you could survey a group of users to evaluate a given experience in much the same way psychological surveys are performed. A numeric value can then be given to the experience in question. Two quick examples illustrate how you can evaluate different kinds of experiences using these facets:

Heart surgery: A very personal experience and very desirable (if you want to live), but not enjoyable.
Solving a rubix cube: A very personal experience, and very negotiable, but not very accessible. Anyone can try, and almost everyone can manipulate the cube, but very few can solve them.

Evaluating successful experiences

A successful experience evaluates differently for different purposes. Successful sales and education require the almost perfect transmission of mental models. The better sales or education evaluate across all five facets, the more effective the sales and education will be.

Compare learning on your own to learning how to program your VCR, and you can see why so many people have learned how to do the latter. I think one can say that communicative experiences, experiences where the primary goal is to communicate, should evaluate highly on all five facets in order to be successful.

Other experiences evaluate differently for success. The success of jail time’s deterrence requires it be neither desirable, nor enjoyable. And we work at making it undesirable: restricted freedoms; small, overcrowded quarters; and a culture of violence and racism. But, in the eyes of someone trapped in a recidivist culture, jail time loses many of its deterrent features.

None of these facets are mutually exclusive. If we assign values for a given experience, there’s an interaction among these values, but no zero-sum interplay. Improvement along one facet may improve or worsen an experience’s value for another facet. And a high value in one facet might suggest high values in another, but this is not always the case.

The more enjoyable sex becomes, the more desirable the experience will be. But desirablity won’t always correlate with enjoyment. A divorce may be very desirable, but it’d be foolish to suggest it’s an enjoyable experience.

Most importantly, just as no two users ever have the same experience, these evaluations are highly subjective and can only measure the users perception of a future experience (their expectation), or it can measure their perception of the past.

Secondly, though we assign values for an experience for each of these facets, we’re not making value assessments. A successful experience does not necessarily achieve a good end. Nazi propaganda had great design, fulfilled goals, successfully transmitted mental models: it’s a stunning portfolio piece. Nazi propaganda was a successful experience even though the result was far from “good.” Tobacco advertising is another example.

Also, “accessibility” has everything and nothing to do with web accessibility. It has to do with how able I am to participate in the experience. For example, when communicating with my cat, I can enter the same room, and even pet her. She’s physically accessible, but verbal communication isn’t possible. I don’t speak Meow. Who knows what she’s saying, or what she’s thinking. We could say the same thing about my girlfriend.

Finally, these facets for experience should work independent of device or medium. Most of us work on the web, but experience happens with everything, so evaluation methods should work any where. We should use the same method to evaluate using a watch as we use for reading a novel, having sex, or ordering books from Amazon. Experience is independent of devices or objects. It happens in the head, so we need to evaluate the way a given experience interacts with what some have come to call a user’s infospace.

Other facets and models

Splitting experience into conveniently digestable bits is nothing new, and I can’t say I’ve even come close to examining every other model (check out “Forlizzi”), but of the few I’ve seen that work independent of device and medium, they inevitably conflate the mechanics of interaction with the experience that results from the interaction. I think this muddles things.

Probably the most well-known model using “facets” and “expereince” in the title is Peter Morville’s “Facets of user experience.” For the most part, the five facets I’ve mentioned here overlap with several of Peter’s seven. For example, that an experience be usable or findable, I would evaluate as accessible and negotiable. However, the center colum for Peter’s UX honeycomb attempts to evaluate the “value” of an experience. For the web and for communicating with clients, I think the honeycomb works great. For other experiences, though, I don’t think usefulness or credibility have anything to do with an experience’s value.

But my partner at Grafofini, Alex, suggests that valuability and credibility represent additional facets we should add to the list. I’m not so sure credibility can’t be reduced and evaluated using the other facets. And isn’t value an abstract conglomeration of how an experience evaluates against all five of these facets for experience?

I’m not sure I have the answer just yet, but my gut says no.