Re: [Usability] Usability Testing Review

[Calum Benson <Calum Benson Sun COM>]

On 25 Feb 2008, at 14:04, Patrick Carlson wrote:

> Hello all,
>
> Can anyone point me towards concrete usability tests for software/ 
> technology?  I need to write a paper reviewing a test (checking  
> validity, reliability, etc.).  Are there many "usability" tests out  
> there or are they mostly just picking random people and having them  
> give feedback?

Well, the most useful usability tests will never pick "random" people;  
the idea is to pick a representative sample of users, typically  
somewhere between 5 and 20.  ("Representative" can cover many things,  
including age, sex, level of education, and prior experience with this  
and/or similar products.)

That said, any user's feedback is usually better than no user  
feedback, and there are several different types of usability study,  
some of which involve representative users, and some of which are  
performed only by usability experts (for want of a better term):

- Usability Inspection.  This can take the form of a heuristic  
evaluation, and/or a cognitive walkthrough:

In a heuristic evaluation, usability experts look through each screen  
design in turn, trying to identify general usability issues.  The  
'heuristics' used to identify issues are typically some combination of  
Jakob Nielsen's classic list of ten <http://www.useit.com/papers/heuristic/heuristic_list.html 
>, and any platform or product UI guidelines that might also apply.

In a cognitive walkthrough, the usability experts play the part of  
potential users, walking through typical use cases.  This way, the  
testers gain a better feel for the navigational structure of the  
application.  On the other hand, it's quite possible that they are not  
experts in the domain that the product's real users would be, and may  
thus exhibit unrealistic usage patterns.

- Focus group: Brings together several representative users in a group  
discussion to talk about the product.  As with any group discussion,  
success depends on deciding what you want to learn beforehand, and a  
good moderator who gives everyone an equal chance to participate while  
keeping the discussion on track.

- Low fidelity test: A low-fidelity prototype is one developed very  
early in the design lifecycle.  For GUIs, there is usually no coding  
involved, and the emphasis is on how it works rather than what it  
looks like.  A paper prototype is a typical example.  Representative  
users are asked (individually) to walk through typical use cases,  
"thinking aloud" and deciding what they would do at each step if it  
were a real GUI, with the facilitator manipulating the prototype (e.g.  
swapping between sketches of different screens in response to a menu  
selection) in response.  Some advantages of this technique are that it  
can be used almost anywhere, and that participants are often more  
forthcoming with ideas and criticism when they can see the product is  
only at the 'rough idea' stage.

- High fidelity test: For GUIs, a prototype with sufficient (but  
usually not complete) functionality to allow the participants to  
experience how the final product will look and behave.  Again,  
representative users are asked individually to walk through typical  
use cases, "thinking aloud" during each task, and giving moderator-led  
feedback when they complete (or fail) each task.  Advantages of this  
technique include the user experiencing realistic interaction methods,  
response times etc.

Videoing the users as they take part in studies can be useful for  
later analysis, but can also be intrusive and impractical.  It's  
certainly not essential.

Then there's the question of what to do with the data.  There are  
quantitative methods available that offer a useful way to compare  
different versions of the same product or feature, such as SUMI <http://sumi.ucc.ie/whatis.html 
>.  More often than not, though, a report will generally list the  
tasks that participants were asked to do, and the most common problems  
found, along with their perceived severity and possible solutions.   
Some statistical analysis may also be required, depending on the type  
of data generated and results required.

For a concrete example, here's an old GNOME usability study from 2001:
<http://developer.gnome.org/projects/gup/ut1_script.html>
<http://developer.gnome.org/projects/gup/ut1_report/report_main.html>

And a Linux one from 2003:
<http://www.linux-usability.de/download/linux_usability_report_en.pdf>

And a GIMP study from 2004:
<http://www.relevantive.de/gimp/report/results_usabilitytest_05.04.html>

And an example of a usability inspection:
<http://eprints.cs.vt.edu:8000/archive/00000619/01/iLumina.pdf>

> Also, are there any good usability resources out there?  Not just  
> specifically for Gnome but software testing in general?

Many. One I like to point people at, which has some nice summaries of  
all the basic techniques, is <http://infodesign.com.au/usabilityresources/evaluation/default.asp 
>.

Cheeri,
Calum.

--
CALUM BENSON, Usability Engineer       Sun Microsystems Ireland
mailto:calum benson sun com            GNOME Desktop Team
http://blogs.sun.com/calum             +353 1 819 9771

Any opinions are personal and not necessarily those of Sun Microsystems