Re: [Usability] user levels, etc.

[Adam Elman <aelman users sourceforge net>]

On Monday, November 12, 2001, at 06:22 PM, Luis Villa wrote:
> On Mon, 2001-11-12 at 21:09, Adam Elman wrote:
> > On Monday, November 12, 2001, at 05:44 PM, Luis Villa wrote:
> > > It has a selection bias towards the uneducated and inexperienced (the
> > > educated and experienced have better things to do than sit and play 
> > > with
> > > boxes while being watched.)
> > > [...]
> >
> > Have you ever actually done or attempted to do any user testing?
>
> Nope. Done some political psychology research, though, which bears a
> strong resemblance in terms of finding focus groups and/or larger bodies
> of people. And yes, there is a large body of research that indicates
> that more or less no matter how hard you try and how nice you are there
> are inherent selection biases at work whenever you try to find such a
> group.

OK.  I'll grant you that there are inherent selection biases in any kind 
of "people-focused" research.

The truth is, however, that in usability testing it's usually possible 
to get very good results with a very small number of people.  There is a 
lot of usability literature that indicates that you tend to get most of 
the same results for a reasonably well-chosen group of, say, 3-5 people 
as for a much larger group.  This is not, of course, true of psychology 
or political polling.

It is also heavily dependent on what is being tested.  If you're really 
testing usability, i.e. whether people can accomplish the tasks you've 
asked them to accomplish using your interface, and whether they find it 
pleasant, then what I said holds.  If you're looking for aesthetic 
opinions, or trying to test longer-term issues (such as the long-term 
preference of ~/ vs. ~/Desktop), then a skewed population makes a big 
difference.

I _do_ disagree with your suggestion that there is a _strong_ bias 
towards the "uneducated and inexperienced."  As I said, I've never had 
trouble recruiting a few educated and experienced people to do a user 
test.  The skew only really matters when you're dealing with much larger 
groups, which most usability testing does not need.

> That said, I'm not trying to imply that user testing is useless. Skewed,
> yes, but still very, very useful. I'm not trying to take away tools from
> the arsenal- just trying to stress that all tools have inherent
> limitations everyone should be up front about.

Sure.  I think you're right that user testing is not a good way to test 
for long-term issues, but wrong that the bias issue really makes that 
big a difference.

> > Making stuff more usable for hackers will make stuff more usable for
> > anyone, and vice versa.  It's a false dichotomy.
>
> As long as we're agreed on this I think we're all on the same page.
>
> I just don't think everyone is on the same page about it, and I wanted
> to raise the red flag, because I see a strong tendency to jump at the
> alluringly simple 'remove options' argument before they even think about
> other solutions to the problem. As long as we give good, hard, long
> thought to the other options (as you said, using /all/ the tools) then
> I'll be perfectly satisfied and quit my whining :)

Yep.  I think it's possible to read your original email as saying that 
there is no such thing as a "useless preference," which I think we can 
both agree is not true.  But to figure out which preferences (features, 
etc.) are actually useful and which aren't, we do need to use all the 
tools in the arsenal, and I agree that we need to do so knowing their 
limitations.

Which goes back to my original point: I think having an "advanced user 
preferences" dumping ground is fine, but I think developers need to take 
responsibility for designing their preferences well in the first place 
so that useful prefs won't end up in the dumping ground, and useless 
prefs won't end up anywhere.  And that means thinking through all the 
options.

So yeah, I think we're on the same page.

Adam