Re: [Usability] user levels, etc.

[Luis Villa <louie ximian com>]

On Mon, 2001-11-12 at 21:39, Adam Elman wrote:
> The truth is, however, that in usability testing it's usually possible 
> to get very good results with a very small number of people.  There is a 
> lot of usability literature that indicates that you tend to get most of 
> the same results for a reasonably well-chosen group of, say, 3-5 people 
> as for a much larger group.  This is not, of course, true of psychology 
> or political polling.

Fair enough. Not having read the research, I'm not really qualified to
say anything, but I will say that the skeptic is me is really, really
curious as to how 'most of the same results' is defined. My gut feeling
is that if you are aiming for knowledge about 75% of users, 3-5 would be
fine, but we shouldn't settle for that. We should aim to do much better
than that.
 
> It is also heavily dependent on what is being tested.  If you're really 
> testing usability, i.e. whether people can accomplish the tasks you've 
> asked them to accomplish using your interface, and whether they find it 
> pleasant, then what I said holds.  If you're looking for aesthetic 
> opinions, or trying to test longer-term issues (such as the long-term 
> preference of ~/ vs. ~/Desktop), then a skewed population makes a big 
> difference.

Again, fair enough, and this time with no disclaimers :)

> I _do_ disagree with your suggestion that there is a _strong_ bias 
> towards the "uneducated and inexperienced."  As I said, I've never had 
> trouble recruiting a few educated and experienced people to do a user 
> test.  The skew only really matters when you're dealing with much larger 
> groups, which most usability testing does not need.

Like I said, my gut feeling is that you do need it unless you have a
very low standard for what 'good enough' is. But I haven't read the
research so I'll defer to those who have.

> >> Making stuff more usable for hackers will make stuff more usable for
> >> anyone, and vice versa.  It's a false dichotomy.
> >
> > As long as we're agreed on this I think we're all on the same page.
> >
> > I just don't think everyone is on the same page about it, and I wanted
> > to raise the red flag, because I see a strong tendency to jump at the
> > alluringly simple 'remove options' argument before they even think about
> > other solutions to the problem. As long as we give good, hard, long
> > thought to the other options (as you said, using /all/ the tools) then
> > I'll be perfectly satisfied and quit my whining :)
> 
> Yep.  I think it's possible to read your original email as saying that 
> there is no such thing as a "useless preference," which I think we can 
> both agree is not true. 

Sure; that's why I labeled it a rant. :) Honestly, I don't have time to
be having this discussion right now... which is why the first letter was
a rant, because I was determined not to 'waste my time' this way. But
now you see what happened to that determination :) 

 
> Which goes back to my original point: I think having an "advanced user 
> preferences" dumping ground is fine, but I think developers need to take 
> responsibility for designing their preferences well in the first place 
> so that useful prefs won't end up in the dumping ground, and useless 
> prefs won't end up anywhere.  And that means thinking through all the 
> options.

Very well said. Can we print this out backwards and stick it on our
foreheads? Better yet, can I quote it next time I get into an argument
with the Ximian gnomecc people about our current packaging scheme for
cc?
 
> So yeah, I think we're on the same page.

Yup. Sounds like it. 
Luis