Re: [Usability] user levels, etc.

[Maciej Stachowiak <mjs noisehavoc org>]

On 12Nov2001 09:22PM (-0500), Luis Villa wrote:
> 
> Nope. Done some political psychology research, though, which bears a
> strong resemblance in terms of finding focus groups and/or larger bodies
> of people. And yes, there is a large body of research that indicates
> that more or less no matter how hard you try and how nice you are there
> are inherent selection biases at work whenever you try to find such a
> group.
> 
> That said, I'm not trying to imply that user testing is useless. Skewed,
> yes, but still very, very useful. I'm not trying to take away tools from
> the arsenal- just trying to stress that all tools have inherent
> limitations everyone should be up front about.
> 

All the user tests I've heard of have had, if anything, mostly
college-educated subjects, which is hardly below average education.


There is also an assumption in this discussion that experience with
computers, educational background, being a hacker, and other such
things will have a huge effect on the outcome of user
tests. 

Historically, that has not proven to be the case. Experience with the
software being tested or with other systems that work similarly or
differently in some crucial way will often bias the tests. Sometimes
cultural factors such as use of a right to left language will also
play a role.

But not so much the other things you mention.

At some companies user testing is done informally just by pulling in
engineers from other groups who happen to be walking by in the hall
and asking them to perform a task. Usually the results are not much
different from more formal user testing.

 - Maciej