Re: Where is the data?

[Shaun McCance <shaunm gnome org>]

On Sat, 2011-08-20 at 18:09 +0300, Felipe Contreras wrote:
> On Sat, Aug 20, 2011 at 6:02 PM, Shaun McCance <shaunm gnome org> wrote:
> > On Sat, 2011-08-20 at 14:43 +0200, Tomasz Torcz wrote:
> >>   I think his objections were justified.  There is really no raw data
> >> at those URLs.
> >
> > Except Allan never claimed he was providing raw data. In fact,
> > he explicitly said that he does not do write-ups of user tests.
> 
> So what's the point of replying to a mail asking specifically for data?

When somebody asks me for something I don't have, I usually
respond telling them so, and explaining why I don't have it.
I think it's rude to ignore people.

> > I also do user tests when working on the help. I also don't do
> > write-ups. I fix problems or I pass information on to those who
> > are in a position to fix the problems.
> >
> > Just because I don't publish reports doesn't mean I don't do
> > user tests. And the constant assertions that nobody is looking
> > at feedback are getting a bit insulting.
> 
> User tests, like surveys, are not perfect and can be both misleading,
> and not significant enough.

You're right, of course. All methods have flaws. But user testing,
at least, gives results that are personal and actionable. When I
get results like "all 5 users were uncertain where to click when
instructed to click the 'user menu'", I know what to do. I have
no idea what to do with "63% of respondents report they are less
happy than they were a year ago".

> If those tests are to be taken seriously, they should be published so
> that they can be scrutinized, otherwise they are not evidence of
> anything, not to the rest of the world.

I agree there are problems with transparency. A lot of things
get done on IRC, because high-bandwidth communication is great
for rapid development. I've been a strong supporter of public
logs for IRC. I think we should discuss ways to better record
what we do and the decisions we make.

But, I don't want to be in a situation where we have to wait
for committees to scrutinize data and approve proposals before
we can make changes. That sounds like an awful project to write
software for.

--
Shaun