Re: GNOME user survey 2011 (v4)

[Matthew Garrett <mjg59 srcf ucam org>]

(Resend: Managed to leave d-d-l off Cc: by accident)

On Fri, Aug 19, 2011 at 06:15:03PM +0100, Alan Cox wrote:
> > Any survey that isn't a carefully controlled randomly selected sample of 
> > users doesn't result in learning. It results in data that forms some 
> 
> You need truely or reasonably random samples for certain kinds of
> activities and analysis in particularly quantitative analysis when you
> want to perform p tests and the like. You don't need it in order to
> learn merely to generate statistical proofs and those are often quite
> useless anyway. Proviing gnome 3 is great/indifferent/sucks doesn't have
> much value. You do not need it for explorative learning. Small children
> do not need to open a statistically valid sample of randomly chosen doors
> to learn about doors !

I am all for making it easier for people to give feedback about Gnome, 
but presenting it as a survey gives a strong implication that the 
results are meaningful as an aggregate rather than as a collection of 
anecdotes. If we want to hear form users, let's make it easy for users 
to talk to us. A survey isn't the way to achieve that.

> > sort of rorschach blot. Everyone will see what they want to see. Those 
> > who believe that Gnome 3 is a step back will point out that the majority 
> > of responses are negative. Those who believe it's a step forward will 
> > point out that happy users are going to be far less inclined to respond. 
> 
> You seem to be assuming the results and that the only question of interest
> is "does gnome 3 suck".

I'm assuming that the sort of people who are going to go to the effort 
of filling out a survey are likely to be closer to the population 
discussing things on lwn than the population of usres in general. That 
may be entirely untrue! But if we get the opposite results then it still 
doesn't tell us anything that's actually true, and it's still an 
opportunity to argue the issue rather than focus on making software 
better.

> > If we want to find out what our users think then the only way to do that is to 
> > have professional involvement and a random sample set.
> 
> Of course, and the only way to produce a kernel or desktop is to hire
> professionals to do it for you no doubt.

If you went back to 1991 and wanted a production-quality kernel within a 
year, Linux probably wouldn't be your starting point. There'd be a 
learning process involved with setting up a professional-quality survey 
team, and the first few attempts would be pretty buggy. We'd get there 
in time, but until then...

-- 
Matthew Garrett | mjg59 srcf ucam org