Re: GNOME user survey 2011 (v4)

On Fri, Aug 19, 2011 at 08:14:25PM +0300, Felipe Contreras wrote:
> On Fri, Aug 19, 2011 at 7:24 PM, Matthew Garrett <mjg59 srcf ucam org> wrote:
> > Any survey that isn't a carefully controlled randomly selected sample of
> > users doesn't result in learning.
> Unless the biases are identified, which we are trying to do.

You can only identify the biases if you already know the population, and 
you can only know the population if you've got a random sample set to 
begin with.

> Moreover, I have tried to push the idea to have an automatic
> notification, which would maximize the number of responders, and thus
> increase the randomization. But apparent this idea is not welcome.

It doesn't help. The people most likely to respond to an irritating 
popup that disrupts their work are people who already feel that gnome 3 
is an irritating piece of software that disrupts their work. You can't 
get a random sample in-band.

> So, ideas to improve the randomization are dismissed, and then you say
> without randomization, the survey is not useful. IOW; you are
> intentionally deadlocking the proposal.

I am saying that your results aren't useful unless your sample is 
random. I don't know of a good way to obtain a representative sample.

> > Everyone will see what they want to see. Those
> > who believe that Gnome 3 is a step back will point out that the majority
> > of responses are negative. Those who believe it's a step forward will
> > point out that happy users are going to be far less inclined to respond.
> >
> > There's no way whatsoever to determine how representative the responses
> > are, and so there's no way whatsoever to learn anything about the
> > population. All we'd learn is that some users like Gnome 3 and some
> > users don't, and that's something we *already know*. So we'd gain
> > nothing, but we'd guarantee another huge set of arguments which would
> > themselves also tell us nothing.
> That's an assumption. What if we get 10 million responses? Would you
> still claim that the results are not representative?

Yes, because you have no idea how big the population is. Maybe 10 
million is the total population and it's representative. Maybe it's 50% 
of the population, disproportionately biased towards those of a given 
prior opinion. You can't know.

> I think only *after* getting the results you would be able to say
> anything about it's representativeness.
> Something more realistic, say you get at least 300 responses that
> don't have any "geek" bias, that would be more than enough to make
> some statistically significant conclusions.

It really wouldn't.

> > I disagree. Doing something that sucks more time and energy away from
> > development without actually telling us anything in return is worse than
> > that not happening. Felipe is obviously free to do whatever he wants,
> > but there's no benefit in Gnome itself participating in any way. If we
> > want to find out what our users think then the only way to do that is to
> > have professional involvement and a random sample set.
> This is not sucking any time and energy from anybody, I just need
> access to the server that has limesurvey installed, or somebody else
> can do that (can't take that much time), I would contact all the
> relevant news sites and make the relevant posts in social media. All
> that that is needed from GNOME people is a blessing.

The sucking of time and energy would come from the argument over the 
results afterwards.

Matthew Garrett | mjg59 srcf ucam org

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]