Re: GNOME user survey 2011 (v4)

[Felipe Contreras <felipe contreras gmail com>]

On Fri, Aug 19, 2011 at 7:24 PM, Matthew Garrett <mjg59 srcf ucam org> wrote:
> On Thu, Aug 18, 2011 at 10:37:33PM +0100, Alan Cox wrote:
>
>> Doing nothing achieves nothing, doing something achieves learning. You
>> may well not learn what you intended but you will learn something
>> including quite possibly how to do future surveys better.
>
> Any survey that isn't a carefully controlled randomly selected sample of
> users doesn't result in learning.

Unless the biases are identified, which we are trying to do.

Moreover, I have tried to push the idea to have an automatic
notification, which would maximize the number of responders, and thus
increase the randomization. But apparent this idea is not welcome.

So, ideas to improve the randomization are dismissed, and then you say
without randomization, the survey is not useful. IOW; you are
intentionally deadlocking the proposal.

> It results in data that forms some
> sort of rorschach blot.

It might if you look at it as a whole, but you can try to dissect it.

> Everyone will see what they want to see. Those
> who believe that Gnome 3 is a step back will point out that the majority
> of responses are negative. Those who believe it's a step forward will
> point out that happy users are going to be far less inclined to respond.
>
> There's no way whatsoever to determine how representative the responses
> are, and so there's no way whatsoever to learn anything about the
> population. All we'd learn is that some users like Gnome 3 and some
> users don't, and that's something we *already know*. So we'd gain
> nothing, but we'd guarantee another huge set of arguments which would
> themselves also tell us nothing.

That's an assumption. What if we get 10 million responses? Would you
still claim that the results are not representative?

I think only *after* getting the results you would be able to say
anything about it's representativeness.

Something more realistic, say you get at least 300 responses that
don't have any "geek" bias, that would be more than enough to make
some statistically significant conclusions.

>> I'm not saying its necessarily a great approach but it's vastly superior
>> to people sitting around picking holes in the idea until it never happens.
>
> I disagree. Doing something that sucks more time and energy away from
> development without actually telling us anything in return is worse than
> that not happening. Felipe is obviously free to do whatever he wants,
> but there's no benefit in Gnome itself participating in any way. If we
> want to find out what our users think then the only way to do that is to
> have professional involvement and a random sample set.

This is not sucking any time and energy from anybody, I just need
access to the server that has limesurvey installed, or somebody else
can do that (can't take that much time), I would contact all the
relevant news sites and make the relevant posts in social media. All
that that is needed from GNOME people is a blessing.

-- 
Felipe Contreras