Re: GNOME user survey 2011 (v4)

[Felipe Contreras <felipe contreras gmail com>]

On Fri, Aug 19, 2011 at 9:20 PM, Matthew Garrett <mjg59 srcf ucam org> wrote:
> On Fri, Aug 19, 2011 at 08:14:25PM +0300, Felipe Contreras wrote:
>> On Fri, Aug 19, 2011 at 7:24 PM, Matthew Garrett <mjg59 srcf ucam org> wrote:
>> > Any survey that isn't a carefully controlled randomly selected sample of
>> > users doesn't result in learning.
>>
>> Unless the biases are identified, which we are trying to do.
>
> You can only identify the biases if you already know the population, and
> you can only know the population if you've got a random sample set to
> begin with.

That's not true. You might need that if you want to account for *all*
the biases, which nobody can do anyway. What most people do is try to
figure them out, chances are you might be missing some of the biases,
but hopefully the unidentified misrepresented group won't be that big
anyway, and thus wouldn't affect so much the analysis.

If it turns out that a significant bias is not identified beforehand,
that can be tackled in the next survey in 2012.

>> Moreover, I have tried to push the idea to have an automatic
>> notification, which would maximize the number of responders, and thus
>> increase the randomization. But apparent this idea is not welcome.
>
> It doesn't help. The people most likely to respond to an irritating
> popup that disrupts their work are people who already feel that gnome 3
> is an irritating piece of software that disrupts their work. You can't
> get a random sample in-band.

It doesn't help? It does randomize the sample more, doesn't it?

Maybe it's not perfectly randomized, but nothing can ever be perfect.

>> So, ideas to improve the randomization are dismissed, and then you say
>> without randomization, the survey is not useful. IOW; you are
>> intentionally deadlocking the proposal.
>
> I am saying that your results aren't useful unless your sample is
> random. I don't know of a good way to obtain a representative sample.

There's no such thing as 0% random, or 100% random, all we can thrive
for is to increase the randomness.

And I already explained that non-random samples are already useful if
you can identify the biases.

>> > Everyone will see what they want to see. Those
>> > who believe that Gnome 3 is a step back will point out that the majority
>> > of responses are negative. Those who believe it's a step forward will
>> > point out that happy users are going to be far less inclined to respond.
>> >
>> > There's no way whatsoever to determine how representative the responses
>> > are, and so there's no way whatsoever to learn anything about the
>> > population. All we'd learn is that some users like Gnome 3 and some
>> > users don't, and that's something we *already know*. So we'd gain
>> > nothing, but we'd guarantee another huge set of arguments which would
>> > themselves also tell us nothing.
>>
>> That's an assumption. What if we get 10 million responses? Would you
>> still claim that the results are not representative?
>
> Yes, because you have no idea how big the population is. Maybe 10
> million is the total population and it's representative. Maybe it's 50%
> of the population, disproportionately biased towards those of a given
> prior opinion. You can't know.

Do you have any idea what is the likelihood of that happening? Try
throwing a dice 10 times and always getting 1-3. Even if the dice is
rigged, it's very unlikely. It gets exponentially less likely 1
million times.

>> I think only *after* getting the results you would be able to say
>> anything about it's representativeness.
>>
>> Something more realistic, say you get at least 300 responses that
>> don't have any "geek" bias, that would be more than enough to make
>> some statistically significant conclusions.
>
> It really wouldn't.

Yes it would. Check Cochran's formulas. 300 unbiased responses gives
you already good statistical power, after a certain point it doesn't
matter much what is the total population; 10m, 30m, 1m. The likelihood
that would would get 300 unbiased responses all pointing to the wrong
direction is almost nothing, in fact a few dozens would do (if they
are truly random).

There are some simple calculators online:
http://www.surveysystem.com/sscalc.htm

But yeah, since there's going to be bias, you need more.

>> > I disagree. Doing something that sucks more time and energy away from
>> > development without actually telling us anything in return is worse than
>> > that not happening. Felipe is obviously free to do whatever he wants,
>> > but there's no benefit in Gnome itself participating in any way. If we
>> > want to find out what our users think then the only way to do that is to
>> > have professional involvement and a random sample set.
>>
>> This is not sucking any time and energy from anybody, I just need
>> access to the server that has limesurvey installed, or somebody else
>> can do that (can't take that much time), I would contact all the
>> relevant news sites and make the relevant posts in social media. All
>> that that is needed from GNOME people is a blessing.
>
> The sucking of time and energy would come from the argument over the
> results afterwards.

Why? If there big unidentified biases, or the analysis is unsound, you
can just ignore them. But what if they are not and there's really
something useful there? Then the energy will be well spent.

Would you rather keep walking carelessly in a dark room, or would you
try to use your cellphone's screen to guide you? Sure, the light of
the cellphone is very poor, and you might see things that are not
really there, but at least it's something. To me GNOME is hitting
everything in the room as it's going forward, and saying; I'm fine, I
know where I'm going...

-- 
Felipe Contreras