Re: GNOME user survey 2011 (v4)

[Matthew Garrett <mjg59 srcf ucam org>]

On Fri, Aug 19, 2011 at 10:26:08PM +0300, Felipe Contreras wrote:
> > Yes, because you have no idea how big the population is. Maybe 10
> > million is the total population and it's representative. Maybe it's 50%
> > of the population, disproportionately biased towards those of a given
> > prior opinion. You can't know.
> 
> Do you have any idea what is the likelihood of that happening? Try
> throwing a dice 10 times and always getting 1-3. Even if the dice is
> rigged, it's very unlikely. It gets exponentially less likely 1
> million times.

That's clearly wrong. If you have a bucket of red balls and blue balls 
and you draw 10 million balls, and you find that you drew 6 million red 
balls and 4 million blue balls, what does that tell you? If you're 
sampling randomly it tells you that there are more red balls than blue 
balls. If you're subconsciously preferring to pick up red balls then it 
tells you nothing. So we need to avoid subconsciously picking red balls, 
which means we need to pick users randomly which is something we can't 
do with a voluntary survey. Cochran's formulas don't apply here because 
you're not picking your sample set at random.

-- 
Matthew Garrett | mjg59 srcf ucam org