Re: GNOME user survey 2011 (v5)

[Felipe Contreras <felipe contreras gmail com>]

On Sun, Sep 18, 2011 at 7:04 PM, Shaun McCance <shaunm gnome org> wrote:
> On Tue, 2011-09-06 at 23:38 +0300, Felipe Contreras wrote:
>> Here's the next version of the survey. Unfortunately, it seems this is
>> not going to be blessed by GNOME, and questionpro.com only allows 10
>> questions in the free version. I haven't found a better free online
>> survey, and unless somebody offers hosting for this survey, it would
>> have to be limited.
>
> [snip]
>
>> What about other kinds of bias? Would the survey be invalidated if we
>> missed some group of people?
>
> Please look up "non-response bias". It's been shown that, in some cases,
> the group of respondents (and non-respondents) can be inherently biased.
> It's reasonable to suspect that polarized opinions could affect people's
> proclivity to respond.

People's proclivity to respond is called self-selection bias, and it's
tackled with the question "What is the main reason you are taking this
survey?", if people respond "Somebody is pushing me", we can identify
people without such proclivity and measure the bias of self-selected
people.

As for non-response bias, I am aware of it, and I am doing all I can
to mitigate it.

> You cannot even begin to deal with non-response bias unless you know the
> non-response rate. And you have no idea what the non-response rate is if
> you have an open-invitation survey on the Internet.

We don't need to know the exact non-response rate, studies have shown
it's not that important[1].

---
As a result of these and other such recent findings, it now seems
clear that a low response rate does not guarantee lower survey
accuracy and instead simply indicates a risk of lower accuracy.
---

Now. Do you actually have a suggestion?

[1] http://en.wikipedia.org/wiki/Response_rate

-- 
Felipe Contreras