Re: GNOME user survey 2011 (v2)

["Jason D. Clinton" <me jasonclinton com>]

On Mon, Aug 1, 2011 at 10:35, Felipe Contreras
<felipe contreras gmail com> wrote:
> After going through all the feedback, here's the second version of the
> proposed survey.

Surveys require careful consideration of the phrasing of questions and
careful consideration of the selection bias of respondents. It doesn't
look like you have considered either, even after the bias in phrasing
of the questions was pointed out to you in the first round of emails.

For example, here's a classic:

  Do you approve of the President's job on foreign affairs?

versus:

  How would you rate the President's job on foreign affairs?

The first question is leading by punching the survey taker in the face
with a statement about their value judgement before you've even
finished saying the sentence. The second asks the survey taker to make
their own value judgement.

Second, you have to carefully consider who is motivated to take a
survey. People who are angry are much more likely to respond to a
survey versus people who are happy. You can mitigate this problem by
offering a survey that appears to have nothing to do with the subject
matter that you're really looking for an answer on so that you get a
truly random sampling of Linux users. You also must be careful not to
recruit people to take the survey from communities which will contain
angry people. For example, going to forums to find people to take a
survey automatically selectively biases from people who were likely
there to solve some kind of problem and are so already in a particular
state of mind. There are hundreds of strategies to work around the
selection bias problem.

It *is* a good idea to run a survey but unless it's done carefully,
you could actually do far more damage than good by reporting results
which are not accurate or--worse--call in to question the motivations
of the study.

The survey you're proposing will require months of work to do
correctly and is likely best achieved as a part of a large group
effort or as part of an academic or non-profit funded venture.