Re: GNOME user survey 2011 (v4)

[Allan Day <allanpday gmail com>]

Felipe Contreras <felipe contreras gmail com> wrote:
> On Fri, Aug 19, 2011 at 1:20 AM, Zeeshan Ali (Khattak)
> <zeeshanak gnome org> wrote:
>> On Fri, Aug 19, 2011 at 12:45 AM, Felipe Contreras
>> <felipe contreras gmail com> wrote:
>>>
>>> Nothing is ever perfect, but having at least some results is better
>>> than nothing.
>>
>>  Since you have repeated this assertion a few times, I must ask: What
>> if the results are all wrong and we don't have any way of knowing
>> that? Would those results still be better than nothing in your
>> opinion?
>
> What do you mean by all wrong? Let's assume that the results show that
> 1000 people are not happy with GNOME. How can that be wrong? 1000
> people responded that, the results were not somehow altered, or
> boycotted, the results are the results, and that's that.
...

'Wrong' in social research typically means that your results lack
validity: that you think the data is measuring one thing (eg. 'GNOME
users' happiness with GNOME 3') but is in fact measuring something
else.

When you do survey research, you have to be certain that one person
understands the questions in the same way that another person does.
Looking at your questionnaire, that won't be the case. An example:

> === 02. Overall, how happy are you with GNOME? ===
> (single choice)
>
>  * unhappy
>  * not so happy
>  * happy
>  * very happy
>  * completely ecstatic

Different people will understand the words GNOME/happy/very
happy/ecstatic in different ways. Some might think 'GNOME' is their
distro (including the lower levels of the stack), some that it's their
'shell', others that it's all their GUI software [1]. Likewise,
'happy' will be thought of differently by different people (a very odd
word to include in a questionnaire, if you don't mind me saying):
there are a vast range of expectations and usage patterns in relation
to desktop computers, all of which will affect how people respond.
Someone could tick 'unhappy' but by most measures have had a perfectly
satisfactory experience.

You've also got the representativeness problem. Your sample will
inevitably be unrepresentative, probably highly so. Even if 100% of
your *unrepresentative sample* tick the unhappy box, that doesn't tell
you much about your target population: you might just have sampled
every 'unhappy' GNOME user that's out there.

tl;dr version: your survey results will be misleading.

We already have a wealth of data about peoples' experiences with GNOME
3 and are working to address the issues that are being raised. It's
great that you want to help, but this survey really won't be useful.

Allan

[1] GNOME's place in the stack means that you can't really do
satisfaction surveys on it. This is one reason why GNOME is a more
difficult research topic than, say, Git.
--
IRC:  aday on irc.gnome.org
Blog: http://afaikblog.wordpress.com/