Re: GNOME user survey 2011 (v4)

[Felipe Contreras <felipe contreras gmail com>]

On Fri, Aug 19, 2011 at 1:14 PM, Allan Day <allanpday gmail com> wrote:
> Felipe Contreras <felipe contreras gmail com> wrote:
>> On Fri, Aug 19, 2011 at 1:20 AM, Zeeshan Ali (Khattak)
>> <zeeshanak gnome org> wrote:
>>> On Fri, Aug 19, 2011 at 12:45 AM, Felipe Contreras
>>> <felipe contreras gmail com> wrote:
>>>>
>>>> Nothing is ever perfect, but having at least some results is better
>>>> than nothing.
>>>
>>>  Since you have repeated this assertion a few times, I must ask: What
>>> if the results are all wrong and we don't have any way of knowing
>>> that? Would those results still be better than nothing in your
>>> opinion?
>>
>> What do you mean by all wrong? Let's assume that the results show that
>> 1000 people are not happy with GNOME. How can that be wrong? 1000
>> people responded that, the results were not somehow altered, or
>> boycotted, the results are the results, and that's that.
> ...
>
> 'Wrong' in social research typically means that your results lack
> validity: that you think the data is measuring one thing (eg. 'GNOME
> users' happiness with GNOME 3') but is in fact measuring something
> else.
>
> When you do survey research, you have to be certain that one person
> understands the questions in the same way that another person does.
> Looking at your questionnaire, that won't be the case. An example:
>
>> === 02. Overall, how happy are you with GNOME? ===
>> (single choice)
>>
>>  * unhappy
>>  * not so happy
>>  * happy
>>  * very happy
>>  * completely ecstatic
>
> Different people will understand the words GNOME/happy/very
> happy/ecstatic in different ways. Some might think 'GNOME' is their
> distro (including the lower levels of the stack),

Which is why we ask more question to understand their level of
"geekness".  That should help the make correlations; the people that
use a terminal all the time more likely know that GNOME is just the
DE. The people that don't have much experience might be confusing
GNOME with the distribution.

> Likewise,
> 'happy' will be thought of differently by different people (a very odd
> word to include in a questionnaire, if you don't mind me saying):

I think everyone understands the word happy. That is what is used in
Git user survey, and seems to be doing the job just fine.

In any case, if you have suggestions that don't have these problems,
feel free to share them.

> You've also got the representativeness problem. Your sample will
> inevitably be unrepresentative, probably highly so. Even if 100% of
> your *unrepresentative sample* tick the unhappy box, that doesn't tell
> you much about your target population: you might just have sampled
> every 'unhappy' GNOME user that's out there.

If you can identify the bias, that's not a huge problem.

> tl;dr version: your survey results will be misleading.

No, the results would not be misleading; the *analysis* of the results
might. But different people can analyze them in different ways. The
important thing is to get *some* results.

> We already have a wealth of data about peoples' experiences with GNOME
> 3 and are working to address the issues that are being raised. It's
> great that you want to help, but this survey really won't be useful.

Where? I haven't seen any.

-- 
Felipe Contreras