Re: GNOME user survey 2011 (v4)

[Felipe Contreras <felipe contreras gmail com>]

On Fri, Aug 19, 2011 at 3:17 PM, Allan Day <allanpday gmail com> wrote:
> Felipe Contreras <felipe contreras gmail com> wrote:
>> On Fri, Aug 19, 2011 at 1:14 PM, Allan Day <allanpday gmail com> wrote:
>>> Felipe Contreras <felipe contreras gmail com> wrote:
>>>> On Fri, Aug 19, 2011 at 1:20 AM, Zeeshan Ali (Khattak)
>>>> <zeeshanak gnome org> wrote:
>>>>> On Fri, Aug 19, 2011 at 12:45 AM, Felipe Contreras
>>>>> <felipe contreras gmail com> wrote:
> ...
>>> Different people will understand the words GNOME/happy/very
>>> happy/ecstatic in different ways. Some might think 'GNOME' is their
>>> distro (including the lower levels of the stack),
>>
>> Which is why we ask more question to understand their level of
>> "geekness".  That should help the make correlations; the people that
>> use a terminal all the time more likely know that GNOME is just the
>> DE. The people that don't have much experience might be confusing
>> GNOME with the distribution.
>
> 'Geekness' is not the only thing that will affect people's
> understandings, and you haven't adequately measured that anyway. Plus
> that doesn't do anything to deal with the problem of what people
> understand by 'GNOME'.

It's easy to throw empty criticism. Provide *suggestions*.

>>> Likewise,
>>> 'happy' will be thought of differently by different people (a very odd
>>> word to include in a questionnaire, if you don't mind me saying):
>>
>> I think everyone understands the word happy.
>
> /ME wipes a mouthful of coffee from my monitor
>
> Then you haven't read enough of the survey research literature.

That doesn't change the fact that everyone understands the word "happy".

> ...
>> In any case, if you have suggestions that don't have these problems,
>> feel free to share them.
>
> My suggestion would be to give up entirely or to rethink the premise
> of your research. The latter is what I'd have advised when I was
> working as a research consultant, or what I would have told one of my
> students when I used to teach this stuff, for that matter.

That's not helpful. If you are such a master, surely you can come up
with a totally brand new user survey that is order of magnitude
better. That would be greatly appreciated.

>>> You've also got the representativeness problem. Your sample will
>>> inevitably be unrepresentative, probably highly so. Even if 100% of
>>> your *unrepresentative sample* tick the unhappy box, that doesn't tell
>>> you much about your target population: you might just have sampled
>>> every 'unhappy' GNOME user that's out there.
>>
>> If you can identify the bias, that's not a huge problem.
>
> So tell me - how will you accurately compensate for the effects of
> self-selection bias? What kinds of claims will you make about
> representativeness?

What would *you* do?

>>> tl;dr version: your survey results will be misleading.
>>
>> No, the results would not be misleading; the *analysis* of the results
>> might. But different people can analyze them in different ways. The
>> important thing is to get *some* results.
>
> It seems bizarre to suggest that research data is valid irrespective
> of how it is gathered. If your questionnaire does not provide valid
> measurements no amount of analysis can compensate.

You can thrown an analysis saying all this data is crap if that makes
you happier, but this survey won't eat babies.

>>> We already have a wealth of data about peoples' experiences with GNOME
>>> 3 and are working to address the issues that are being raised. It's
>>> great that you want to help, but this survey really won't be useful.
>>
>> Where? I haven't seen any.
>
> We've had incredible amounts of feedback; most (if not all) of which
> has been read, and which does get taken seriously. I also know that
> those of us who are influencing the design of GNOME 3 take a strong
> interest in peoples' experiences with it and ask them questions
> (that's certainly what I do). There's also a small series of user
> tests last I did Christmas, the results of which have been fed into
> the development process. Believe me, that is more than enough to be
> going on for now. (Some more user testing would be useful at some
> point in the future, though.)

For a "professor" you should know better. I want the data.

Anyway, I am going to ignore your comments, unless you provide some
*suggestions* for improvement.

-- 
Felipe Contreras