Re: GNOME user survey 2011 (v4)

[Allan Day <allanpday gmail com>]

Felipe Contreras <felipe contreras gmail com> wrote:
> On Fri, Aug 19, 2011 at 1:14 PM, Allan Day <allanpday gmail com> wrote:
>> Felipe Contreras <felipe contreras gmail com> wrote:
>>> On Fri, Aug 19, 2011 at 1:20 AM, Zeeshan Ali (Khattak)
>>> <zeeshanak gnome org> wrote:
>>>> On Fri, Aug 19, 2011 at 12:45 AM, Felipe Contreras
>>>> <felipe contreras gmail com> wrote:
...
>> Different people will understand the words GNOME/happy/very
>> happy/ecstatic in different ways. Some might think 'GNOME' is their
>> distro (including the lower levels of the stack),
>
> Which is why we ask more question to understand their level of
> "geekness".  That should help the make correlations; the people that
> use a terminal all the time more likely know that GNOME is just the
> DE. The people that don't have much experience might be confusing
> GNOME with the distribution.

'Geekness' is not the only thing that will affect people's
understandings, and you haven't adequately measured that anyway. Plus
that doesn't do anything to deal with the problem of what people
understand by 'GNOME'.

>> Likewise,
>> 'happy' will be thought of differently by different people (a very odd
>> word to include in a questionnaire, if you don't mind me saying):
>
> I think everyone understands the word happy.

/ME wipes a mouthful of coffee from my monitor

Then you haven't read enough of the survey research literature.

...
> In any case, if you have suggestions that don't have these problems,
> feel free to share them.

My suggestion would be to give up entirely or to rethink the premise
of your research. The latter is what I'd have advised when I was
working as a research consultant, or what I would have told one of my
students when I used to teach this stuff, for that matter.

>> You've also got the representativeness problem. Your sample will
>> inevitably be unrepresentative, probably highly so. Even if 100% of
>> your *unrepresentative sample* tick the unhappy box, that doesn't tell
>> you much about your target population: you might just have sampled
>> every 'unhappy' GNOME user that's out there.
>
> If you can identify the bias, that's not a huge problem.

So tell me - how will you accurately compensate for the effects of
self-selection bias? What kinds of claims will you make about
representativeness?

>> tl;dr version: your survey results will be misleading.
>
> No, the results would not be misleading; the *analysis* of the results
> might. But different people can analyze them in different ways. The
> important thing is to get *some* results.

It seems bizarre to suggest that research data is valid irrespective
of how it is gathered. If your questionnaire does not provide valid
measurements no amount of analysis can compensate.

>> We already have a wealth of data about peoples' experiences with GNOME
>> 3 and are working to address the issues that are being raised. It's
>> great that you want to help, but this survey really won't be useful.
>
> Where? I haven't seen any.

We've had incredible amounts of feedback; most (if not all) of which
has been read, and which does get taken seriously. I also know that
those of us who are influencing the design of GNOME 3 take a strong
interest in peoples' experiences with it and ask them questions
(that's certainly what I do). There's also a small series of user
tests last I did Christmas, the results of which have been fed into
the development process. Believe me, that is more than enough to be
going on for now. (Some more user testing would be useful at some
point in the future, though.)

Allan
--
IRC:  aday on irc.gnome.org
Blog: http://afaikblog.wordpress.com/