Re: GNOME user survey 2011

[Christophe Fergeau <teuf gnome org>]

2011/8/1 Felipe Contreras <felipe contreras gmail com>:
>> What are your
>> plans to make sure people unhappy with GNOME are not over represented
>> in the poll results?
>
> Publicize the survey as much as possible.

So what? This is not how "real-life" polls work.

>
> Would you like to rephrase the survey to don't assume the respondent
> is using GNOME, and then ask this question?
> == Which desktop environment are you currently using? ==
>
> This should spot the people that are more likely to be unhappy.

Ah? How so? Is it an initial assumption that people using G3 are more
unhappy with their DE than people using something else?

>
>> Given the kind of questions, it's bound to
>> attract answers from people who want more options, and I don't think
>> how we can go from "N% of the people who took the survey said they
>> wanted more options" to "M% of *all* G3 users want more options". To
>> me, these figures will be totally unrelated, unless I missed something
>> in the way you want to run the poll.
>
> They are certainly not the same, but if N = 100, you can say with
> certain degree of certainty that M is certainly not 0.

Of course, but since we don't know how many people in total are using
G3, N=100 will only tell us that there are 100 people out of an
unknown number of users that want more options, and nothing more. In
particular, if this 100 people end up as 60% of the poll respondant,
we'll have absolutely no way to go from the poll results to a
percentage which means anything for G3 users as a whole. Ie without a
careful selection of who answers the poll, even if 60% of the poll
respondant said they would like more options, we don't know if this
maps to 1% of all the users, or to 90% or to something in between.
And I'm ready to bet that this "60% of the people who answered the
poll want more options" will quickly become "60% of gnome users want
more options, and developers don't listen!". That's why I'm
questioning the gathering of these numbers. They are only very
marginally useful, and in spite of that, they will be misused.

>> Since your actual goal seems to use this survey results to pressure
>> people into adding more options to GNOME (where "more options"
>> probably really means "the options you want to have"), I'm afraid this
>> poll will turn into "I want to prove X, let's design a survey whose
>> result will be X".
>
> That's speculation. How would you design a survey to prove !X? It will
> be the same.

It's not that hard to write questions that are biased towards the
result you want (see this thread as a proof), actually what is hard is
doing the opposite, something that is not biased either way.

>
>> I even suspect that you'd get different results by
>> adding something like "Do you get confused by software with too many
>> configuration options ? Yes/No" " before asking the question about the
>> amount of configuration options ;)
>
> So you want to add bias? If you have been following the thread, you
> would see that the proposed changes are intended to decrease the bias.

No no, I was not saying this question should be added, on the
contrary. I was just illustrating how hard it is to write a good
survey, and how easy it is to change the outcome. The efforts in this
thread to improve the questions are really appreciated.

>
> So you prefer, the status quo, which is "we have no idea about what
> anybody thinks".

Not necessarily, but if we gather these numbers, we have to agree from
the start that they unfortunately won't mean a lot, and we have to be
really careful to make sure they won't be used as if they proved
anything about G3 users as a whole.
And with that in mind, the survey becomes a lot less interesting ;)

Christophe