Re: GNOME user survey 2011 (v4)

[Felipe Contreras <felipe contreras gmail com>]

On Fri, Aug 19, 2011 at 9:28 PM, Matthew Garrett <mjg59 srcf ucam org> wrote:
> (Resend: Managed to leave d-d-l off Cc: by accident)
> On Fri, Aug 19, 2011 at 06:15:03PM +0100, Alan Cox wrote:
>> > Any survey that isn't a carefully controlled randomly selected sample of
>> > users doesn't result in learning. It results in data that forms some
>>
>> You need truely or reasonably random samples for certain kinds of
>> activities and analysis in particularly quantitative analysis when you
>> want to perform p tests and the like. You don't need it in order to
>> learn merely to generate statistical proofs and those are often quite
>> useless anyway. Proviing gnome 3 is great/indifferent/sucks doesn't have
>> much value. You do not need it for explorative learning. Small children
>> do not need to open a statistically valid sample of randomly chosen doors
>> to learn about doors !
>
> I am all for making it easier for people to give feedback about Gnome,
> but presenting it as a survey gives a strong implication that the
> results are meaningful as an aggregate rather than as a collection of
> anecdotes. If we want to hear form users, let's make it easy for users
> to talk to us. A survey isn't the way to achieve that.

Again, any better suggestions? I tried many of them back in 2007, and
got nowhere, I think a user survey is the best one we've got.

>> > sort of rorschach blot. Everyone will see what they want to see. Those
>> > who believe that Gnome 3 is a step back will point out that the majority
>> > of responses are negative. Those who believe it's a step forward will
>> > point out that happy users are going to be far less inclined to respond.
>>
>> You seem to be assuming the results and that the only question of interest
>> is "does gnome 3 suck".
>
> I'm assuming that the sort of people who are going to go to the effort
> of filling out a survey are likely to be closer to the population
> discussing things on lwn than the population of usres in general. That
> may be entirely untrue! But if we get the opposite results then it still
> doesn't tell us anything that's actually true, and it's still an
> opportunity to argue the issue rather than focus on making software
> better.

We most likely are going to be able to identify that bias.

Let's make some wild guesses; 50% of the people that use GNOME 3 like
it, and 50% don't. Of that amount, 90% seem to be geeks. In the
remaining 10%, the people that use GNOME 2 show 80% happiness, and of
GNOME 3 it's only 60%.

But you still don't think there's any value in there, fair enough.
Then we dig through that 40% subset that didn't like GNOME 3 and take
a look at their comments, and we find "Very strange", "Can't get used
to it", and things like that. At that point we might want to see if
they left an email to contact them, and then try to gather more
detailed feedback.

I think there's a chance that this survey could tell us something
that's "actually true".

>> > If we want to find out what our users think then the only way to do that is to
>> > have professional involvement and a random sample set.
>>
>> Of course, and the only way to produce a kernel or desktop is to hire
>> professionals to do it for you no doubt.
>
> If you went back to 1991 and wanted a production-quality kernel within a
> year, Linux probably wouldn't be your starting point. There'd be a
> learning process involved with setting up a professional-quality survey
> team, and the first few attempts would be pretty buggy. We'd get there
> in time, but until then...

Until then it's better to have nothing?

-- 
Felipe Contreras