Re: survey Friends of GNOME donors

[John Williams <john williams lists gmail com>]

At last, a topic that I can actually help with, rather than simply bike-shedding!

I teach marketing research, so I feel able to comment with a fair degree of background knowledge.  That said, there's nothing so tedious as a disagreement between experts (especially if they're academics).

Number one survey tip: don't include ANY question unless you really need it.  

Example: Why on Earth would you want to know about the respondents demographic profile?  If you want to "get inside their head" ask them about their opinions, beliefs and feelings.

Unless you're going to use a variable to perform a meaningful analysis that will answer an important question (the answer to which is _actionable_) there is no point in including it.

Thus endeth the sermon.

2009/5/22 Stormy Peters <stormy gnome org>

One of the things the speaker recommended doing was survey donors every 12-18 months.

For what purpose, exactly?  And I mean _exactly_.  To what use will the resulting information be put?

1. Know what you want to find out, such as awareness and brand recognition.

Check.

2. Use survey monkey

As a last resort.  Better to use an on-line survey tool of your own, that way you have more control.  Don't let survey monkey do the analysis for you.  Get the data from them and do the analysis yourself.

3. No open ended questions. Use choices with an other option and room for comments.

Complete horse-shit.  Open-ended questions are best (they are information-rich), but they are time-consuming to analyse. Only use closed questions when you are _absolutely_ sure that you have a complete and mutually exclusive list of all possible responses.  That's often harder than it seems.

4. Ask about services and impact.

What, precisely, about these things.

5. Less than 3 minutes. "Please help us by taking this survey in 3 minutes or less."

Nice.  Except it's better worded as something like

"We'd like to ask for your help in finding out how we can do better.  When we tested this survey we found that most people could complete it in three minutes or less.  If you can help us by participating, we will use the information to improve our offerings."

That's a bit wordy and slightly pretentious, but (I hope) you get the idea.

6. Ask demographic questions: gender, age, ... (Lots of people leave this off and she thought it was vital.

It's almost never used in any meaningful way.  Why is it vital?  (Who was this person, anyway?)
 

She talked a lot about getting in their head. You need to know what they read, what they like to have for breakfast, who their friends are, ... if you want to reach them.

She's talking about media buying.  The only reason most marketers ask demographic questions is so they can choose which media to buy.  The rates for various media are priced according to penetration of various demographically defined segments.  That suits the media sellers, but no marketer in their right mind would ever segment their market demographically.
 

(One of her other big things was if it's not relationship building or reputation enhancing, don't do it.)

Nice, nice, nice.

7. Don't make people answer any questions. Every question should be optional or you'll lose people.

Check.
 
 

Here's my first pass.

Goal: I think the most important thing to find out is "why GNOME?" Why did they decide to support us? (Are they users, developers, ... How did they find out about donating? etc.)

Once you have that information, what will you _do_ with it?  How will it inform and guide management _action_? 

Imagine you've got the results and written the report.  Will it be discussed in a meeting and then forgotten?  Will it sit on a shelf untouched?  Or will people refer to it when they need to decide on some course of action?
 

You donated to Friends of GNOME in the past year. Please help us by taking this survey in 3 minutes or less.

1. Are you ... (check all that apply)
- GNOME user
- GNOME developer
- GNOME contributor (event planning, documentation, translation, web page creation, ...)
- GNOME Foundation member
- GNOME fan
- Other ______________

(Some further explanation --- by popup, hover etc. is helpful. For example: If I develop on GNOME but don't have a module in the official module-set, am I a GNOME developer?

2. Do you use ... (check all that apply)
- GNOME desktop
- GNOME applications like Abiword, Gimp, Banshee, Inkscape (which ones should we list here?)
- GNOME products on Windows
- GNOME products on KDE
- None of the above

What is the purpose of this question?  I can't see it.
 

3. Which GNOME Foundation activities do you value most?
- Hackfests
- GUADEC
- 6 month releases
- web pages
- forums
- translations
- documentation
- other ____________

Nice.  Perhaps some more on what you mean by "value"?
 

4. What would you like to see the GNOME Foundation do over the next year? (check all that apply)
- exactly what it's doing now
- more developer events
- more outreach to end users
- more end user events
- more new products
- more support of existing products
- more work with distributors

Excellent.  But is this list exhaustive?
 

5. How did you hear about the Friends of GNOME donation page?
- from the gnome.org web page
- from Planet GNOME
- from a friend
- from a blog
- from an email
- other _________

What is the purpose of this question?

6. Did you know about our Friends of GNOME badges, http://www.gnome.org/friends/promote.html.
- yes, I have one on my blog
- yes
- no

What is your employment status?
- full time student
- software developer
- other full time job
- retired
- unemployed
- other ___________

How old are you? [Open to suggestions for better breaking this down.]

When forming categories of any continuous variable there are only really two considerations: (1) what categories are meaningful/useful to the researcher; and (2) if the sample profile will be compared with something else, e.g. population profile, what categories does the information that the sample will be compared with use?  The most common example would be to use the came categories as a National Census.

- Under 18
- 18 - 21
- 22 - 25
- 26 - 29
- 30 - 44
- 45 - 60
- 60 - 74
- 75+

Are you male or female?
- male
- female

What is your annual household income? [Are there standards for this?]
- $0-$20,000
- $20,000 - $40,000
- $40,000 - $60,000
- $60,000 - $80,000
- $80,000 - $100,000
- $100,000 - $149,000
- $150,000+

I don't see the relevance of this question.

What is your current education level?
- some school
- high school equivalent
- some college
- college degree
- graduate degree
- other___________

Or this one.

Anything else you'd like to tell us? _____________________

This is the best question. But it will be the most time-consuming to analyse.

Thoughts? Comments? Ideas?

Stormy

P.S. I also put some of these questions in SurveyMonkey[1].  With SurveyMonkey I can currently only ask 10 questions (which with comments for the "other" category is not enough.) We could also use Google forms or pay $20 to upgrade SurveyMonkey for a month. Or use some other tool ...

I strongly recommend a home-brewed, or at least Free, system.  Partially because of the inevitable comments and flames from using proprietary software, but mostly because of the control that gives you over your research.  If it's important and on-going, there's no real reason to use a half-assed approach like survey monkey.

The Mozilla people have been using their own system and they may be willing to share.  I have worked with them in the past [1] and could approach them on our behalf if necessary.

[1] http://www.surveymonkey.com/s.aspx?sm=4xY5BTO_2f4tKft9lxRYxtWg_3d_3d


--
marketing-list mailing list
marketing-list gnome org
http://mail.gnome.org/mailman/listinfo/marketing-list

Thanks for kicking this off, Stormy.  I have tried to make my critique constructive; I hope it doesn't read too negatively ;-)

P.S.  I'm teaching an Honours course in Marketing Research in the second half of this year (it's near the end of the first semester here in New Zealand).  There is a chance that we could make your problem part of the course work.  Would that be helpful?  The problem from your point of view would be that you wouldn't get a final report from students until October or so.

[1] http://blog.mozilla.com/seth/2008/05/28/some-help-from-a-kiwi/
[2] http://marketing.otago.ac.nz/studying/papers/martpapers.asp?papercode=mart303

-- 
John Williams
Professional Idiot