Collect Data with a Survey

Your Goals are to...

Get as much data as possible (avoid attrition and missing data)

• Have an introduction to remind people why it is important
• Start with interesting questions, not demographics
  • Put all boring questions, including demographics, as late as possible
• Group related questions, have a logical order
• Word sensitive questions carefully and put at the end when possible
• Always have a question asking for any final comments at the end 

Get Good Data

• Ask balanced questions 
• Ask questions that people can actually answer, given limitations on memory and knowledge
  • "How often..." questions are difficult for non-regular actions, especially if there is no time frame.  
• Do a pilot--have a few people complete the survey with you around so they can give feedback on the questions

For Web Surveys

• Use logic to make sure participants only see relevant questions
• Put just 1 or 2 related questions on a page
  • If on one screen, people will read it before deciding whether to answer at all
  • Too easy to accidently scroll past a question and accidently skip it
• See Using Qualtrics Effectively (pdf) for tips, some applicable to all software

Notes: Use "Logic" in your survey software to hide irrelevant questions from participants. In an online survey, you should never have a question starting with "If yes,...". or "If no,....":

Articles

• How to design a questionnaire  by Wai-Ching Leung, Student BMJ (2001)
  • Brief and important overview of factors to consider
• 20 tips for writing web surveys  by David Travis, UserFocus
  • Reminders of the factors to check for. 
• A Great Survey is Like a Great Conversation -- Here's What We Mean!  by SurveyMonkey
  • Imagine a survey like a first date, and see where it gets awkward!
• The Qualtrics Handbook of Question Design by Qualtrics
  • Survey Questions 101
  • 7 Tips for Writing Great Questions
  • Video: Writing Good Question

Videos

• 7 tips for good survey questions (~4 min) Elon University Poll
  • Consistent Interpretation, Personal Questions, Social Desirability, Known Answer, Double Barreled Questions, Biased Wording, Pretest.
• Writing Good Survey Questions (~3.5 min) Dr Nic's Maths and Stats
  • Open-Ended vs Closed-Ended Questions. Easy to answer, Short, Correct Spelling and Grammar, Appropriate language, Specific, Realistic, Define your terms, One idea, Positive if possible, No double negative, Alternatives, Avoid overlap, Provide all alternatives, Avoid bias, Neutral questions, Pilot test
• Methods 101: Question Wording (~5.5 min) Pew Research Center
  • Remind respondents, Leading questions, Double-Negatives,  Acquiescence Bias, Context Effects (general first), Diverse reviewers, Pretest.
• Scales and response options (~6 min) - Research Methods and Statistics
  • Likert scales; Scale construction; Well formulated questions: Short and simple, Unambiguous, Not suggestive, Answerable, Avoid Extreme wording; Response options: unambiguous, consistent, exhaustive, mutually exclusive
• Response and rater bias (~4.5 min) - Research Methods and Statistics
  • Response Biases: acquiescence, social desirability, extreme response style, bias toward the middle; Rater biases: halo effect, generosity error, severity error.

How to Pilot Test your Survey

• Wear the Respondent's Shoes (Practical Surveys)
  • See also tips on what to learn by taking surveys
• Pre-Launch Checklist (SurveyGizmo)
  • The details may be specific to their platform, but the topics are important for everybody.