Collect Data with a Survey

Keep in mind that you have to ask many people in order to get a few responses as your respondents are busy. If people do not respond to your email right when they get it, they are less likely to ever do so.  If you can do so without annoying people, sending reminders after a few days or a week can help--up to about 5 times when necessary.

• Respondent Flow Chart - see all the criteria that must be met to get a response (Practical Surveys)
• Typical Response Rates - Get an idea of typical rates and tips how to improve them (Practical Surveys)
• Tips for Increasing Survey Completion Rates - Data on how various aspects of surveys affects responses (SurveyMonkey)
• Tip Sheet: Sampling, Coverage, Nonresponse pdf  - Learn about the concepts involved in sampling  (Harvard PSR)

Note that this is the number of responses, not the number of people to contact (typical response rates range from 4% to 40%). 

• Most statistical sotware (including SPSS v27+, Stata, and SAS) have special tools
• G*Power (Win/Mac) - Power and sample size calculator. Quite powerful with a useful manual (pdf).
• Raosoft Sample Size Calculator
  • ​Gives example sample sizes for binary questions
• Survey Random Sample Calculator (custom insight)
  • Simple calculator, plus calculate the sample given a specified response rate
  • Note 70% is very high for anything but employee surveys

Margin of Error

• The 'Margin of Error' Is More Controversial Than You Think (Huffington Post)
• A Debate on Using Margin of Error with non-probability panels
• “Margin of Error”, When Used by Pollsters, Is Widely Misunderstood and Confuses Most People pdf   (Harris Polls)
• Margin of Sampling Error and Credibility Interval (pdf) (AAPOR)
  • Understanding a “credibility interval” and how it differs from the “margin of sampling error” (AAPOR)

It takes time to collect good data, from designing and pilot testing the questionnaire to recruiting a large and representative sample. If you are doing a data analysis project in one semester, time is the one thing you don't have. Thus, you will probably be doing convenience sampling, which restricts generalizability.

The importance of recruitment depends on your purpose. Are you trying to:

• Gather ideas
  • You can and should get as many people to participate in any way you can.
• Compare people/responses
  • You should attempt to ensure that the people who take your survey are diverse representatives of the population you are interested in. And, consider whether the relationships you are interested in might be impacted by personal characteristics.  
• Conduct an experiment
  • You should attempt to ensure that the people who take your survey will have the same responses to your materials as the population you are interested in. In most cases, representing the population is less important. Focus instead on making sure that your manipulation is as small as possible and applied randomly. 
• Describe a population
  • Your sample must be representative of the population in order to claim that the percentage and means you find in your data can be generalized to the whole group. Only random sampling is generally accepted to produce a representative sample. Otherwise, you will have to justify your claim, perhaps by comparing demographics if the characteristics of the population are known.