Text & Data Mining Sources

Twitter Data Sets

• Tweet ID Data Sets (Documenting the Now)
  Selected, topical twitter datasets surrounding a significant event. Data are curated by Documenting the Now.
• TweetSets create your own subset of data from Twitter datasets on specific topics.
  Use this excellent tutorial - Beginner's Guide to Twitter Data - from The Programming Historian for using TweetSets and hydrating tweets.

Social Media Data from Multiple Sources (Twitter, Facebook, etc.)

• Facebook Data from the Humanitarian Data Exchange
• Stanford Large Network Dataset Collection: SNAP
• Kaggle contains data extracted from social media platforms.
• Dataverse includes social media datasets contributed by academic researchers.
• ICPSR has collections of social media data in its archive.

Apply for Data Access for Academic Research

• Twitter Data
  • Accessing Twitter data for analysis depends on your question and time period. For Twitter searches (keyword or hashtag), data older than around seven days typically cannot be retrieved. You can collect data going forward in time. If you need to extract tweets from an account, you can collect data going back and going forward in time. Twitter's Developer Agreement and Policy has details about what you can do with the data.
  • Unless you are using an existing dataset or resource such as Social Media Macroscope, you will need to apply for access to collect Twitter data through the Twitter for Academic Research site.
• CrowdTangle for Facebook and Instagram
• Social Science One

Additional Guides

• Social Media Research (University of Minnesota) is an excellent, in-depth guide to social media resources and tools.
• Social Media Research Toolkit (Ryerson University) created a directory of options for locating and extracting social media data. 
• Where to Get Twitter Data for Academic Research, an excellent guide written by the creators of Social Feed Manager (George Washington University Libraries). Some of the information has changed, but the guide remains useful.

Tools that collect social media data

• Social Media Macroscope
  • A project by University of Illinois Technology Services and the National Center for Supercomputing Applications (NCSA) with the goal of making social media data, analytics, and visualization tools accessible to researchers and students of all levels of expertise. You will need to set up an account.
• Social Feed Manager (SFM) (see the tutorial on our Text Analysis guide)
  • Developed by George Washington University Libraries, SFM was designed to help researchers extract data through Twitter, Flickr, Sina Weibo, and Tumblr. SFM is available through the Mason Libraries' Digital Scholarship Center.
• Twitter Archiver or TAGS add-ons for Google Sheets.
  • This add-on is a quick way to get started with collecting Twitter data for your research.
• NCapture Plug-in for NVivo
  • NCapture captures data from Twitter, YouTube, LinkedIn as well as Web pages and online PDFs. You need to use NCapture with NVivo in order to extract the data. The DiSC Lab in Fenwick Library has NVivo installed on its computers.
• MAXQDA
  • How to analyze Twitter data with MAXQDA 2020. The DiSC Lab has access to MAXQDA.

Advanced Collection Methods — coding required

• Facebook Graph API
• rtweet for R
• TweePy Python library