Text & Data Mining Sources

Twitter Data

Accessing Twitter data for analysis depends on your question and time period. For Twitter searches (keyword or hashtag), data older than seven days cannot be retrieved without paying a fee. You can collect data going forward in time. If you need to extract tweets from an account, you can collect data going back and going forward in time. Twitter's Developer Agreement and Policy has details about what you can do with the data.

More details are available from Where to Get Twitter Data for Academic Research, an excellent guide written by the creators of Social Feed Manager (George Washington University Libraries).

Twitter Data Sets

• Tweet ID Data Sets (Documenting the Now)
  Selected, topical twitter datasets surrounding a significant event. Data are curated by Documenting the Now.
• TweetSets create your own subset of data from Twitter datasets on specific topics.
  Use this excellent tutorial - Beginner's Guide to Twitter Data - from The Programming Historian for using TweetSets and hydrating tweets.

Social Media Data from Multiple Sources (Twitter, Facebook, etc.)

• Stanford Large Network Dataset Collection: SNAP
• Kaggle contains data extracted from social media platforms.
• Dataverse includes social media datasets contributed by academic researchers.

Tools that collect social media data

• Social Feed Manager (SFM) (see the tutorial on our Text Analysis guide)
  • Developed by George Washington University Libraries, SFM was designed to help researchers extract data through Twitter, Flickr, Sina Weibo, and Tumblr. SFM is available through the Mason Libraries' Digital Scholarship Center.
• Twitter Archiver Add-on for Google Sheets. This add-on is a quick way to get started with collecting Twitter data for your research.
• NCapture Plug-in for NVivo
  • NCapture captures data from Twitter, YouTube, Facebook, LinkedIn as well as Web pages and online PDFs. You need to use NCapture with NVivo in order to extract the data. The DiSC Lab in Fenwick Library has NVivo installed on its computers.
• DiscoverText provides access to older social media data as well as an online platform for performing analysis. They do charge a fee.

Advanced Collection Methods — coding required

• Facebook Graph API
• rtweet for R
• TweePy Python library

Other Options

Social Media Research Toolkit (Ryerson University) created a directory of options for locating and extracting social media data.