InfoGuides: Working with Data: Regular Expressions

Regular Expressions are fancy wildcards. Typically abbreviated "regex", they allow you to find / match as well as replace inexact patterns and even special characters (like tabs and line breaks) in text. This is useful in many programming languages, but also for finding-and-replacing in documents. Most programming languages and many text editors, including Microsoft Word (2010+) allow the use of standard regex symbols and syntax. There are some differences in how regular expressions work in different software, but much of the basics are identical. The term "grep" which is used by programmers synonymous with search, stands for "global regular expression print".

Regular Expressions and Wildcards can be used for finding and replacing or removing text in a file, database, or filename. It is very powerful and can even assist in cleaning and reformatting data.

xkcd cartoon about Regular Expressions "saving the day"

Background

Wildcards (the concept behind RegEx)

Searching Effectively Using Truncation - Animated Java tutorial explains why wildcards are useful
Wildcard Definition (Computer Hope) - Examples of *, ?, and [] from DOS Windows, and Linux (more examples)
How to Use Wildcards When Searching in Word 2013 - Examples of using Word's Special Wildcards

Showcase for what RegEx can do

Writing: Why writers want to learn regular expressions by Jakob Persson - Several useful examples in contetxt
Business: Everyday Text Patterns by Dan Nguyen -- A narrative showcases several uses for a business
History: Understanding Regular Expressions by the Programming Historian - Complex follow-along example withLibreOffice Calc.
- See also Cleaning OCR'd text, a complete example utilizing Python.

Getting Started

Follow-Along Exercises

RegexOne - Step-by-step instruction with real-time practice
Learn regular expressions in about 55 minutes by Sam Hughes - Guided practice
Video: Using Regular Expressions from Lynda.com

Build and Test

RegExr - Lots of help, quick reference and a pleasing interface; see capture groups by replacing $& with $1, $2, etc
Regular Expressions 101 - Good for programmers; can mimic php, javascript, and python; has a library of scripts

Reference

What is Regex and Quick Reference from Computer Hope
Regular Expressions Cheat Sheet by Dave Child
Regular Expression Cheat Sheet by Nicola Pietroluongo

Advanced

Test Yourself

The regular expression game (Machine Learning Lab) - State what you know and get 12 timed exercises to see if you can capture the correct strings.
Regex Crossword
Regex Golf

Tips

How to Build a Good RegEx (Sitepoint) - Examples illustrate the process of designing a string match
Regex tips (David Birnbaum) - Shows and explains some commonly needed expressions, especially for XSLT

Books

All books are available ONLINE through Mason.

Mastering Regular Expressions by Jeffrey E. F. Fried
Regular Expressions Cookbook by Steven Levithan; Jan Goyvaerts
Beginning Regular Expressions by Andrew Watt

Comprehensive Websites

Regular-Expressions.info - From Jan Goyvaerts, creater of the software Regex Buddy -- See also the Quick Start
Rex Egg - from Rex, a regex consultant -- Including a Cheat Sheet

Implementation in Specific Programs

The vast majority of RegEx symbols are exactly the same across software, so these resources are broadly useful.

Stata	What are regular expressions and how can I use them in Stata? (STATA) More examples: How can I extract a portion of a string variable using regular expressions? (UCLA)
R	Finding: Regular Expression in R (Gloria Li and Jenny Bryan) Extracting Extract information from texts with regular expressions in R (Kun Ren)
Python	Regular Expression HOW TO and Operations (Python Documentation) Text Processing in Python by David Mertz -- related: Matching Patterns in Text: The Basics
Notepad++	How to use regular expressions in Notepad++ (NP++ Wiki) A guide to using regular expressions and extended search mode (Mark Antoniou)
Atom	Find and Replace (Atom Manual) Regular expressions in Atom for code consistency (Modelit)
Office	Microsoft: Finding and replacing characters using wildcards (Graham Mayor, MVP) OpenOffice: Regular Expressions in Writer (OpenOffice Documentation)
Perl	Perl Regular Expression (Perl Documentation) Teach Yourself Perl in 21 Days by Laura Lemay - excerpt: Pattern Matching
SQL	SQL Wildcards (W3 Schools)