String Cleaning with pandas 2.0

1h 2mAdvanced2023-10-10

Authors

Seth Berry

Associate Teaching Professor and MSBA Academic Co-Director at the University of Notre Dame

Course details

Cleaning strings is no easy task, but it’s important, necessary work. From basic find-and-replace operations to more complex regular expressions, this course provides an overview of how to clean strings with pandas 2.0, the latest release of the open-source library for the Python programming language.

Gather insights from University of Notre Dame professor Seth Berry as he explores the fundamentals of string cleaning using pandas expressions, functions, operations, and methods, including PyArrow backends, regular expressions to find and replace text for complex patterns, speed tests, text extractions, match and split functions, quantifiers, backreferences, lookarounds, boundaries, chains, and more. By the end of this course, you’ll be ready to leverage a wide variety of pandas solutions to start cleaning strings on your own.

Skills covered

pandasData AnalysisData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceOne-Off

Concepts

0. Introduction

01 - Cleaning strings with pandas
02 - What you need to know

1. Foundations for String Cleaning

03 - Using the PyArrow backend
04 - Testing speed
05 - Removing special characters
06 - Matching patterns
07 - Extracting patterns
08 - Counting strings
09 - Splitting strings
10 - Creating new columns from text

2. Taking Action in pandas

11 - Finding and replacing patterns
12 - Using quantifiers
13 - Reordering strings with backreferences
14 - Naming backreference groups
15 - Extracting strings with lookarounds
16 - Using boundaries and sets
17 - Chaining pandas string methods
18 - Regular expression golf

Conclusion

19 - Advancing regular expressions