Fundamentals of Data Transformation for Data Engineering

Course details

Fundamentals of Data Transformation with pandas and DuckDB SQL presents the most essential concepts and best practices in a clear and concise format that allows students to side-step the noise and complexity. While this course is wide rather than narrow, it was designed to help you understand your options for development and make an informed choice about where to drill down.

Instructor Matt Palmer covers transformation in three separate environments: data transformation in a lake, data transformation in the warehouse, and data transformation in data lake houses.

Concepts

Introduction

Welcome to data transformation
What we'll cover and what you should know

Setup and Beyond

Codespaces and setup
Why SQL Why Python Why not Spark
Types of data transformation
The goal of data transformation

Data Transformation with SQL and DuckDB

DuckDB basics and query structure
Wrangling unstructured data
Joins and comparisons
Aggregations
Windows functions - A quick refresher
Window functions
Advanced filters
Advanced joins
Lambdas and UDFs
Data generation
SQL challenge
SQL solution

Data Transformation with Python and pandas

DataFrame basics
Wrangling unstructured data
Select and filter
Order and aggregate
Advanced filters
Data generation
Windows
Apply
pandas challenge
pandas solution

Conclusion

What you learned, how to practice and grow, and next steps
SQL bonus challenge
SQL bonus solution