ETL in Python and SQL
1h 21mIntermediate2024-03-06
Authors
Jennifer Ebe
Course details
Data is increasing exponentially as the world continues to generate an immense amount. In this course, a data engineer with over 5 years of experience, Jennifer Ebe, will show you how to build systems that gather data from various sources, transform it and store data so you can derive insights from it effectively. We will go over what ETLs are and how you create them. We will learn how to use Python to extract and transform data, we will explore our data using pandas and SQL. And finally, load data into our target system. You will also get practice scheduling ETL jobs using Python. Each chapter comes with helpful, hands-on challenges to help you practice what you learn.
Skills covered
SQLDatabase AdministrationData EngineeringDatabase DevelopmentPythonDatabase ManagementData AnalysisProgramming LanguagesData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceSoftware DevelopmentOne-Off
Concepts
Introduction
- Create an ETL in Python and SQL
- Tools used in this course
What Are ETLs
- What are ETLs and how do you create them
- ETL process overview
- Exploring your data with pandas (Python) and SQL
- Understanding your data
- Challenge - Reading data using Python
- Solution - Reading data using Python
Extracting and Transforming Data with Python
- Loading data from different sources
- Extracting your data
- Cleaning, preprocessing data, and data formatting
- Standardization, handling duplicates, and missing values
- Challenge - Load the data using SQLAlchemy and pandas
- Solution - Load the data using SQLAlchemy and pandas
Loading Data into Target Systems
- Introduction to data warehouses and data lakes
- Loading data into relational databases
- Data quality checks and validation with SQL
- Challenge - Transform the data and remove duplicates and nulls
- Solution - Transform the data and remove duplicates and nulls
Automating ETL Jobs - Scheduling ETL Jobs with Python
- Querying your data with SQL
- Scheduling ETL jobs with Airflow - Part 1
- Scheduling ETL jobs with Airflow - Part 2
- Challenge - Load the data into a database and automate
- Solution - Load the data into a database and automate
Conclusion
- Expand your knowledge of ETLs