Hands-On Introduction: Data Engineering
1h 29mBeginner2023-04-28
Authors

Vinoo Ganesh
CEO and founder of Stealth Startup
Course details
Today’s world is flooded with data, which puts businesses up to task. In response, organizations, companies, and employees around the world have adopted data-driven decision-making techniques. However, the vast majority of new data practitioners haven’t been formally trained. They don’t know how to build and construct stable data pipelines that can function effectively at speed and scale. Moreover, they haven't been given the framework(s) to distill data-oriented tasks into discrete components.
In this course, instructor Vinoo Ganesh gives you an overview of the fundamental skills you need to know to solve complex data problems in a scalable, productive way. Explore the core principles of the data engineer toolkit—including ELT, OLTP/OLAP, orchestration, DAGs, and more—as well as how to set up a local Apache Airflow deployment and full-scale data engineering ETL pipeline. Along the way, Vinoo helps you boost your technical skill set using real-world, hands-on scenarios.
In this course, instructor Vinoo Ganesh gives you an overview of the fundamental skills you need to know to solve complex data problems in a scalable, productive way. Explore the core principles of the data engineer toolkit—including ELT, OLTP/OLAP, orchestration, DAGs, and more—as well as how to set up a local Apache Airflow deployment and full-scale data engineering ETL pipeline. Along the way, Vinoo helps you boost your technical skill set using real-world, hands-on scenarios.
Skills covered
Data EngineeringData ScienceOne-Off
Concepts
0. Introduction
- 01 - Hands-on data engineering
- 02 - Background knowledge
- 03 - The history of data in the enterprise
- 04 - How to use GitHub Codespaces with this course
1. Getting Your Bearings
- 05 - Data engineering and data pipelines
- 06 - Orchestration - In the modern sense
- 07 - Extract, transform, load (ETL)
- 08 - Tasks, DAGs, and dependencies
2. Airflow Introduction
- 09 - Introduction to Airflow
- 10 - Installing Airflow
- 11 - Running the Airflow webserver and scheduler
- 12 - Adjusting Airflow configuration settings
- 13 - Build a 1 Task DAG
- 14 - Build a 2 Task DAG
3. Basic ETL in Airflow
- 15 - ETL in Airflow
- 16 - A real-world data engineering task
- 17 - Extracting data manually
- 18 - Extracting data with Airflow
- 19 - Transforming data manually
- 20 - Transforming data with Airflow
- 21 - Loading data manually
- 22 - Loading data with Airflow
- 23 - Building an ETL DAG with Airflow
- 24 - Challenge - Review ETL questions
- 25 - Solution - Solutions to ETL questions
Conclusion
- 26 - The future of data engineering
Related courses
- Big Data in the Age of AI
- Complete Guide to Analytics Engineering
- Advanced Analytics Engineering: Real-World Practice
- Complete Guide to Google BigQuery for Data and ML Engineers
- PySpark Essential Training: Introduction to Building Data Pipelines
- Cleaning Data for Effective Data Science: Data Ingestion, Anomaly Detection, Value Imputation, and Feature Engineering
- Scala Essential Training for Data Science
- SPSS: Wrangling, Visualizing, and Modeling Data