End-to-End Data Engineering Project
1h 16mIntermediate2023-11-03
Authors

Thalia Barrera
Course details
The world of data engineering is ever-changing, with new tools and technologies emerging on a regular basis. Building an effective analytics platform can be a daunting task, especially if you’re not familiar with all the tools available. How do you turn scattered, complex data into a model that drives insights and decision-making? In this course, Thalia Barrera teaches data professionals how to implement an end-to-end data engineering project using open tools from the modern data stack. She touches on best practices such as data modeling, testing, documentation and version control and shows you how to efficiently extract, load, and transform data into a unified, analytics-ready format. Thalia shows you how to confidently select and use tools through practical examples—taking you through the construction of a robust data pipeline for a fictional ecommerce company—and how to implement best practices in data engineering.
Skills covered
Data EngineeringProjectData Science
Concepts
0. Introduction
- 01 - Transform complex data into insights
- 02 - What you should know
1. Project Overview and Preparation
- 03 - Project architecture overview
- 04 - Project setup
- 05 - Understanding the Big Star Collectibles database
- 06 - Setting up your data warehouse
2. Data Extraction and Loading
- 07 - Getting started with ELT tools - An introduction to Airbyte
- 08 - Deploying Airbyte for data synchronization
- 09 - Setting up sources and destinations in Airbyte
- 10 - Establishing connections in Airbyte
- 11 - Synchronizing and navigating through data
3. Starting Data Transformation and Modeling
- 12 - Introduction to data modeling with dbt
- 13 - Understanding the structure of a dbt project
- 14 - Initiating your dbt project
- 15 - Configuring data sources in dbt
- 16 - Challenge - Add a freshness check
- 17 - Solution - Add a freshness check
4. Data Transformation and Modeling
- 18 - Creating and customizing your dbt models
- 19 - Reviewing and executing dbt
- 20 - Securing your data with dbt tests
- 21 - Challenge - Add tests to the Marts model
- 22 - Solution - Add tests to the Marts model
- 23 - Automating documentation in dbt
- 24 - Completing your dbt project - A full development cycle
5. Data Orchestration
- 25 - Introduction to data orchestration with Dagster
- 26 - Integrating dbt models with Dagster assets
- 27 - Integrating Airbyte connections with Dagster assets
- 28 - Materializing assets using Dagit
- 29 - Challenge - Add a schedule to your data pipeline
- 30 - Solution - Add a schedule to your data pipeline
Conclusion
- 31 - An evolving field