Special offers now — see discounted courses.
day
:
hour
:
min
:
sec
See special offers
End-to-End Data Engineering Project

End-to-End Data Engineering Project

1h 16mIntermediate2023-11-03

Authors

Thalia Barrera

Thalia Barrera

Course details

The world of data engineering is ever-changing, with new tools and technologies emerging on a regular basis. Building an effective analytics platform can be a daunting task, especially if you’re not familiar with all the tools available. How do you turn scattered, complex data into a model that drives insights and decision-making? In this course, Thalia Barrera teaches data professionals how to implement an end-to-end data engineering project using open tools from the modern data stack. She touches on best practices such as data modeling, testing, documentation and version control and shows you how to efficiently extract, load, and transform data into a unified, analytics-ready format. Thalia shows you how to confidently select and use tools through practical examples—taking you through the construction of a robust data pipeline for a fictional ecommerce company—and how to implement best practices in data engineering.

Skills covered

Data EngineeringProjectData Science

Concepts

0. Introduction

  • 01 - Transform complex data into insights
  • 02 - What you should know

1. Project Overview and Preparation

  • 03 - Project architecture overview
  • 04 - Project setup
  • 05 - Understanding the Big Star Collectibles database
  • 06 - Setting up your data warehouse

2. Data Extraction and Loading

  • 07 - Getting started with ELT tools - An introduction to Airbyte
  • 08 - Deploying Airbyte for data synchronization
  • 09 - Setting up sources and destinations in Airbyte
  • 10 - Establishing connections in Airbyte
  • 11 - Synchronizing and navigating through data

3. Starting Data Transformation and Modeling

  • 12 - Introduction to data modeling with dbt
  • 13 - Understanding the structure of a dbt project
  • 14 - Initiating your dbt project
  • 15 - Configuring data sources in dbt
  • 16 - Challenge - Add a freshness check
  • 17 - Solution - Add a freshness check

4. Data Transformation and Modeling

  • 18 - Creating and customizing your dbt models
  • 19 - Reviewing and executing dbt
  • 20 - Securing your data with dbt tests
  • 21 - Challenge - Add tests to the Marts model
  • 22 - Solution - Add tests to the Marts model
  • 23 - Automating documentation in dbt
  • 24 - Completing your dbt project - A full development cycle

5. Data Orchestration

  • 25 - Introduction to data orchestration with Dagster
  • 26 - Integrating dbt models with Dagster assets
  • 27 - Integrating Airbyte connections with Dagster assets
  • 28 - Materializing assets using Dagit
  • 29 - Challenge - Add a schedule to your data pipeline
  • 30 - Solution - Add a schedule to your data pipeline

Conclusion

  • 31 - An evolving field

About us

LyndaKade is a leading learning platform that helps people learn business, software, technology, and creative skills to achieve personal and professional goals.

Phone numberAparat ChannelTelegram SupportTelegram ChannelInstagram Page

All rights to this site belong to LyndaKade.

Terms of Service|Privacy Policy

نماد الکترونیک enamad در صورت اتصال با آی‌پی داخل کشور، نمایش داده خواهد شد.
logo-samandehi - لوگو ساماندهی
Zarinpal
Zibal