End-to-End Real-World Data Engineering Project with Snowflake

Course details

This comprehensive course offers a hands-on deep dive into building a robust data pipeline using Snowflake, focusing on a real-world project case study. Join instructor Deepak Goyal as he shows you how to transform a data lake into a sophisticated Snowflake-based solution. Deepak covers the entire data lifecycle from ingestion to analytics-ready datasets. Learn how an end-to-end project works in Snowflake and how to implement the three-layer medallion architecture—bronze, silver, and gold. Ideal for professional data engineers as well as data analysts, this course can help you take your technical skills to the next level and put you in a more in-demand position to land the data role of your dreams.

Concepts

0. Introduction

01 - Project kickoff - Data lake to Snowflake
02 - What you should know

1. Snowflake Real-World Project Details

03 - Snowflake real-world project case study - Overview
04 - Data lake to Snowflake - High-level solution
05 - Snowflake project architecture diagram

2. Snowflake Project Setup

06 - ADLS account creation
07 - Snowflake account creation
08 - Loading the sample data in ADLS
09 - Create external storage in Snowflake

3. Snowflake Bronze Layer Implementation

10 - Create the database in Snowflake and the Bronze layer schema
11 - Create the CSV file format in Snowflake to load CSV file data
12 - Create a Snowflake task to load customer data into the raw table
13 - Create the JSON file format in Snowflake to load JSON file data
14 - Create a Snowflake task to load product data into the raw table
15 - Create the Parquet file format in Snowflake to load Parquet file data
16 - Create a Snowflake task to load order data into the raw table

4. Snowflake Silver Layer Implementation

17 - Create the Silver layer schema in the database
18 - Create the stream in Snowflake to capture Customer table changes
19 - Create the stream in Snowflake to capture Product table changes
20 - Create the stream in Snowflake to capture Order table changes
21 - Create the task to merge customer changes in the Silver layer
22 - Create the task to merge product changes in the Silver layer
23 - Create the task to merge order changes in the Silver layer

5. Snowflake Gold Layer Implementation

24 - Create the Gold layer schema in the database
25 - Create the Gold layer demo table
26 - Create View 1 in the Gold layer
27 - Create View 2 in the Gold layer

Conclusion

28 - Next steps