End-to-End Real-World Data Engineering Project with Databricks

Course details

This comprehensive course offers a practical, hands-on approach to building a modern data pipeline using Databricks. Join instructor Deepak Goyal as he guides you through a real-world data engineering project that transforms a data lake into an efficient, analytics-ready solution using the Databricks ecosystem. Learn how an end-to-end project works in Databricks, how to implement the three-layer medallion architecture—bronze, silver, and gold—and how to use Spark SQL in a real-world data project. Ideal for professional data engineers as well as data analysts, this course can help you take your technical skills to the next level and put you in a more in-demand position to land the data role of your dreams.

Concepts

0. Introduction

01 - Project kickoff - Data engineering with Databricks
02 - What you should know

1. Databricks Real-World Project Details

03 - Databricks case study - Overview
04 - Data lakehouse - High-level solution
05 - Databricks project architecture diagram

2. Databricks Project Setup

06 - Databricks account creation
07 - Initial folder creation
08 - Loading the sample data

3. Databricks Bronze Layer Implementation

09 - Create the Bronze layer database
10 - Create a Databricks notebook to load a CSV file to the customer delta table
11 - Create a Databricks notebook to load a JSON file to the product delta table
12 - Create a Databricks notebook to load a Parquet file to the order delta table

4. Databricks Silver Layer Implementation

13 - Create the Silver layer database
14 - Create a Databricks notebook to load the Silver layer - Customer delta table
15 - Create a Databricks notebook to load the Silver layer - Product delta table
16 - Create a Databricks notebook to load the Silver layer - Order delta table

5. Databricks Gold Layer Implementation

17 - Create the Gold layer database
18 - Create a Databricks notebook to load the Gold table - Part 1
19 - Create a Databricks notebook to load the Gold table - Part 2

6. Databricks and Power BI Data Visualization

20 - Install Power BI Desktop
21 - Integrate Databricks and Power BI
22 - Create the Power BI dashboard from Databricks

Conclusion

23 - Next steps