End-to-End Real-World Data Engineering Project with Databricks

Course details

This comprehensive course offers a practical, hands-on approach to building a modern data pipeline using Databricks. Join instructor Deepak Goyal as he guides you through a real-world data engineering project that transforms a data lake into an efficient, analytics-ready solution using the Databricks ecosystem. Learn how an end-to-end project works in Databricks, how to implement the three-layer medallion architecture—bronze, silver, and gold—and how to use Spark SQL in a real-world data project. Ideal for professional data engineers as well as data analysts, this course can help you take your technical skills to the next level and put you in a more in-demand position to land the data role of your dreams.

Concepts

Introduction

Project kickoff - Data engineering with Databricks
What you should know

Databricks Real-World Project Details

Databricks case study - Overview
Data lakehouse - High-level solution
Databricks project architecture diagram

Databricks Project Setup

Databricks account creation
Initial folder creation
Loading the sample data

Databricks Bronze Layer Implementation

Create the Bronze layer database
Create a Databricks notebook to load a CSV file to the customer delta table
Create a Databricks notebook to load a JSON file to the product delta table
Create a Databricks notebook to load a Parquet file to the order delta table

Databricks Silver Layer Implementation

Create the Silver layer database
Create a Databricks notebook to load the Silver layer - Customer delta table
Create a Databricks notebook to load the Silver layer - Product delta table
Create a Databricks notebook to load the Silver layer - Order delta table

Databricks Gold Layer Implementation

Create the Gold layer database
Create a Databricks notebook to load the Gold table - Part 1
Create a Databricks notebook to load the Gold table - Part 2

Databricks and Power BI Data Visualization

Install Power BI Desktop
Integrate Databricks and Power BI
Create the Power BI dashboard from Databricks

Conclusion

Next steps