Databricks Certified Data Engineer Associate Cert Prep: 4 Production Pipelines

Databricks Certified Data Engineer Associate Cert Prep: 4 Production Pipelines

19mIntermediate2024-01-24

Authors

Noah Gift

Noah Gift

MLOps Expert | Solopreneur | Author | Adjunct Professor | CTO

Course details

The Databricks Certified Data Engineer Associate certification exam assesses your ability to use the Databricks Lakehouse Platform to complete basic data engineering tasks. It also assesses your ability to use Apache Spark SQL and Python to perform multihop architecture ETL tasks that incrementally process new data from a variety of data sources to power analytic applications and dashboards in the Lakehouse.

In this course, instructor Noah Gift provides hands-on, practical instruction on the core concepts and skills covered in the fourth section of the exam: Production Pipelines. Explore the fundamentals of creating and managing data pipelines with Delta Live Tables, including vacuuming and garbage collection, table documentation, automated pipelines, continuous versus triggered pipelines, configuring Auto Loader, querying pipeline events, and much more.

Skills covered

DatabricksData EngineeringData AnalysisCert PrepData ScienceBusiness Analysis and StrategyBusiness Software and Tools

Concepts

Data Pipelines with Delta Live Tables

  • Vacuuming and garbage collection
  • Table documentation
  • Automated pipelines with Delta Live Tables
  • Delta Live Tables components
  • Continuous vs. triggered pipelines
  • Configuring Auto Loader
  • Querying pipeline events
  • End-to-end example of Delta Live Table