Data Engineering Pipeline Management with Apache Airflow (2023)
2h 9mIntermediate2023-08-15
Authors
Janani Ravi
Certified Google Cloud Architect and Data Engineer
Course details
Take a deeper dive into data engineering pipeline management using Apache Airflow. In this course, certified Google cloud architect and data engineer Janani Ravi guides you through using Apache Airflow to complete your data engineering pipeline management workflows. Learn how to work with role-based access control, including creating users with different roles, executing a branching DAG and a SQL DAG, recalling actions that users with different roles can perform, and more. Go over how to manage SLAs and schedule DAGs with datasets. Find out how to work with AirFlow plugins and explore the CSV reader plugin. Plus, discover how to scale Apache Airflow, set up a data transformation pipeline, execute tasks, and more.
Skills covered
Apache AirflowApacheOperations ManagementDevOps ToolsData EngineeringDevOpsProject ManagementData ScienceBusiness Analysis and StrategyOne-Off
Concepts
Introduction
- Features for data engineering pipeline management
Working with Role-Based Access Control
- Prerequisites
- Quick install overview
- Creating an admin user and exploring roles
- Creating users with different roles
- Executing a simple branching DAG
- Executing a simple SQL DAG
- The public and viewer roles
- The user role
- The op role
- Actions, resources, and permissions
- Adding permissions to the public role
- Creating and configuring a custom role
Managing SLAs
- Configuring emails for SLA management
- Configuring task-level SLAs
- Triggering and viewing SLA misses
- Configuring DAG-level SLAs
- Configuring DAG failed action
Scheduling DAGs with Datasets
- Dataset producer pipeline
- Dataset consumer pipeline
- Data-aware scheduling
- Purchases producer pipeline and join pipeline
- Data-aware scheduling with multiple datasets
Working with Airflow Plugins
- Introducing plugins
- Adding menu items using plugins
- Exploring the CSV reader plugin
- Implementing the CSV reader plugin
Scaling Airflow
- Scaling Apache Airflow
- Basic setup for the transformation pipeline
- DAG for the transformation pipeline
- Install RabbitMQ on macOS and Linux
- Set up an admin user for RabbitMQ
- Configuring the CeleryExecutor for Airflow
- Executing tasks on a single Celery worker
- Executing tasks on multiple Celery workers
- Assigning tasks to queues
Conclusion
- Summary and next steps