Data Engineering Pipeline Management with Apache Airflow

Course details

Embark on a comprehensive journey into data engineering pipeline management with Apache Airflow. Learn how to navigate role-based access control to ensure secure and effective data pipeline operations. Review the intricacies of execution timeouts and deadline management to keep your workflows efficient and timely. Discover how to schedule Directed Acyclic Graphs (DAGs) based on data asset updates, maximizing your pipeline's responsiveness. Find out how to leverage Airflow plugins to customize and enhance your user interface for improved productivity. See how to manage task concurrency with pools, and learn how to scale Airflow using the Celery executor for distributed processing. This course caters to data engineering professionals eager to gain knowledge an skills in pipeline management. By the end of this course, you'll be more adept at orchestrating and scaling complex data workflows with precision and confidence.

Concepts

Introduction

Features for data engineering pipeline management
Prerequisites
Quick install overview

Default Authentication with Simple Auth Manager

Authentication management and role-based access control
Configuring users in the Simple Auth Manager
Simple Auth Manager - the Admin, Op, User, and Viewer roles
Configuring multi-teams in the Simple Auth Manager
Associate resources with different teams
Resource isolation using teams

Role-Based Access Control with FAB Auth Manager

Configuring the FAB Auth Manager
Exploring predefined roles in the FAB Auth Manager
Creating users with different roles
Executing a simple branching DAG
Executing a simple SQL DAG
The Public and Viewer roles
The User role
The Op role
Actions, resources, and permissions
Adding permissions to the Public role
Creating and configuring a custom role

ExecutionTimeout and Deadline Management

Deadline management
Configure email notifications for execution timeout
Configuring task-level execution timeouts
Configuring task retries and retry delay
Configuring deadlines alerts for DAG runs
Configuring failure callbacks for DAG runs

Scheduling DAGs with Assets

Asset-aware scheduling
Asset producer pipeline
Asset consumer pipeline
Implementing asset-aware scheduling
Purchases producer pipeline and join consumer pipeline
Asset-aware scheduling with multiple assets and multiple DAGs

Customizing UI with Plugins

Plugins
Introducing plugins
Adding menu items using plugins
The CSV Reader Plugin in action
An overview of the CSV Reader Plugin
An overview of the Plugin Jinja templates

Managing Concurrency with Pools

Concurrency and pools
Running tasks in the default pool
Limiting concurrency using pools
Configuring pools slots for resource-intensive tasks
Configuring concurrency for multiple DAGs

Scaling Airflow with Celery

Scaling Apache Airflow
Installing the Celery provider
DAG for the transformation pipeline
Running RabbitMQ on a Docker container
Set up an Admin user for RabbitMQ
Configuring the Celery Executor for Airflow
Executing tasks on a single Celery worker
Executing tasks on multiple Celery workers
Assigning tasks to queues

Conclusion

Summary and next steps