Learning Apache Airflow

Course details

Get an introduction to Apache Airflow with Google cloud architect and data engineer Janani Ravi. With Apache Airflow—an open-source platform for developing, scheduling, and monitoring batch oriented workflows—you can schedule individual tasks that you want to run as a part of your processing pipeline, specify dependencies between these tasks, all programmatically in Python. Airflow manages these dependencies and ensures that tasks are executed in the right order. Join Janani as she shows you how to run workflows in Airflow, define tasks and dependencies, and use Python and SQLite operators. Learn about conditional branching, and how catch up and backfill work in Apache Airflow.

Concepts

Introduction

An overview of Apache Airflow

Introducing Apache Airflow

Apache Airflow concepts
Airflow architecture
Windows users - install WSL2
Windows and macOS users - installing Airflow
Running Airflow standalone

Running Workflows on Airflow

Navigating and exploring the Airflow UI
Running an example DAG
Creating your first DAG
Running your first DAG
Creating a DAG using the with keyword

Defining Tasks and Dependencies

Specifying task dependencies
Using the bitshift operator for dependency specification
Using bash scripts in the DAG
Clearing task state to rerun tasks

Running Python-Based Tasks

Introducing the Python operator
Python operators and dependencies
Passing parameters to Python callables
Injecting task context
Introducing XCom
Accessing values in XCom
Accessing values in XCom using custom keys
Implementing a data transformation DAG pipeline part-I
Implementing a data transformation DAG pipeline part-II

Executing SQL Queries in Tasks

Setting up the SQLite database and connection
Introducing the SQLExecuteQueryOperator
Implementing a SQL based DAG pipeline part-I
Implementing a SQL based DAG pipeline part-II

Conditional Branching and TaskGroups

Introducing branching
Conditional branching with variables
Task groups and edge labels

Catch Up, Backfill, CRON Expressions, and Trigger Configurations

Catch up with all previously scheduled runs
Backfill scheduled runs
Configuring logical dates
Data interval and run configuration

Conclusion

Summary and next steps