Learning Apache Airflow
2h 54mAdvanced2026-06-16
Authors
Janani Ravi
Certified Google Cloud Architect and Data Engineer
Course details
Get an introduction to Apache Airflow with Google cloud architect and data engineer Janani Ravi. With Apache Airflow—an open-source platform for developing, scheduling, and monitoring batch oriented workflows—you can schedule individual tasks that you want to run as a part of your processing pipeline, specify dependencies between these tasks, all programmatically in Python. Airflow manages these dependencies and ensures that tasks are executed in the right order. Join Janani as she shows you how to run workflows in Airflow, define tasks and dependencies, and use Python and SQLite operators. Learn about conditional branching, and how catch up and backfill work in Apache Airflow.
Skills covered
Apache AirflowApacheIT AutomationDevOps ToolsData EngineeringDevOpsNetwork and System AdministrationData ScienceOne-Off
Concepts
Introduction
- An overview of Apache Airflow
Introducing Apache Airflow
- Apache Airflow concepts
- Airflow architecture
- Windows users - install WSL2
- Windows and macOS users - installing Airflow
- Running Airflow standalone
Running Workflows on Airflow
- Navigating and exploring the Airflow UI
- Running an example DAG
- Creating your first DAG
- Running your first DAG
- Creating a DAG using the with keyword
Defining Tasks and Dependencies
- Specifying task dependencies
- Using the bitshift operator for dependency specification
- Using bash scripts in the DAG
- Clearing task state to rerun tasks
Running Python-Based Tasks
- Introducing the Python operator
- Python operators and dependencies
- Passing parameters to Python callables
- Injecting task context
- Introducing XCom
- Accessing values in XCom
- Accessing values in XCom using custom keys
- Implementing a data transformation DAG pipeline part-I
- Implementing a data transformation DAG pipeline part-II
Executing SQL Queries in Tasks
- Setting up the SQLite database and connection
- Introducing the SQLExecuteQueryOperator
- Implementing a SQL based DAG pipeline part-I
- Implementing a SQL based DAG pipeline part-II
Conditional Branching and TaskGroups
- Introducing branching
- Conditional branching with variables
- Task groups and edge labels
Catch Up, Backfill, CRON Expressions, and Trigger Configurations
- Catch up with all previously scheduled runs
- Backfill scheduled runs
- Configuring logical dates
- Data interval and run configuration
Conclusion
- Summary and next steps