Learning Apache Airflow (2023)
2h 6mAdvanced2023-07-12
Authors
Janani Ravi
Certified Google Cloud Architect and Data Engineer
Course details
Get an introduction to Apache Airflow with Google cloud architect and data engineer Janani Ravi. With Apache Airflow—an open-source platform for developing, scheduling, and monitoring batch oriented workflows— you can schedule individual tasks that you want to run as a part of your processing pipeline, specify dependencies between these tasks, all programmatically in Python. Airflow manages these dependencies and ensures that tasks are executed in the right order. Join Janani as she shows you how to run workflows in Airflow, define tasks and dependencies, and use Python and SQLite operators. Learn about conditional branching, and how catch up and backfill work in Apache Airflow.
Skills covered
Apache AirflowApacheIT AutomationDevOps ToolsData EngineeringAdvancedDevOpsNetwork and System AdministrationData Science
Concepts
Introduction
- An overview of Apache Airflow
Introducing Apache Airflow
- Apache Airflow concepts and architecture
- Initial setup - macOS and Windows
- Understanding Airflow configuration
- Running Airflow
Running Workflows on Airflow
- Navigating and exploring the Airflow UI
- Creating your first DAG
- Running your first DAG
- Creating a DAG using the with keyword
Defining Tasks and Dependencies
- Specifying task dependencies
- Using the Bitshift operators for dependency specification
- Using bash scripts in the DAG
Using the Python Operator and XCom
- Introducing the Python operator
- Python operators and dependencies
- Passing parameters to Python callables
- Introducing XCom
- Using XCom to pass values
- Implementing a data transformation DAG pipeline, part 1
- Implementing a data transformation DAG pipeline, part 2
Using the SQLite Operator for SQL Operations
- Setting up the SQLite DB and connection
- Introducing the SQLite operator
- Implementing a SQL-based DAG pipeline, part 1
- Implementing a SQL-based DAG pipeline, part 2
Conditional Branching and TaskGroups
- Introducing branching
- Conditional branching with variables
- Task groups and edge labels
Catch Up, Backfill, and CRON Expressions
- Catch up with all previously scheduled runs
- Backfill scheduled runs
Conclusion
- Summary and next steps