Apache Airflow Essential Training (2023)

Course details

If you already know the basics of Apache Airflow but are eager to learn more, this course is for you. Certified Google cloud architect and data engineer Janani Ravi guides you through several essential features in Apache Airflow. After a quick review of what Airflow is all about, Janani dives into working with the TaskFlow API and using it to perform a variety of operations. She shows you how to integrate PostgreSQL in Airflow, as well as how to use sensors and work with hooks in Airflow. Level up your use of Apache Airflow with this essential course.

Concepts

Introduction

Apache Airflow essential training

Working with the TaskFlow API

Prerequisites
Quick Airflow setup overview
DAG using PythonOperators
DAG using TaskFlow
Passing data using XCom with operators
Passing data using the TaskFlow API
Tasks with multiple outputs
Passing multiple outputs in TaskFlow

Performing Operations Using TaskFlow

Branching using operators
Branching using the TaskFlow API
Complete branching pipeline using TaskFlow
Interoperating and passing data between operators and TaskFlow
Performing SQL operations with TaskFlow
Data transformation and storage using TaskFlow

Integrating PostgreSQL in Airflow

Installing PostgreSQL on macOS
Installing PostgreSQL on WSL
Connecting to PostgreSQL
Using the PostgreSQL operator
Performing PostgreSQL insert operations
Performing PostgreSQL join operations
A complete end-to-end pipeline with PostgreSQL
Configuring PostgreSQL as a metadata database and using the LocalExecutor

Using Sensors in Airflow

Introducing the FileSensor
Creating a pipeline using the FileSensor
Running DAGs with a FileSensor
Creating and running a pipeline with SQLSensors

Working with Hooks in Airflow

Setting up for a PostgreSQL pipeline with hooks
Creating and running a pipeline with PostgresSQL hooks
Setting up access to Amazon S3 buckets
Setting up a connection to Amazon S3 buckets
Creating and running a pipeline with an S3 hook

Conclusion

Summary and next steps