Apache Airflow Essential Training

Course details

Eager to elevate your workflow management skills? In this course, discover the latest capabilities of the TaskFlow API as you learn how to define and manage workflows. Find out how to integrate PostgreSQL with Airflow for seamless data operations across systems. Learn how to leverage sensors to monitor crucial external events and automate responses, enhancing the efficiency of your workflows. Review how to utilize hooks for streamlined access to external databases, cloud platforms, and APIs. This course is well-suites for data engineers, software developers, and IT professionals driven to optimize workflow automation processes. By the end of this course, you will be more equipped to orchestrate complex data pipelines with confidence and precision.

Concepts

Introduction

Apache Airflow Essential Training
Prerequisites

Working with the TaskFlow API

Quick Airflow setup overview
DAG using Python operators
DAG using TaskFlow
Passing data using XCom with operators
Passing data using the TaskFlow API
Tasks with multiple outputs
Passing multiple outputs in TaskFlow
Passing context variables and custom arguments

Branching and Conditional Tasks with TaskFlow

Branching using operators
Branching using the TaskFlow API
Modeling data and task dependencies with TaskFlow
Cleaner DAG structures by modeling just task dependencies
Interoperating with traditional tasks
Setup, teardown, and conditional decorators in TaskFlow

Trigger Rules for Tasks

The all success trigger rule
The none failed trigger rule
Nuances of trigger rules
The none failed min one success trigger rule

Integrating PostgreSQL in Airflow

Installing PostgreSQL on macOS
Installing PostgreSQL on WSL
Connecting to PostgreSQL
Using the SQLExecuteQueryOperator
Performing PostgreSQL insert operations
Performing PostgreSQL join operations
A complete end-to-end pipeline with PostgreSQL
Configuring PostgreSQL as a metadata database

Using Sensors in Airflow

Introducing the File Sensor operating in poke mode
File Sensor operating in reschedule mode
Creating a pipeline using the File Sensor
Running DAGs with a File Sensor
Creating and running a pipeline with SQL sensors
Deferrable operators

Working with Hooks in Airflow

Setting up for a PostgreSQL pipeline with hooks
Creating and running a pipeline with PostgreSQL hooks
Setting up access to Amazon S3 buckets
Setting up a connection to Amazon S3 buckets
Creating and running a pipeline with an S3 hook

Conclusion

Summary and next steps