Apache Flink: Real-Time Data Engineering

Course details

From an engineering perspective, scalability is one of the most pressing challenges in data science. Apache Flink, the powerful and popular stream-processing platform, offers features and functionality that can help developers tackle this challenge. In this course, learn how to build a real-time stream processing pipeline with Apache Flink. Instructor Kumaran Ponnambalam begins by reviewing key streaming concepts and features of Apache Flink. He then takes a deeper look at the DataStream API and explores various capabilities available for real-time stream processing, including windowing and joins. After delving into the platform's event-time processing and state management features, he provides a use case project that allows you to put your new skills to the test.

Topics include:
- Streaming with Apache Flink
- Using the DataStream API for basic stream processing
- Working with process functions
- Windowing and joins
- Setting up event-time processing
- State management in Flink

Concepts

0. Introduction

01 - Real-time processing and analytics

1. Apache Flink

02 - What is Apache Flink
03 - Streaming with Apache Flink
04 - DataStream API
05 - Related prerequisite courses
06 - Setting up exercise files

2. DataStream API

07 - Setting up the Flink environment
08 - Reading from a stream source
09 - Processing streaming data
10 - Writing to a stream sink
11 - Using keyed streams
12 - ProcessFunction
13 - Splitting a stream
14 - Merging multiple streams

3. Windowing

15 - Windowing concepts
16 - Using a Kafka streaming source
17 - Using sliding windows
18 - Using session windows
19 - Window joins

4. Event Time Processing

20 - Time attributes in Flink
21 - Watermarks
22 - Setting up event time
23 - Processing with event time
24 - Writing to a Kafka sink

5. State Management

25 - State management in Flink
26 - Defining states
27 - Using states
28 - Advanced state management

6. Use Case Project

29 - Problem definition
30 - Computing summary counts
31 - Computing activity durations

Conclusion

32 - Next steps