Apache Flink: Exploratory Data Analytics with SQL

1h 8mAdvanced2020-02-21

Authors

Kumaran Ponnambalam

Working with data for 20+ years

Course details

Exploratory data analytics is a key phase in data science that deals with investigating data to extract insights. In a world of big data, exploring massive datasets is a challenge, since it requires technologies that are scalable, fast, and feature rich. Apache Flink—the popular stream-processing platform—is well suited for this effort. This course focuses on exploring datasets with SQL on Apache Flink. Instructor Kumaran Ponnambalam starts off by reviewing the relational APIs that Flink provides for big data analytics. Kumaran then takes a deeper look at the Table API and SQL functions. He explores various SQL capabilities available for exploring data, including filtering, aggregations and joins. To wrap up, he provides a use case project that allows you to practice your new skills.

Topics include:
- Connectors and integrations available in Flink APIs
- Creating tables from a CSV
- Selecting and filtering table data
- Using aggregation functions in SQL
- Joining tables
- Windowing on streams
- Event time with Flink tables

Skills covered

ApacheSQLDatabase AdministrationData EngineeringDatabase DevelopmentDatabase ManagementData AnalysisProgramming LanguagesData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceSoftware DevelopmentDeep Dive (X:Y)

Concepts

Introduction

Apache Flink for exploratory analysis

Flink Relational APIs

What is Apache Flink
Flink relational APIs
Integrations and connectors
Course prerequisites
Setting up the exercise files

Basic Batch Analytics

Creating a table environment
Creating tables from a CSV
Selecting table data
Filtering data in tables
Writing tables to files

Advanced Batch Analytics

Aggregations on tables
Ordering and limiting data
Adding new columns
Joining tables
Working with datasets

Streaming SQL

Challenges with streaming SQL
Dynamic tables
Appending and retracting data
Consuming Kafka sources
Running continuous queries

Advanced Streaming Analytics

Windowing on streams
Using tumbling and sliding windows
Writing tables to Kafka
Working with data streams
Using event time

Use Case Project

Use case problem definition
Read source data into a Flink table
Compute total scores
Compute aggregations

Conclusion

Next steps