Kafka Essentials: Quick Start for Building Effective Data Pipelines by Pearson

Course details

Apache Kafka is a popular message broker providing data flow management between producer application sources and consumer destinations. In this course, experienced consultant Douglas Eadline explores the many essential and practical aspects of using and running the Apache Kafka event streaming platform. Discover the ins and outs of Kafka, including core concepts, operations considerations, streaming weather data, moving Kafka topic data to external storage, image streaming with Kafka Python, data pipelines, and Kafka Connect. Along the way, discover quick tips and best practices on installation, administration, and troubleshooting.

Learning objectives
Write a Kafka Python application to produce and consume data.
Use the KafkaEsque GUI.
Use keys and multiple partitions with Kafka topics.
Develop Python consumers that access data by index or time stamp.
Save Kafka log data to external storage and databases.
Use and configure Kafka Connect services.
Conduct image streaming with Kafka size hardware for a Kafka cluster.
Install Kafka and Zookeeper across multiple servers.
Configure partitions across multiple brokers.
Administer Kafka broker partition allocation, log file management, topic management, monitoring, and benchmarking.

Concepts

Introduction

Course welcome

Kafka Background Concepts

Topics
Why do I need a message broker
Understanding Kafka components
Running Kafka on the LHM virtual machine
Understanding basic Kafka usage

Viewing Kafka Operations

Topics
Exploring the KafkaEsque UI features
Replaying basic examples with KafkaEsque

Streaming NOAA Weather Data with Kafka Python

Topics
Understanding basic Kafka Python producers and consumers
Constructing a Python producer using keys and multiple partitions
Constructing a Python consumer using keys, offsets, and time windows

Moving Kafka Topic Data to External Storage

Topics
Using a PySpark consumer to stream data to HDFS
Using a Python consumer to write data to MariaDB (MySQL)
Using a Python consumer to write data to HBASE

Edge Image Streaming with Kafka Python

Topics
Detecting edge 3D-printing problems
Understanding image capture, streaming, and real-time CNN analysis using Kafka producers and consumers
Emulating an edge 3D-printing streaming example on the LHM virtual machine

Data Pipelines and Kafka Connect

Topics
Understanding Kafka Connect design and use cases
How to manage Kafka connectors
Kafka Connect examples using files, HDFS, and MariaDB (MySQL)

Installation Considerations

Topics
Installing Zookeeper and Kafka
Understanding hardware choices
Creating a Kafka cluster
Configuring partitions

Basic Administration Topics

Topics
Understanding partition allocation and management
Understanding file management and compaction
Learning Kafka topic operations
Using Kafka console consumer and producer tools

Conclusion

Summary and next steps