Kafka: Building Reliable Real-Time Event Systems

Course details

Imagine building systems that react instantly to streams of data—fraud alerts triggered within milliseconds, dashboards pulsing with live updates, and digital platforms scaling to millions of users in real time. In this hands-on course on Apache Kafka, you'll learn about the basics of these capabilities through a dynamic mix of real-world examples, interactive labs, and practical demonstrations. Then, progress from the basics of event streaming to deploying advanced, production-ready architectures. By the end of this course, you will be prepared to design, deploy, and manage production-ready Kafka architectures and seamlessly integrate Kafka with other systems.

Audience
Software developers
DevOps engineers
Data engineers
Technical leads eager to design, deploy, and manage real-time, event-driven systems using industry-leading technologies

Concepts

Introduction

Course introduction

Foundations of Event Streaming

What is event streaming
Introduction to Apache Kafka
Event-driven architecture basics
Kafka in finance - Real-time transaction processing
Use cases and real-world applications - Use case 2
Demo Setting up Kafka cluster and Kafka UI using Docker
Demo Creating first topic in Kafka

Building Blocks of Kafka

Kafka brokers - The foundation of message storage
Understanding Kafka topics - Organizing your data streams
Kafka partitions - Scaling and parallelizing your data processing
Kafka replication - Ensuring data reliability and fault tolerance
Demo Topics, partitions, and brokers

Kafka Producers and Consumers - The Message Flow

What is a Kafka producer
Demo Kafka producers
Serialization in Kafka
Understanding the role of message keys
Producer acknowledgments (acks) and reliability guarantees
What is a Kafka consumer
Demo Kafka consumers
Consumer groups and how they work
Consumer rebalancing
Demo Consumer rebalancing
Section recap

Deep Dive into Kafka - Beyond the Basics

Offset management in Kafka
Poison pill in Kafka
Mitigation strategies - Handling poison pill
ZooKeeper's role in Kafka
Demo Kafka setup with ZooKeeper
Kafka KRaft
KRaft in action - New broker joining Kafka cluster
Demo Kafka setup with KRaft
Understanding Kafka security
Securing Kafka
Section recap

Confluent Kafka and Its Offerings

Beyond installation - the true complexity of managing Kafka
Confluent Cloud - Kafka reimagined for the cloud era
Demo Getting started with Confluent for free

Kafka Connect - Effortless Data Pipelines

Streaming data from Kafka to other systems
Demo Kafka Connect - Setting up Kafka using KRaft
Demo Setting up S3 Kafka Connect
Demo Kafka Connect in action

Building an Event-Driven System

Architecture overview of event-driven system
Demo Setting up Kafka on EC2
Demo Starting our frontend interface
Demo Starting our internal interface
Section recap