Apache Kafka Essential Training: Building Scalable Applications (2021)

Course details

Scalable and distributed message queuing plays an important role in building real time big data pipelines. Asynchronous publisher/subscriber models are required to handle unpredictable loads in these pipelines. Apache Kafka is the leading technology today that provides these capabilities and is an essential skill for a big data professional. In this course, Kumaran Ponnambalam provides insights into the scalability and manageability aspects of Kafka and demonstrates how to build asynchronous applications with Kafka and Java. Kumaran starts by demonstrating how to set up a Kafka cluster and explores the basics of Java programming in Kafka. He then takes a deep dive into the various messaging and schema options available. Kumaran also goes over some best practices for designing Kafka applications before finishing with a use case project that applies the lessons covered in the course.

Concepts

Introduction

Why are Kafka skills so high in demand

Introduction to Kafka

What is Kafka
Prerequisites for the course
Kafka scaling and resiliency
Setting up the exercise files

Kafka Scaling Concepts

Clusters and controllers
Replication
Partition leaders
Mirroring
Security

Building a Kafka Cluster

Kafka cluster setup
Running the cluster
Creating topics with replication
Kafka cluster in action
Kafka resiliency in action

Building Scalable Producers

Producer internals
Producer publishing options
Acknowledgments in Kafka
Additional producer parameters
Java producer options example

Building Scalable Consumers

Consumer How it works
Batching message consumption
Committing messages
Java consumer example
Multi-threaded consumers

Kafka Best Practices

Managing partition counts
Managing messages
Managing consumer settings
Managing resiliency

Use Case Project

Kafka applications use case Problem definition
Setting up topics
Producing data in Java
Consuming data in Java

Conclusion

How can you extend your Kafka learning journey