Apache Kafka Essential Training: Getting Started

Course details

One of the key components of a big data processing pipeline is a scalable and distributed message queue. Message queues enable real-time streaming capabilities with multiple producers and consumers of data. This enables real-time applications that can analyze data and produce insights in a scalable fashion. Apache Kafka provides these capabilities. As the de facto standard for open-source messaging, Apache Kafka is an essential skill for data scientists, big data engineers, data architects, and solution architects. In this course, instructor Kumaran Ponnambalam introduces Apache Kafka and explains its fundamental concepts and basic operations. Kumaran covers basic concepts like messages, topics, logs, and more. He shows you how to use the Kafka command line, as well as partitions and groups. He goes over Kafka Java programming, then concludes with a use case project.

Concepts

Introduction

Getting started with Apache Kafka

Introduction to Kafka

Message queues
What is Kafka
Benefits of Kafka
Kafka use cases
Setting up the exercise files
Setting up Kafka

Basic Concepts

Messages
Topics
Kafka brokers
Logs in Kafka
Producers and consumers

Using Kafka Command Line

Kafka client scripts
Creating a topic
Exploring topics
Publishing messages to topics
Using console consumer
Topics management
Review Kafka setup

Kafka Partitions and Groups

Intro to partitions
Creating topics with partitions
Publishing with keys
Consumer groups
Consumer offset management
Consuming partitioned data

Kafka Java Programming

Kafka client libraries
Creating a producer in Java
Publishing messages in Java
Creating a consumer in Java
Consuming messages in Java

Use Case Project

Kafka basics use case - Problem definition
Setting up topics
Producing and consuming data with scripts
Producing and consuming data with Java

Conclusion

Extending your Kafka knowledge