Learn Apache Kafka for Beginners (2019)

Course details

Kafka is the leading open-source, enterprise-scale data streaming technology. It helps you move your data where you need it, in real time, reducing the headaches that come with integrations between multiple source and target systems. This training course helps you get started with all the fundamental Kafka operations, explore the Kafka CLI and APIs, and perform key tasks like building your own producers and consumers. Learn how to start a personal Kafka cluster on Mac, Windows, or Linux; master fundamental concepts including topics, partitions, brokers, producers, and consumers; and start writing, storing, and reading data with producers, topics, and consumers. Instructor Stephane Maarek includes practical use cases and examples, such as consuming data from sources like Twitter and ElasticSearch, that feature real-world architecture and production deployments. Plus, learn how to start Kafka from annex locations, such as Docker containers and remote machines, and launch Kafka clusters.

Learning objectives
Apache Kafka basics
Kafka theory and architecture
Setting up Kafka to run on Mac, Linux, and Windows
Working with the Kafka CLI
Creating and configuring topics
Writing Kafka producers and consumers in Java
Writing and configuring a Twitter producer
Writing a Kafka consumer for ElasticSearch
Working with Kafka APIs: Kafka Connect, Streams, and Schema Registry
Kafka case studies
Kafka monitoring and security
Advanced Kafka configuration
Starting Kafka using binaries, Docker, and remote machines

Concepts

0. Introduction

01 - Intro to Apache Kafka
02 - Apache Kafka in five minutes

1. Kafka Theory

03 - Kafka theory overview
04 - Topics, partitions, and offsets
05 - Brokers and topics
06 - Topic replication
07 - Producers and message keys
08 - Consumer and consumer group
09 - Consumer offsets and delivery semantics
10 - Kafka broker discovery
11 - ZooKeeper
12 - Kafka guarantees
13 - Theory roundup

2. Starting Kafka

14 - Important - Starting Kafka
15 - macOS - Download and set up Kafka in PATH
16 - macOS - Using brew
17 - macOS - Start ZooKeeper and Kafka
18 - Linux - Download and set up Kafka in PATH
19 - Linux - Start ZooKeeper and Kafka
20 - Windows - Download and set up Kafka in PATH
21 - Windows - Start ZooKeeper and Kafka

3. Command Line Interface (CLI) 101

22 - CLI introduction
23 - Kafka topics CLI
24 - Kafka console producer CLI
25 - Kafka console consumer CLI
26 - Kafka consumers in groups
27 - Kafka consumer groups CLI
28 - Resetting offsets
29 - Kafka Tool UI

4. Kafka Java Programming 101

30 - Intro to Kafka programming
31 - Creating a Kafka project
32 - Java producer
33 - Java producer callbacks
34 - Java producer with keys
35 - Java consumer
36 - Java consumer inside consumer group
37 - Java consumer with threads
38 - Java consumer seek and assign
39 - Client bidirectional compatibility

5. Kafka Real-World Project

40 - Real-world project overview

6. Kafka Twitter Producer and Advanced Configurations

41 - Producer and advanced configurations overview
42 - Twitter setup
43 - Producer, part 1 - Writing a Twitter client
44 - Producer, part 2 - Writing the Kafka producer
45 - Producer configurations introduction
46 - Acks and min.insync.replicas
47 - Retries and max.in.flight.requests.per.connection
48 - Idempotent producer
49 - Producer, part 3 - Safe producer
50 - Producer compression
51 - Producer batching
52 - Producer, part 4 - High throughput producer
53 - Producer default partitions and key hashing
54 - Refactoring the project

7. Kafka Elasticsearch Consumer and Advanced Configurations

55 - Consumer and advanced configuration overview
56 - Setting up Elasticsearch in the cloud
57 - Elasticsearch 101
58 - Consumer, part 1 - Set up project
59 - Consumer, part 2 - Write the consumer and send to Elasticsearch
60 - Consumer, part 3 - Delivery semantics
61 - Delivery semantics for consumers
62 - Consumer, part 3 - Idempotence
63 - Consumer poll behavior, part 1
64 - Consumer offset commit strategies
65 - Consumer, part 4 - Manual commit of offsets
66 - Consumer, part 5 - Performance improvement using batching
67 - Consumer offsets reset behavior
68 - Consumer, part 6 - Replaying data
69 - Consumer internal threads

8. Kafka Ecosystem and Real-World Architectures

70 - Kafka in the real world

9. Kafka Extended APIs

71 - Kafka Connect introduction
72 - Kafka Connect Twitter - Hands-on example
73 - Kafka Streams introduction
74 - Kafka Streams - Hands-on example
75 - Kafka Schema Registry introduction

10. Real-World Insights and Case Studies

76 - ZooKeeper
77 - Case study - MovieFlix
78 - Case study - GetTaxi
79 - Case study - MySocialMedia
80 - Case study - MyBank
81 - Case study - Big data ingestion
82 - Case study - Logging and metrics aggregation

11. Kafka in the Enterprise for Admins

83 - Kafka cluster setup, high-level architecture overview
84 - Kafka monitoring and operations
85 - Kafka security
86 - ZooKeeper

12. Advanced Topic Configurations

87 - Changing a topic configuration
88 - Segment and indexes
89 - Log cleanup policies
90 - Log cleanup delete
91 - Log compaction theory
92 - Log compaction practice
93 - min.insync.replicas reminder
94 - Unclean leader election

13. Annexes

95 - What are annexes

14. Starting Kafka Differently

96 - Annex 1 - Overview
97 - Starting Kafka with the Confluent CLI
98 - Starting a multibroker Kafka cluster using binaries
99 - Start Kafka development environment using Docker
100 - Starting a multibroker Kafka cluster using Docker
101 - Kafka advertised host setting
102 - Starting Kafka on a remote machine