Tuning Kafka
1h 57mAdvanced2023-06-08
Authors
Janani Ravi
Certified Google Cloud Architect and Data Engineer
Course details
Looking to take your skills to the next level with Apache Kafka? If you’re an experienced Kafka user, you probably already know that the performance of your Kafka environment is affected by many factors. But how do you tune them? In this course, instructor Janani Ravi shows you how to fine-tune Kafka using the Kafka optimization theorem and other powerful tuning tools.
Get an introduction to Kafka tuning, how it affects producers and consumers, and how to set up Kafka on your machine so you can start publishing messages. Explore the basics of tuning Kafka producers such as running a cluster with multiple brokers, running performance tests, partitioning, replication, compression, batch size, and more. Discover the essentials of tuning Kafka consumers with basic performance tests, fetch bytes and wait time, session timeout, max poll intervals and records, end-to-end latency, and throughput. By the end of this course, you’ll also be prepared to run performance tests using Python.
Get an introduction to Kafka tuning, how it affects producers and consumers, and how to set up Kafka on your machine so you can start publishing messages. Explore the basics of tuning Kafka producers such as running a cluster with multiple brokers, running performance tests, partitioning, replication, compression, batch size, and more. Discover the essentials of tuning Kafka consumers with basic performance tests, fetch bytes and wait time, session timeout, max poll intervals and records, end-to-end latency, and throughput. By the end of this course, you’ll also be prepared to run performance tests using Python.
Skills covered
KafkaData Resource ManagementApacheData EngineeringAdvancedDatabase ManagementData Science
Concepts
0. Introduction
- 01 - Tuning Kafka
1. Introducing Kafka Tuning
- 02 - Prerequisites
- 03 - An overview of Kafka
- 04 - Producers and consumers
- 05 - Kafka optimization theorem
- 06 - End-to-end latency in Kafka
- 07 - Install and set up Apache Kafka
- 08 - Publish and consume messages using console scripts
2. Tuning Kafka Producers
- 09 - Running a Kafka cluster with three brokers
- 10 - Running basic producer performance tests
- 11 - Effects of partitioning on producers
- 12 - Effects of replication on producers
- 13 - Tuning producer acks
- 14 - Tuning producer compression
- 15 - Tuning batch size and linger time
- 16 - Effect of message size and record count
3. Tuning Kafka Consumers
- 17 - Running a basic consumer performance test
- 18 - Tuning fetch bytes and wait time for consumers
- 19 - Tuning session timeout and heartbeat interval
- 20 - Tuning max poll interval and max poll records
4. Tuning Kafka Brokers
- 21 - End-to-end latency with replication
- 22 - Tuning throughput
- 23 - Tuning log retention
5. Performance Testing with Python
- 24 - Producer performance testing using Python
- 25 - Consumer perf testing using Python
Conclusion
- 26 - Summary and next steps