Stream Processing Design Patterns with Kafka Streams
1h 8mAdvanced2020-11-02
Authors
Kumaran Ponnambalam
Working with data for 20+ years
Course details
Stream processing is rapidly growing in popularity, as more and more data is generated every day by websites, devices, and communications. Platforms such as Apache Kafka Streams can help you build fast, scalable stream processing applications, but big data engineers still need to design smart use cases to achieve maximum efficiency. In this course, get insight into how to solve stream processing problems with Kafka Streams in Java as you learn how to build use cases with popular design patterns. Review some of the significant features of Kafka Streams and discover four popular patterns for stream processing: streaming analytics, alerts and thresholds, leaderboards, and real-time predictions. Along the way, review example use cases, and discover how to leverage Kafka Streams, as well as key technologies like MariaDB and Redis, to implement key examples.
Skills covered
KafkaApacheData EngineeringData ScienceDeep Dive (X:Y)
Concepts
0. Introduction
- 01 - Stream processing with Kafka
- 02 - Prerequisites
1. Stream Processing with Kafka Streams
- 03 - What is stream processing
- 04 - Streaming opportunities and challenges
- 05 - Streaming with Kafka Streams
- 06 - Kafka Streams DSL
- 07 - Setting up the exercise files
- 08 - Setting up Kafka
- 09 - Setting up MariaDB and Redis
2. Streaming Analytics
- 10 - Streaming analytics - Pattern
- 11 - Streaming analytics - Use case design
- 12 - Streaming analytics - Helper classes
- 13 - Streaming analytics - Pipeline implementation
- 14 - Streaming analytics - Results review
3. Alerts and Thresholds
- 15 - Alerts and thresholds - Pattern
- 16 - Alerts and thresholds - Use case design
- 17 - Alerts and thresholds - Helper classes
- 18 - Alerts and thresholds - Pipeline implementation
- 19 - Alerts and thresholds - Review
4. Leaderboards
- 20 - Leaderboards - Pattern
- 21 - Leaderboards - Use case design
- 22 - Leaderboards - Helper classes
- 23 - Leaderboards - Pipeline implementation
- 24 - Leaderboards - Review
5. Real-Time Predictions
- 25 - Real-time predictions - Pattern
- 26 - Real-time predictions - Use case design
- 27 - Real-time predictions - Helper classes
- 28 - Real-time predictions - Pipeline implementation
- 29 - Real-time predictions - Review
6. Use Case Project
- 30 - Use case definition
- 31 - Design of the project
- 32 - Code walk-through
- 33 - Execute and analyze
Conclusion
- 34 - Next steps