Architecting Big Data Applications: Batch Mode Application Engineering
1h 28mAdvanced2023-10-06
Authors
Kumaran Ponnambalam
Working with data for 20+ years
Course details
Big data applications allow data scientists and analysts to acquire, store, manage, and use big data to generate more consistent, data-driven results. In this course, instructor Kumaran Ponnambalam explores real-world business use cases and best practices for architecting big data applications using existing open-source technologies.
Learn how to architect both simple and complex batch processing applications, as you discover the basic principles of big data architectures such as horizontal scaling, distributed processing, technology selection and integration, and scheduling. Kumaran shows you how to minimize data volumes and distribute data loads uniformly, as well as how to use caches, reprocess data, troubleshoot errors, and more. Along the way, take your new skills to the next level with hands-on use cases that cover a variety of functional and technology domains.
Learn how to architect both simple and complex batch processing applications, as you discover the basic principles of big data architectures such as horizontal scaling, distributed processing, technology selection and integration, and scheduling. Kumaran shows you how to minimize data volumes and distribute data loads uniformly, as well as how to use caches, reprocess data, troubleshoot errors, and more. Along the way, take your new skills to the next level with hands-on use cases that cover a variety of functional and technology domains.
Skills covered
HiveApache SparkData CentersApacheData EngineeringDatabase ManagementData AnalysisData ScienceBusiness Analysis and StrategyBusiness Software and ToolsDeep Dive (X:Y)
Concepts
Introduction
- Architecting big data applications
Introduction to Big Data Applications
- Characteristics of batch processing
- Challenges building batch applications
- Technologies for batch big data engineering
- Use cases for batch big data
- Architecture process for data engineering
Big Data Architecture Principles
- Making the choice - Real-time vs. batch
- Horizontal scaling
- Distributed processing
- Technology selection
- Technology integrations
Batch Application Architecture Principles
- Schedule selection
- Minimizing data volumes
- Uniform load distribution
- Using caches
- Reprocessing
Use Case 1 - Audit Trail Data Archive
- Audit trail - Define the problem
- Audit trail - Study requirements
- Audit trail - Create a workflow
- Audit trail - Scale the workflow
- Audit trail - Select technologies
- Audit trail - Review final architecture
Use Case 2 - Advertising Analytics
- Advertising analytics - Define the problem
- Advertising analytics - Study requirements
- Advertising analytics - Create a workflow
- Advertising analytics - Scale the workflow
- Advertising analytics - Select technologies
- Advertising analytics - Review final architecture
Use Case 3 - Product Recommendations
- Product recommendations - Define the problem
- Product recommendations - Study requirements
- Product recommendations - Create a workflow
- Product recommendations - Scale the workflow
- Product recommendations - Select technologies
- Product recommendations - Review the final architecture
Conclusion
- Continuing to architect big data applications