Big Data Analytics with Hadoop and Apache Spark (2020)
1h 2mIntermediate2020-02-24
Authors
Kumaran Ponnambalam
Working with data for 20+ years
Course details
Apache Hadoop was a pioneer in the world of big data technologies, and it continues to be a leader in enterprise big data storage. Apache Spark is the top big data processing engine and provides an impressive array of features and capabilities. When used together, the Hadoop Distributed File System (HDFS) and Spark can provide a truly scalable big data analytics setup. In this course, learn how to leverage these two technologies to build scalable and optimized data analytics pipelines. Instructor Kumaran Ponnambalam explores ways to optimize data modeling and storage on HDFS; discusses scalable data ingestion and extraction using Spark; and provides tips for optimizing data processing in Spark. Plus, he provides a use case project that allows you to practice your new techniques.
Topics include:
- Explain where and why Apache Spark stores its data.
- Differentiate between the types of data to work with.
- Explain how bucketing can be used to partition data.
- Analyze the execution plan when reading HDFS files with schema.
- Determine when and how to apply best practices for data processing.
- Leverage various tools and techniques to build a solution using Apache Spark and Hadoop.
Topics include:
- Explain where and why Apache Spark stores its data.
- Differentiate between the types of data to work with.
- Explain how bucketing can be used to partition data.
- Analyze the execution plan when reading HDFS files with schema.
- Determine when and how to apply best practices for data processing.
- Leverage various tools and techniques to build a solution using Apache Spark and Hadoop.
Skills covered
HadoopApache SparkApacheData EngineeringData AnalysisData ScienceBusiness Analysis and StrategyBusiness Software and ToolsDeep Dive (X:Y)
Concepts
Introduction
- The combined power of Spark and Hadoop Distributed File System (HDFS)
Introduction and Setup
- Apache Hadoop overview
- Apache Spark overview
- Integrating Hadoop and Spark
- Setting up the environment
- Using exercise files
HDFS Data Modeling for Analytics
- Storage formats
- Compression
- Partitioning
- Bucketing
- Best practices for data storage
Data Ingestion with Spark
- Reading external files into Spark
- Writing to HDFS
- Parallel writes with partitioning
- Parallel writes with bucketing
- Best practices for ingestion
Data Extraction with Spark
- How Spark works
- Reading HDFS files with schema
- Reading partitioned data
- Reading bucketed data
- Best practices for data extraction
Optimizing Spark Processing
- Pushing down projections
- Pushing down filters
- Managing partitions
- Managing shuffling
- Improving joins
- Storing intermediate results
- Best practices for data processing
Use Case Project
- Problem definition
- Data loading
- Total score analytics
- Average score analytics
- Top student analytics
Conclusion
- Next steps