Big Data Analytics with Hadoop and Apache Spark

Course details

Apache Hadoop was a pioneer in the world of big data technologies, and it continues to lead in enterprise big data storage. Apache Spark is the top big data processing engine and provides an impressive array of features and capabilities. When used together, the Hadoop Distributed File System (HDFS) and Spark can provide a truly scalable setup for big data analytics. In this course, data analytics expert Kumaran Ponnambalam shows you how to leverage these two technologies to build scalable and optimized data analytics pipelines. Explore ways to optimize data modeling and storage on HDFS; discuss scalable data ingestion and extraction using Spark; and review actionable tips for optimizing data processing in Spark. Plus, complete a use case project that allows you to practice your new techniques.

Learning objectives
Explain where and why Apache Spark stores its data.
Differentiate between the types of data to work with.
Explain how bucketing can be used to partition data.
Analyze the execution plan when reading HDFS files with schema.
Determine when and how to apply best practices for data processing.
Leverage various tools and techniques to build a solution using Apache Spark and Hadoop.

Concepts

0. Introduction

01 - The combined power of Spark and Hadoop Distributed File System (HDFS)

1. Introduction and Setup

02 - Apache Hadoop overview
03 - Apache Spark overview
04 - Integrating Spark and Hadoop
05 - Using exercise files

2. HDFS Data Modeling for Analytics

06 - Storage formats
07 - Compression
08 - Partitioning
09 - Bucketing
10 - Best practices for data storage

3. Data Ingestion with Spark

11 - Reading external files into Spark
12 - Writing to HDFS
13 - Parallel writes with partitioning
14 - Parallel writes with bucketing
15 - Best practices for ingestion

4. Data Extraction with Spark

16 - How Spark works
17 - Reading HDFS files with schema
18 - Reading partitioned data
19 - Reading bucketed data
20 - Best practices for data extraction

5. Optimizing Spark Processing

21 - Pushing down projections
22 - Pushing down filters
23 - Managing partitions
24 - Improving joins
25 - Storing intermediate results
26 - Best practices for data processing

6. Use Case Project

27 - Problem definition
28 - Data loading
29 - Total score analytics
30 - Average score analytics
31 - Top student analytics

Conclusion

32 - Continuing on with big data analytics