Learning Google Dataflow

Course details

Would you like to find out what you can do with Google Dataflow? In this course, Kishan Iyer gives you an overview of Google Dataflow, walks you through the Dataflow service, explores ways to configure and monitor pipelines, and shows you how to use Dataflow templates. Kishan explains how to define your options for an Apache Beam pipeline, then shows you how to run an Apache Beam application, define a pipeline transformation step, and build an Apache Beam pipeline. He steps through how to set up the Dataflow service and get it to run an Apache Beam pipeline. Kishan covers computing an average value from data, configuring the number of workers, performing a global computation, and more. Plus, he discusses how to put together the pieces for a Dataflow template, define and run a job using a pre-built Dataflow template, and run your Apache Beam pipelines using Dataflow Prime.

Concepts

0. Introduction

01 - An overview of Dataflow

1. A Quick Overview of Dataflow

02 - Setting up a Google Cloud project for Dataflow
03 - Creating credentials for a data processing pipeline
04 - Understanding Apache Beam
05 - The role of Dataflow
06 - Creating an Apache Beam project with Maven
07 - Defining pipeline options
08 - Running an Apache Beam application
09 - Defining a pipeline transformation step
10 - Building and executing an Apache Beam pipeline

2. The Dataflow Service

11 - Setting up a cloud storage bucket
12 - Working with files in Apache Beam
13 - Running the file processing job
14 - Executing a pipeline with Dataflow
15 - Viewing the files generated by a Dataflow job
16 - Aggregating values in a pipeline
17 - Running and verifying an aggregation job

3. Configuring and Monitoring Pipelines

18 - Computing an average value from data
19 - Configuring the number of workers
20 - Performing a global computation
21 - Feeding a side input to a pipeline step
22 - Verifying the side input pipeline run

4. Using Dataflow Templates

23 - Putting together the pieces for a Dataflow template
24 - Defining and running a job with a Dataflow template

5. Using Dataflow Prime

25 - Working with Dataflow Prime

Conclusion

26 - Summary and next steps