Data Science on Google Cloud Platform: Building Data Pipelines
1h 8mIntermediate2018-08-22
Authors
Kumaran Ponnambalam
Working with data for 20+ years
Course details
Cloud computing brings unlimited scalability and elasticity to data science applications. Expertise in the major platforms, such as Google Cloud Platform (GCP), is essential to the IT professional. This course—one of a series by veteran cloud engineering specialist and data scientists Kumaran Ponnambalam—shows how to use the latest technologies in GCP to build a big data pipeline that ingests, transports, and transforms data entirely in the cloud. Learn how to set up data processing jobs using Apache Beam and Cloud Dataflow. Discover how to leverage Cloud Pub/Sub for stream ingestion and real-time messaging. Finally, find out how to process the stream events in Cloud Dataflow. The course uses an end-to-end use case that shows how to apply the knowledge and best practices from the course in a practical data science workflow.
Learning objectives
GCP products for data pipelines
Setting up a pipeline with Apache Beam and Cloud Dataflow
Processing data with Beam and Dataflow
Ingesting streams with Cloud Pub/Sub
Performing stream analysis with Dataflow
Learning objectives
GCP products for data pipelines
Setting up a pipeline with Apache Beam and Cloud Dataflow
Processing data with Beam and Dataflow
Ingesting streams with Cloud Pub/Sub
Performing stream analysis with Dataflow
Skills covered
Google CloudData Science FoundationsSoftware Development ToolsGoogleCloud PlatformsCloud ComputingData ScienceSoftware DevelopmentOne-Off
Concepts
0. Introduction
- 01 - What goes into a data pipeline
- 02 - Data science modules covered
1. GCP Data Pipeline Products
- 03 - GCP data pipeline options
- 04 - Cloud Dataproc
- 05 - Cloud Dataflow
- 06 - Cloud Pub Sub
2. Apache Beam
- 07 - What is Apache Beam
- 08 - Beam pipelines
- 09 - PCollections
- 10 - Transforms
- 11 - Pipeline I O
- 12 - Runners
3. Setting Up Dataflow
- 13 - Setting up GCP for Dataflow
- 14 - Setting up Python
- 15 - Creating a simple pipeline
- 16 - Executing in Dataflow
4. Data Processing with Beam and Dataflow
- 17 - Reading text files
- 18 - ParDo
- 19 - GroupBy
- 20 - Map
- 21 - Combine
- 22 - Writing data to text files
- 23 - Other capabilities
5. Cloud Pub Sub
- 24 - What is Pub Sub
- 25 - Topics and messages
- 26 - Publishers
- 27 - Subscribers
- 28 - Create a topic
- 29 - Create a subscription
- 30 - Publish and receive
- 31 - Python SDK
6. Streaming with Dataflow
- 32 - Streaming with Dataflow
- 33 - Windowing with Dataflow
- 34 - Streaming and windowing example
Conclusion
- 35 - Next steps