برای استفاده کامل از سایت، جاوااسکریپت را فعال کنید.

JavaScript is required for the full experience. You can still browse these pages:

Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS

50mIntermediate2024-02-09

Authors

Sagar Suri

Course details

As data engineers, it’s difficult to ignore the ever-growing importance and demand for streamed data. In this course, designed uniquely for experienced data professionals looking to modernize their technical skill set, join instructor and data engineer Sagar Suri as he demystifies the process of building streaming ingestion pipelines for the industry-leading lakehouse solution, Snowflake, using popular tools such as Kafka and AWS. Get an overview of data streaming, AWS, and Snowflake to prepare to start building a streaming pipeline on your own. Along the way, learn how to install and set up Snowpipe and Kafka to gather and send data directly into Snowflake.

Skills covered

SnowflakeCloud StorageDatabase DevelopmentAmazon Web Services (AWS)AmazonDatabase ManagementCloud ServicesCloud PlatformsCloud ComputingSoftware DevelopmentOne-Off

Concepts

0. Introduction

01 - Building streaming data pipelines
02 - What you should know
03 - Prerequisites

1. Introduction to Data Streaming, AWS, and Snowflake

04 - Why AWS and Snowflake
05 - Introduction to streaming ingestion pipelines
06 - Streaming data pipelines - A deeper dive
07 - Overview of streaming pipeline project

2. Building a Streaming Pipeline

08 - Setting up your MSK cluster and EC2 instance
09 - Setting up your keys
10 - What is Snowpipe
11 - Installing Snowpipe Kafka connector
12 - Set up config for provider-MSK connection and create topic
13 - Preparing your Snowflake environment
14 - Setting up your Snowpipe Kafka connect config
15 - Sending data to Snowflake
16 - Final considerations

Conclusion

17 - Course review and next steps