Data Pipeline Automation with GitHub Actions Using R and Python
2h 12mIntermediate2024-04-23
Authors
Rami Krispin
Course details
In this course, learn how to set up workflows on GitHub Actions to automate processes with both R and Python. Instructor Rami Krispin takes you through the automation process, sharing real-world examples. He shows you how to set up a data pipeline, pull metadata from a pipeline, and deploy a live dashboard with GitHub Actions and Pages. If you’re tired of spending hours running scripts manually, or slowing down your workflow by pulling data from APIs or updating dashboards, join Rami in this course to see how automation can speed up your work.
Skills covered
Version ControlStatisticsGitHubData EngineeringSoftware Development ToolsProgramming LanguagesData ScienceSoftware DevelopmentOne-Off
Concepts
Introduction
- Data pipeline automation with GitHub Actions
- What you should know
EIA API
- EIA API
- Setting an environment variable
- The EIA API dashboard
- GET request structure
- Querying the data via the browser
- Querying data with R and Python
- Pulling metadata from API with R
- Sending a simple GET request with R
- API limitations with R
- Handling a large data request with R
- Pulling metadata from API with Python
- Sending a simple GET request with Python
- API limitations with Python
- Handling a large data request with Python
- Challenge - Query the API
- Solution - Query the API with R
- Solution - Query the API with Python
Data Automation
- Data pipeline scope and requirements
- Data pipeline architecture
- Data refresh process
- ETL supporting functions
- Data backfilling
- Data refresh output
- Data quality checks
Deployment
- Introduction to GitHub Actions
- Deployment with Docker
- Setting GitHub Actions workflow
- Reviewing workflows logs
- Setting secrets and environment variables
- Advanced workflow
- Data pipeline deployment
Monitoring
- Data pipeline maintenance
- Deploying dashboard to GitHub Pages
Conclusion
- Next steps