Azure Data Engineer Associate (DP-203) Cert Prep: 4 Monitor and Optimize Data Storage and Data Processing
34mIntermediate2022-12-15
Authors
Noah Gift
MLOps Expert | Solopreneur | Author | Adjunct Professor | CTO
Course details
Are you preparing for the Microsoft Azure Data Engineering (DP-203) exam, or seeking a better understanding of how to design and develop data processing? This course, the fourth in a series, can help you. Noah Gift, founder of Pragmatic A.I. Labs and a Python Software Foundation Fellow, covers how to monitor and optimize data storage and data processing. Noah first covers how to monitor data storage and data processing, showing how to: implement logging used by Azure Monitor; configure monitoring services; measure query performance; and more. He then details topics pertaining to optimizing and troubleshooting data storage and processing, including how to: rewrite user-defined functions; tune queries by using indexers and cache; handle skew in data and data spill; troubleshoot a failed Spark job and pipeline run.
Skills covered
Cloud StorageData EngineeringAzureNetwork AdministrationCloud PlatformsCert PrepNetwork and System AdministrationCloud ComputingData ScienceMicrosoft
Concepts
Introduction
- Course introduction
Monitor Data Storage and Data Processing
- Implement logging used by Azure Monitor
- Configure monitoring services
- Measure performance of data movement
- Monitor data system pipeline cluster performance
- Measure query performance
- Schedule and monitor pipeline tests
- Interpret a Spark directed acyclic graph (DAG)
Optimize and Troubleshoot Data Storage and Data Processing
- Rewrite user-defined functions (UDFs)
- Handle skew in data and data spill
- Tune shuffle partitions pipelines
- Optimize resource management
- Tune queries by using indexers and cache
- Troubleshoot a failed Spark job and pipeline run
Conclusion
- Summary and next steps