MLOps Essentials: Monitoring Model Drift and Bias

Course details

As more and more ML models are developed and deployed, the need arises to ensure that the models are effective and safe and that they perform as desired. Model monitoring, a core function of MLOps, helps data scientists and MLOps engineers to meet this need. In this course, data analytics expert Kumaran Ponnambalam discusses the types of monitoring needed for ML models. He deep dives into model drift monitoring and bias. For model drift, Kumaran goes over the types of drift monitoring and their causes. He explains different techniques for drift monitoring and how to execute them in python using open source libraries. For bias, Kumaran highlights various sources of bias and their impact. He also analyzes bias in python with open source libraries. Finally, he recommends some best practices for drift and bias monitoring.

Concepts

0. Introduction

01 - The need for model monitoring
02 - Setting up the exercise files

1. Introduction to Model Monitoring

03 - ML models in production
04 - Challenges with serving models in production
05 - Metrics to monitor
06 - Data for model monitoring

2. Model Drift Basics

07 - Introduction to model drift
08 - Concept drift
09 - Feature drift
10 - What causes drift
11 - Drift remediation process

3. Detecting Model Drift

12 - Detecting concept drift
13 - Concept drift detection example
14 - Detecting feature drift
15 - Feature drift detection example
16 - Detecting drift in text and images
17 - Software for drift detection

4. Drift Monitoring Process and Best Practices

18 - Drift monitoring pipeline
19 - Analyzing drift trends
20 - Discovering root causes for drift
21 - Retraining to overcome drift

5. Introduction to Model Bias

22 - Fairness and bias
23 - Fairness in ML
24 - Sources of ML bias
25 - Protected attributes
26 - Demographic parity

6. Bias Detection and Best Practices

27 - Bias detection techniques
28 - Equal opportunity score
29 - EOS example
30 - Bias detection software
31 - Overcoming bias in ML

Conclusion

32 - Next steps