AI Orchestration: Validation and User Feedback and Performance Metrics

Course details

Ready to get up to speed with the latest, most cutting-edge trends in AI orchestration? In this course, instructor Janani Ravi provides a comprehensive overview of how to effectively evaluate and integrate user feedback into AI models. Focusing on both traditional models and large language models (LLMs), Janani introduces essential concepts such as validation techniques, performance metrics, user feedback mechanisms, and more. Along the way, get hands-on experience with validating models through cross-validation and k-fold cross-validation, as well as practical applications of performance metrics like accuracy, precision, recall, using reinforcement learning for human feedback, and the BLEU, ROUGE, and METEOR scores for LLM evaluation.

Concepts

0. Introduction

01 - Prerequisites
02 - Quick overview of this course

1. Validating ML Models and LLMs

03 - Validation in the ML workflow
04 - Types of cross-validation
05 - Regular and k-fold cross-validation
06 - Stratified cross-validation and nested cross-validation
07 - K-fold cross-validation
08 - Validating LLMs
09 - Offline validation
10 - Golden datasets
11 - Benchmarking
12 - AI validating AI

2. Evaluating ML Models and LLMs

13 - Evaluating models using metrics
14 - Evaluating regression models
15 - Evaluating classification models
16 - Evaluating clustering models
17 - Accuracy precision recall
18 - Evaluating large language models (LLMs)
19 - Human evaluation
20 - Statistical methods for LLM evaluation
21 - ROUGE scores
22 - BLEU score
23 - METEOR score
24 - Perplexity
25 - Model-based methods for LLM evaluation
26 - Natural language inference
27 - BLEURT
28 - Judge models
29 - LLM evaluation

3. Collecting and Using User Feedback in LLMs

30 - User feedback in LLMs
31 - Implicit and explicit feedback
32 - Reinforcement learning
33 - Reinforcement learning from human feedback

Conclusion

34 - Summary and next steps