LLM Evaluations and Grounding Techniques

Course details

Are you looking to learn more about large language models (LLMs)? Join instructor Denys Linkov as he explores hallucinations, their causes, the implications they have on the reliability and usability of LLMs, and how to mitigate structural and contextual inaccuracies to ensure high-quality, time-sensitive output. Develop practical techniques for addressing hallucinations, including few-shot learning, model fine-tuning, and templates for guiding LLM outputs. You'll also delve into more advanced topics like the chain of thought reasoning, retrieval-augmented generation, and model routing to enhance LLM performance. Test out your new skills along the way with real-world challenges that provide hands-on experience to solidify your learning. Whether you’re an AI researcher, a data scientist, or a tech enthusiast intrigued by the evolving capabilities of LLMs, this course offers valuable insights on navigating the complexities of AI with ease.

Concepts

0. Introduction

01 - Understanding grounding techniques for LLMs
02 - Setting up your LLM environment

1. Basic LLM Hallucinations

03 - What is a hallucination
04 - Hallucination examples
05 - Comparing hallucinations across LLMs
06 - Dangers of hallucinations
07 - Challenge - Finding a hallucination
08 - Solution - Finding a hallucination

2. Types of Hallucinations

09 - Training LLMs on time-sensitive data
10 - Poorly curated training data
11 - Faithfulness and context
12 - Ambiguous responses
13 - Incorrect output structure
14 - Declining to respond
15 - Fine-tuning hallucinations
16 - LLM sampling techniques and adjustments
17 - Bad citations
18 - Incomplete information extraction

3. Mitigating Hallucinations

19 - Few-shot learning
20 - Chain of thought reasoning
21 - Structured templates
22 - Retrieval-augmented generation
23 - Updating LLM model versions
24 - Model fine-tuning for mitigating hallucinations
25 - Orchestrating workflows through model routing
26 - Challenge - Automating ecommerce reviews with LLMs
27 - Solution - Automating ecommerce reviews with LLMs

4. Detecting Hallucinations

28 - Creating LLM evaluation pipelines
29 - LLM self-assessment pipelines
30 - Human-in-the-loop systems
31 - Specialized models for hallucination detection
32 - Building an evaluation dataset
33 - Optimizing prompts with DSPY
34 - Optimizing hallucination detections with DSPY
35 - Real-world LLM user testing
36 - Challenge - A more well-rounded AI trivia agent
37 - Solution - A more well-rounded AI trivia agent

5. Hallucination Paper Review

38 - Ragas - Evaluation paper
39 - Hallucinations in large multilingual translation models
40 - Do LLMs know what they don t know
41 - Set the Clock - LLM temporal fine-tuning
42 - Review of hallucination papers

Conclusion

43 - Continue your practice of grounding techniques for LLMs