AI Observability: Monitoring and Debugging LLMs in Production

Course details

Most teams don't realize their LLM is misbehaving until a user complains. This course helps you get ahead of that, covering how LLMs (large language models) fail, why the failures are difficult to catch, and what you can do about it. Review the failure modes that show up most in production: hallucinations, prompt injection, drift, and toxic outputs, then move into the practical side of observability—what to measure, what to watch, and how to debug when things go sideways. Get hands-on with Langfuse and Arize Phoenix, two free tools that give you real visibility into what your LLM is actually doing. Leave with a monitoring strategy and debugging approach you can put to work straight away — usable across a variety of roles and backgrounds.

Learning objectives
Identify the most common failure modes in LLM-based AI systems.
Describe constraints of traditional software monitoring and AI observability.
Implement observability practices using free open-source tools.
Integrate observability with different architectures.
Design a simple monitoring strategy for an LLM application.

Concepts

Introduction

Improve LLM experiences through monitoring and debugging

LLM Failure Clinic - Spotting Hallucinations, Injection, and Drift

Reproduce hallucinations and understand root cause
Run a prompt-injection red team mini-lab
Detect drift and toxicity over time

From Metrics to Meaning - What to Measure in LLM Systems

Compare a traditional dashboard to an LLM dashboard
Translate traces, metrics, and evaluations into implementation

Instrument an LLM App with Langfuse

Add SDK instrumentation to a single LLM call
Trace a multi-step chain
Logging user feedback and scores
Use the dashboard to answer what happened

Run LLM Evals Locally with Arize Phoenix

Send phoenix traces from a notebook
Run built-in evals for hallucination and relevance
Slice and query traces to find failure clusters

Build Your LLM Ops Playbook - Metrics, Alerts, and Incident Response

Choose metrics and alerts for your LLM monitoring stack
Create a debugging playbook
Select tools from LangSmith, Weights & Biases, and Helicone