Reinforcement Learning for LLM Alignment and Reasoning by Pearson

Course details

Pretraining gives LLMs capability, not judgment. In this course, learn how reinforcement learning techniques like direct preference optimization (DPO) and group relative policy optimization (GRPO) shape model behavior, safety, and reasoning, and how to build the evaluation and governance systems that keep alignment on track. This course is an ideal fit for developers, data scientists, and ML engineers who are fine-tuning or deploying LLMs and want to improve their safety, effectiveness, and reasoning capabilities.

Learning objectives
Distinguish between instructional, behavioral, style, and value alignment.
Build and evaluate reward models using pairwise comparisons and the Bradley-Terry model.
Apply DPO, GRPO, and other modern post-training techniques.
Design evaluation harnesses to monitor safety, toxicity, and model drift.
Implement a constitutional AI pipeline with self-critique and revision.

Concepts

Introduction

Course introduction

Foundations of Alignment

Key topics
Alignment goals and failure modes
Framing the alignment problem for LLMs
Post-training overview - SFT, preference optimization, and RLHF

Reinforcement Learning Basics for LLMs

Key topics
Policies, rewards, and exploration
Common algorithms used in post-training
PPO in action using Flan-T5

Building a Reward Model

Topics
Foundations of reward modeling
Training a reward model

Modern Post-Training

Topics
Why RLHF evolved
Direct preference optimization (DPO)
REINFORCE leave-one-out (RLOO)

Advanced Post-Training for Reasoning

Key topics
Why reasoning needs a different post-training method
Group relative policy optimization (GRPO)
Case study - Inducing reasoning in LLMs with GRPO

Future Directions, Safety, and Governance

Key topics
Practical safety and robustness benchmarks
Building a simple safety evaluation harness
Governance and risk frameworks
Joint safety evaluations and red teaming
Case study - Constitutional AI plus safety evaluation

Conclusion

Course summary and next steps