Mastering Reasoning Models: Algorithms, Optimization, and Applications

Course details

This course provides a comprehensive exploration of modern reasoning models, focusing on the algorithmic innovations that power models like DeepSeek R1, OpenAI o1, and their open-source alternatives. Master the four key approaches to building reasoning LLMs: inference-time scaling, pure reinforcement learning, SFT+RL, and knowledge distillation. Through concrete examples and technical deep dives, learn how to implement test-time compute scaling, understand the mechanics of Group Relative Policy Optimization (GRPO), and build efficient inference pipelines for reasoning tasks. By the end of the course, you should have both the theoretical knowledge and practical skills to leverage these cutting-edge techniques in your own applications, whether you're working with enterprise-scale resources or more limited computational budgets.

Learning objectives
Distinguish between different approaches to building reasoning LLMs and their respective tradeoffs.
Implement and optimize test-time compute scaling techniques including majority voting, Best-of-N, and beam search.
Understand the principles behind Group Relative Policy Optimization (GRPO) and how it differs from standard RLHF approaches.
Apply knowledge from various reasoning model architectures to make informed implementation decisions.
Select the appropriate reasoning technique based on computational constraints and application requirements.

Concepts

Introduction

Why do you need reasoning models

The Power of Reasoning Models

The Shift to Reasoning Models
The reasoning landscape

The Four Approaches to Building Reasoning Models

Inference-time scaling
Pure reinforcement learning (RL)
Supervised fine-tuning (SFT) and RL
Distillation and pure SFT

Test-Time Compute Scaling Deep Dive

Majority voting and self-consistency
Best-of-n and weighted aggregation
Beam search with process reward models
Diverse Verifier Tree Search (DVTS)

Reinforcement Learning for Reasoning

Beyond RLHF - Group Relative Policy Optimization (GRPO)
Reward functions for reasoning
The aha moment - Self-verification through RL

Building Efficient Reasoning Systems

Compute-optimal scaling in production
Budget-friendly reasoning models
Balancing cost and performance

Conclusion

Future directions in reasoning LLMs