Содержание курса
Introduction: Reinforcement Learning for LLMs
8 уроков
1.
The Minimum Language of Reinforcement Learning
↗
2.
Choosing an RL Formulation for an LLM
↗
3.
A Modern Map of LLM Post-Training
↗
4.
The Practice Contract and First Numerical Experiments
↗
5.
Conceptual tasks
↗
6.
Analytical tasks
↗
7.
Programming tasks
↗
8.
Sources and further reading
↗
State, Value, and Bellman Equations
8 уроков
1.
From history to state, then to value
↗
2.
Bellman equations: evaluation and improvement
↗
3.
Model-free prediction: from full trajectories to bootstrapping
↗
4.
Control: behavior and target policies
↗
5.
Conceptual tasks
↗
6.
Analytical tasks
↗
7.
Programming tasks
↗
8.
Sources and further reading
↗
Bandits, exploration, and one-step decisions
8 уроков
1.
One-step decisions and the price of uncertainty
↗
2.
Exploration directed by uncertainty
↗
3.
Contextual bandits and logged feedback
↗
4.
Entropy, temperature, and multiple candidates
↗
5.
Conceptual tasks
↗
6.
Analytical tasks
↗
7.
Programming tasks
↗
8.
Sources and further reading
↗
Policy gradients: from REINFORCE to PPO
9 уроков
1.
Estimating a policy gradient and REINFORCE
↗
2.
Baselines and group estimators
↗
3.
Actor–critic methods and generalized advantage estimation
↗
4.
From an old policy to PPO
↗
5.
The bridge to LLM training
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
Fixed-data learning: imitation, deep Q-methods, and offline RL
9 уроков
1.
Imitation learning and SFT
↗
2.
DQN: Q-learning with a neural network
↗
3.
Offline RL: what can be learned from a fixed log
↗
4.
Decision Transformer: a policy as a conditional trajectory model
↗
5.
A map of fixed-data LLM post-training
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
The LLM as a policy: tokens, log-probabilities, and KL
9 уроков
1.
A token-level MDP without hidden assumptions
↗
2.
Logits, masks, and log-probabilities
↗
3.
KL to a reference policy
↗
4.
The actual behavior policy during generation
↗
5.
A rollout contract for RL with verifiable rewards
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
Reward models and preference data
9 уроков
1.
Feedback is data, not ready-made truth
↗
2.
Training a scalar reward model
↗
3.
Evaluation: ordering, probability, transfer, and uncertainty
↗
4.
Proxy-reward overoptimization
↗
5.
Modern reward systems
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
RLHF and direct preference optimization
9 уроков
1.
From preferences to a KL-regularized policy
↗
2.
DPO: derivation, loss, and gradient
↗
3.
Where the simple picture breaks
↗
4.
Variants of direct preference optimization
↗
5.
From equations to a reproducible run
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
RLVR and GRPO: verifiable rewards, group scores, and stability
9 уроков
1.
RLVR as a feedback contract
↗
2.
Canonical GRPO
↗
3.
Normalization and algorithm variants
↗
4.
Training dynamics and diagnosis
↗
5.
From formulas to a reproducible run
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
Search, verification, and test-time compute
9 уроков
1.
What scales at generation time
↗
2.
Repeated sampling and aggregation
↗
3.
Verifiers and Best-of-N
↗
4.
Process scores and structured search
↗
5.
Adaptive budgets and economics
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
RL for LLM agents: tools, environments, and long trajectories
9 уроков
1.
From an answer to environment interaction
↗
2.
Trajectories, masks, and loss construction
↗
3.
Credit assignment over long horizons
↗
4.
Exploration and agent-policy optimization
↗
5.
Environments, safety, and reproducible practice
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
RL infrastructure for LLMs: memory, rollouts, and asynchrony
9 уроков
1.
The RL loop as a dataflow
↗
2.
Memory and throughput
↗
3.
Pipelines, queues, and stragglers
↗
4.
Staleness, mismatch, and weight synchronization
↗
5.
Recovery, observability, and framework choice
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗
RL for LLM evaluation: statistics, reward hacking, and safety
9 уроков
1.
Evaluation as a measurement system
↗
2.
pass@k, benchmark life cycles, and contamination
↗
3.
Reward hacking and judges under pressure
↗
4.
Safety and the protocol passport
↗
5.
A minimal harness and the course finale
↗
6.
Conceptual tasks
↗
7.
Analytical tasks
↗
8.
Programming tasks
↗
9.
Sources and further reading
↗