Курс на Stepik
Обложка курса «Reinforcement Learning for LLM» на Stepik
Бесплатно

Reinforcement Learning for LLM ★ 5.000

Открыть на
STEPIK.ORG

An engineering course on reinforcement learning for LLMs and agentic systems: from MDPs, bandits, REINFORCE/PPO, and offline RL to reward models, RLHF/DPO, RLVR/GRPO, test-time search, agentic RL, infrastructure, and evaluation. In addition to the open lectures, the Stepik course includes more than 500 tasks: auto-graded tests, exercises with solutions, self-check questions with answers, and notebooks. The course is part of an evolving series on ML and LLMs.

Показатель Текущие показатели Рост
Значение 🏆 Рейтинг 3 дн 7 дн 30 дн
Количество учеников на курсе «Reinforcement Learning for LLM»Учеников на курсе 30
Сертификаты, выданные на курсе «Reinforcement Learning for LLM»Сертификатов выдано 0
Отзывы о курсе «Reinforcement Learning for LLM»Отзывов получено 1
Рейтинг курса «Reinforcement Learning for LLM»Рейтинг курса 5.000
Уроки в курсе «Reinforcement Learning for LLM»Количество уроков 114
Тесты в курсе «Reinforcement Learning for LLM»Количество квизов 136
Время прохождения курса «Reinforcement Learning for LLM»Время прохождения курса —
Обновления курса «Reinforcement Learning for LLM»Обновления курса — —
Дата публикации курса «Reinforcement Learning for LLM»Дата публикации курса — ———
Последнее обновление курса «Reinforcement Learning for LLM»Последнее обновление — ———
Сложность normal — ———

Содержание курса

Разделы в курсе «Reinforcement Learning for LLM» 13 разделов Уроки в курсе «Reinforcement Learning for LLM» 114 уроков Тесты в курсе «Reinforcement Learning for LLM» 136 тестов Время прохождения курса «Reinforcement Learning for LLM» 0 ч. Последнее обновление курса «Reinforcement Learning for LLM» обн. 15 августа 2026

Introduction: Reinforcement Learning for LLMs

8 уроков
1. The Minimum Language of Reinforcement Learning ↗
2. Choosing an RL Formulation for an LLM ↗
3. A Modern Map of LLM Post-Training ↗
4. The Practice Contract and First Numerical Experiments ↗
5. Conceptual tasks ↗
6. Analytical tasks ↗
7. Programming tasks ↗
8. Sources and further reading ↗

State, Value, and Bellman Equations

8 уроков
1. From history to state, then to value ↗
2. Bellman equations: evaluation and improvement ↗
3. Model-free prediction: from full trajectories to bootstrapping ↗
4. Control: behavior and target policies ↗
5. Conceptual tasks ↗
6. Analytical tasks ↗
7. Programming tasks ↗
8. Sources and further reading ↗

Bandits, exploration, and one-step decisions

8 уроков
1. One-step decisions and the price of uncertainty ↗
2. Exploration directed by uncertainty ↗
3. Contextual bandits and logged feedback ↗
4. Entropy, temperature, and multiple candidates ↗
5. Conceptual tasks ↗
6. Analytical tasks ↗
7. Programming tasks ↗
8. Sources and further reading ↗

Policy gradients: from REINFORCE to PPO

9 уроков
1. Estimating a policy gradient and REINFORCE ↗
2. Baselines and group estimators ↗
3. Actor–critic methods and generalized advantage estimation ↗
4. From an old policy to PPO ↗
5. The bridge to LLM training ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

Fixed-data learning: imitation, deep Q-methods, and offline RL

9 уроков
1. Imitation learning and SFT ↗
2. DQN: Q-learning with a neural network ↗
3. Offline RL: what can be learned from a fixed log ↗
4. Decision Transformer: a policy as a conditional trajectory model ↗
5. A map of fixed-data LLM post-training ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

The LLM as a policy: tokens, log-probabilities, and KL

9 уроков
1. A token-level MDP without hidden assumptions ↗
2. Logits, masks, and log-probabilities ↗
3. KL to a reference policy ↗
4. The actual behavior policy during generation ↗
5. A rollout contract for RL with verifiable rewards ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

Reward models and preference data

9 уроков
1. Feedback is data, not ready-made truth ↗
2. Training a scalar reward model ↗
3. Evaluation: ordering, probability, transfer, and uncertainty ↗
4. Proxy-reward overoptimization ↗
5. Modern reward systems ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

RLHF and direct preference optimization

9 уроков
1. From preferences to a KL-regularized policy ↗
2. DPO: derivation, loss, and gradient ↗
3. Where the simple picture breaks ↗
4. Variants of direct preference optimization ↗
5. From equations to a reproducible run ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

RLVR and GRPO: verifiable rewards, group scores, and stability

9 уроков
1. RLVR as a feedback contract ↗
2. Canonical GRPO ↗
3. Normalization and algorithm variants ↗
4. Training dynamics and diagnosis ↗
5. From formulas to a reproducible run ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

Search, verification, and test-time compute

9 уроков
1. What scales at generation time ↗
2. Repeated sampling and aggregation ↗
3. Verifiers and Best-of-N ↗
4. Process scores and structured search ↗
5. Adaptive budgets and economics ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

RL for LLM agents: tools, environments, and long trajectories

9 уроков
1. From an answer to environment interaction ↗
2. Trajectories, masks, and loss construction ↗
3. Credit assignment over long horizons ↗
4. Exploration and agent-policy optimization ↗
5. Environments, safety, and reproducible practice ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

RL infrastructure for LLMs: memory, rollouts, and asynchrony

9 уроков
1. The RL loop as a dataflow ↗
2. Memory and throughput ↗
3. Pipelines, queues, and stragglers ↗
4. Staleness, mismatch, and weight synchronization ↗
5. Recovery, observability, and framework choice ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗

RL for LLM evaluation: statistics, reward hacking, and safety

9 уроков
1. Evaluation as a measurement system ↗
2. pass@k, benchmark life cycles, and contamination ↗
3. Reward hacking and judges under pressure ↗
4. Safety and the protocol passport ↗
5. A minimal harness and the course finale ↗
6. Conceptual tasks ↗
7. Analytical tasks ↗
8. Programming tasks ↗
9. Sources and further reading ↗