Курс на Stepik
Обложка курса «Reinforcement Learning for LLM» на Stepik
5 000 ₽

Reinforcement Learning for LLM 0.000

Открыть на
STEPIK.ORG

An engineering course on reinforcement learning for LLMs and agentic systems: from MDPs, bandits, REINFORCE/PPO, and offline RL to reward models, RLHF/DPO, RLVR/GRPO, test-time search, agentic RL, infrastructure, and evaluation. In addition to the open lectures, the paid Stepik course includes more than 500 tasks: auto-graded tests, exercises with solutions, self-check questions with answers, and notebooks. The course is part of an evolving series on ML and LLMs.

Показатель Текущие показатели Рост
Значение 🏆 Рейтинг 3 дн 7 дн 30 дн
Количество учеников на курсе «Reinforcement Learning for LLM»Учеников на курсе 0
Сертификаты, выданные на курсе «Reinforcement Learning for LLM»Сертификатов выдано 0
Отзывы о курсе «Reinforcement Learning for LLM»Отзывов получено 0
Рейтинг курса «Reinforcement Learning for LLM»Рейтинг курса 0.000
Уроки в курсе «Reinforcement Learning for LLM»Количество уроков 114
Тесты в курсе «Reinforcement Learning for LLM»Количество квизов 136
Задачи с кодом в курсе «Reinforcement Learning for LLM»Количество задач с кодом 7
Стоимость курса «Reinforcement Learning for LLM»Стоимость курса 5 000 ₽
Обновления курса «Reinforcement Learning for LLM»Обновления курса
Дата публикации курса «Reinforcement Learning for LLM»Дата публикации курса
Последнее обновление курса «Reinforcement Learning for LLM»Последнее обновление
Сложность normal

Содержание курса

Разделы в курсе «Reinforcement Learning for LLM» 13 разделов Уроки в курсе «Reinforcement Learning for LLM» 114 уроков Тесты в курсе «Reinforcement Learning for LLM» 136 тестов Задачи в курсе «Reinforcement Learning for LLM» 7 задач Последнее обновление курса «Reinforcement Learning for LLM» обн. 9 августа 2026

Introduction: Reinforcement Learning for LLMs

8 уроков
1. The Minimum Language of Reinforcement Learning
2. Choosing an RL Formulation for an LLM
3. A Modern Map of LLM Post-Training
4. The Practice Contract and First Numerical Experiments
5. Conceptual tasks
6. Analytical tasks
7. Programming tasks
8. Sources and further reading

State, Value, and Bellman Equations

8 уроков
1. From history to state, then to value
2. Bellman equations: evaluation and improvement
3. Model-free prediction: from full trajectories to bootstrapping
4. Control: behavior and target policies
5. Conceptual tasks
6. Analytical tasks
7. Programming tasks
8. Sources and further reading

Bandits, exploration, and one-step decisions

8 уроков
1. One-step decisions and the price of uncertainty
2. Exploration directed by uncertainty
3. Contextual bandits and logged feedback
4. Entropy, temperature, and multiple candidates
5. Conceptual tasks
6. Analytical tasks
7. Programming tasks
8. Sources and further reading

Policy gradients: from REINFORCE to PPO

9 уроков
1. Estimating a policy gradient and REINFORCE
2. Baselines and group estimators
3. Actor–critic methods and generalized advantage estimation
4. From an old policy to PPO
5. The bridge to LLM training
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

Fixed-data learning: imitation, deep Q-methods, and offline RL

9 уроков
1. Imitation learning and SFT
2. DQN: Q-learning with a neural network
3. Offline RL: what can be learned from a fixed log
4. Decision Transformer: a policy as a conditional trajectory model
5. A map of fixed-data LLM post-training
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

The LLM as a policy: tokens, log-probabilities, and KL

9 уроков
1. A token-level MDP without hidden assumptions
2. Logits, masks, and log-probabilities
3. KL to a reference policy
4. The actual behavior policy during generation
5. A rollout contract for RL with verifiable rewards
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

Reward models and preference data

9 уроков
1. Feedback is data, not ready-made truth
2. Training a scalar reward model
3. Evaluation: ordering, probability, transfer, and uncertainty
4. Proxy-reward overoptimization
5. Modern reward systems
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

RLHF and direct preference optimization

9 уроков
1. From preferences to a KL-regularized policy
2. DPO: derivation, loss, and gradient
3. Where the simple picture breaks
4. Variants of direct preference optimization
5. From equations to a reproducible run
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

RLVR and GRPO: verifiable rewards, group scores, and stability

9 уроков
1. RLVR as a feedback contract
2. Canonical GRPO
3. Normalization and algorithm variants
4. Training dynamics and diagnosis
5. From formulas to a reproducible run
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

Search, verification, and test-time compute

9 уроков
1. What scales at generation time
2. Repeated sampling and aggregation
3. Verifiers and Best-of-N
4. Process scores and structured search
5. Adaptive budgets and economics
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

RL for LLM agents: tools, environments, and long trajectories

9 уроков
1. From an answer to environment interaction
2. Trajectories, masks, and loss construction
3. Credit assignment over long horizons
4. Exploration and agent-policy optimization
5. Environments, safety, and reproducible practice
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

RL infrastructure for LLMs: memory, rollouts, and asynchrony

9 уроков
1. The RL loop as a dataflow
2. Memory and throughput
3. Pipelines, queues, and stragglers
4. Staleness, mismatch, and weight synchronization
5. Recovery, observability, and framework choice
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading

RL for LLM evaluation: statistics, reward hacking, and safety

9 уроков
1. Evaluation as a measurement system
2. pass@k, benchmark life cycles, and contamination
3. Reward hacking and judges under pressure
4. Safety and the protocol passport
5. A minimal harness and the course finale
6. Conceptual tasks
7. Analytical tasks
8. Programming tasks
9. Sources and further reading