Чему вы научитесь
- Formalize LLM training and inference problems as an MDP, a contextual bandit, or full-response selection by stating the state, action, reward, and horizon explicitly.
- Compute returns, V/Q/A functions, Bellman targets, Monte Carlo and TD estimates, and reason about the bias–variance trade-off.
- Apply REINFORCE, actor–critic, GAE, and PPO; analyze the policy loss, value loss, clipping, entropy bonus, and KL regularization.
- Distinguish on-policy, off-policy, imitation learning, and offline RL; check support/coverage assumptions and interpret off-policy evaluation.
- Build preference-data and reward-model pipelines; analyze annotation noise, calibration, LLM-judge biases, and reward hacking.
- Compare RLHF, DPO and related preference objectives, RLVR, and GRPO by their data source, feedback contract, and estimator.
- Design verifiers and group-relative signals while separating Best-of-N and test-time search from policy-parameter updates.
- Design agentic RL environments and rollout contracts covering tools, partial observability, stop reasons, masks, policy versions, and stale data.
- Organize asynchronous RL infrastructure and reproducible experiments with versioned checkpoints, data, rewards, configurations, and logs.
- Evaluate LLM/RL systems at matched budgets using pass@k, paired comparisons, confidence intervals, multiple seeds, and an explicit failure taxonomy.
- Run robustness and safety evaluations, detect proxy overoptimization, and write protocol passports that make results auditable.
- Translate formulas into notebook experiments, verify numerical invariants, and explain the limitations and failure modes of the results.
О курсе
Для кого этот курс
Начальные требования
Familiarity with introductory machine learning is expected: train/validation/test splits, loss functions, gradient optimization, overfitting, and the basic probabilistic interpretation of a model.
You should know Python and be able to read basic PyTorch code. Comfort with tensors, shapes, broadcasting, and simple training loops is sufficient.
Basic probability, linear algebra, and calculus are needed: distributions, conditional probability, expectation, vectors, matrices, derivatives, and gradients.
It is helpful to understand Transformers, tokenization, autoregressive generation, softmax, and log-probabilities. Modern LLMs offers additional depth but is not a prerequisite.
No previous reinforcement-learning course is required: Modules 1–5 introduce the necessary classical machinery. Readers who already know RL can use them as a fast refresher.
Jupyter or Google Colab is needed for the practical work. Core exercises run on CPU and in accessible environments, while a free Colab T4 covers short model runs. Paid APIs, powerful GPUs, and distributed training are used only in optional extensions.
Преподаватели курса
2
Как проходит обучение
The course is self-paced and primarily text-based, with equations, illustrations, tables, code, and links to primary sources.
The core route contains 13 modules in three parts: foundations and a fast introduction to classical RL; LLM post-training and RL; infrastructure, evaluation, and safety.
Most modules follow a recurring learning route:
• a precise problem statement and data contract;
• the mechanism, equations, and small numerical checks;
• the mapping to LLMs and modern implementations;
• auto-graded assessments after the main sections;
• analytical and computational exercises;
• detailed solutions, a notebook, and key papers for further study.
Assessments are graded automatically. Regular exercises are not an external examination: attempt a problem first, then open the detailed solution.
Practice is divided into resource tiers. Tier A runs on CPU and Stepik; Tier B uses a free Colab T4; optional B+ and C work may require a paid API, a powerful GPU, or a distributed run.
The notebooks run in Colab or locally and separate inexpensive smoke checks from real branches using open models. Extended experiments require explicitly recording model and dataset versions, the config, seed, and metrics.
A study load of roughly five to seven hours per week is recommended, but there is no fixed schedule. Readers with classical RL experience can move through the first five modules more quickly.
Что вы получите
- 13 modules and 114 lessons: 62 lecture lessons, 39 separate assignment lessons, and 13 primary-source lessons.
- 750 Stepik steps, including 414 assignments and self-checks; the RU and EN editions are synchronized, and lecture lessons contain clickable tables of contents.
- 151 self-checks embedded in the lectures, each with a prompt and an expandable worked solution.
- 109 conceptual tasks in native Stepik test formats, each worth 1 point and accompanied by an expandable explanation.
- 77 analytical and computational tasks with detailed solutions and explicit accepted tolerances wherever a numeric answer is required.
- 77 programming tasks for algorithm design and implementation, with expandable reference solutions and no mandatory code execution on the platform.
- 13 localized notebooks—one per module—attached to the relevant lectures and suitable for local Jupyter or Colab.
- Detailed derivations, negative cases, assumption checks, and failure-mode diagnostics instead of one-line answer comments.
- 13 separate Sources lessons with key papers, primary references, claim status, and limits on how far results can be transferred.
- A practical route from MDPs, bandits, REINFORCE, and PPO to reward models, RLHF/DPO, RLVR/GRPO, test-time search, agentic RL, infrastructure, evaluation, and safety.