Modern LLMs ★ 0.000

Engineering course on modern LLMs—from tokenization, RoPE, attention, MoE, and Transformer alternatives to pretraining, KV caches, quantization, SFT, RLHF/DPO/GRPO, reasoning, RAG, agents, and evaluation. Beyond open lectures, the paid Stepik course has 500+ learning activities in total: auto-graded tests, exercises with solutions, and self-check questions with answers; tested notebooks; and a custom MiniLLM—dense S/M/L and their MoE twins—from data to ablations. Part of an evolving series on ML and LLMs.

ML, AI, and LLM engineers who need a coherent map of modern architectures, training, inference, and evaluation rather than a collection of isolated recipes. Practitioners working on NLP, retrieval/RAG, agents, or multimodal systems who want to understand the contracts between the model and the application layer. Researchers, senior undergraduates, graduate students, and independent learners looking for a bridge from papers to reproducible experiments. Developers who already use LLM APIs or open-weight models and want to move from parameter tuning to understanding mechanisms and engineering trade-offs. The course is not a first introduction to programming or machine learning, and it is not a quick-start tutorial for a single framework or API.

Familiarity with introductory machine learning is expected: train/validation/test splits, loss functions, gradient descent, overfitting, and the basic probabilistic interpretation of a model.

You should know Python and be able to read PyTorch code. Comfort with tensors, shapes, broadcasting, and simple training loops is sufficient.

Basic linear algebra and calculus are needed: vectors, matrices, inner products, derivatives, gradients, and softmax. Probability is used at the level of distributions, conditional probability, and expectation.

Prior familiarity with the Transformer is helpful but not mandatory. Reference Module 18 provides a detailed route through the basic encoder–decoder architecture, attention, and the main equations; it can be completed before Module 1.

Jupyter or Google Colab is needed for the practical work. Core demonstrations and smoke routes are designed for accessible environments; a GPU is useful for extended runs but is not required for the theory or most controlled experiments.

Experience with CUDA, distributed training, production serving, or reinforcement learning is not required. The course marks the boundaries of these specialized disciplines explicitly.

The course is self-paced and primarily text-based, with equations, illustrations, tables, and links to primary sources.

The core route contains 17 modules. Optional branch 7b covers diffusion LLMs, Module 18 is a reference on the basic Transformer, and the final module contains the Capstone.

Most modules follow a recurring learning route:

• an engineering question and a precise problem statement;
• the mechanism, equations, and numerical checks;
• modern implementations and limits of applicability;
• auto-graded assessments after the main sections;
• analytical, computational, and project exercises;
• detailed solutions, a notebook, and key papers for further study.
Assessments are graded automatically. Regular exercises are not an external examination: attempt a problem first, then open the detailed solution. Project exercises form a separate practical route, with demonstration notebooks available for selected projects.

The English and Russian courses use separate notebooks. They run in Colab or locally and focus on testable mechanisms and small experiments; extended profiles may require a GPU.

The Capstone combines pretraining, post-training, inference, and ablations in a MiniLLM project. It is not reviewed externally: you retain the config, seed, metrics, final weights or a checkpoint, write a report, and use the self-assessment table.

A study load of roughly five to seven hours per week is recommended, but there is no fixed schedule. The reference module and optional advanced branches can be read selectively.