Содержание курса
19
разделов
368
уроков
299
тестов
обн. 19 августа 2026
The Modern LLM Landscape
23 урока
1.
Why an engineer needs a map of the landscape
↗
2.
How the course reads code, and why a few systems recur
↗
3.
Nine years in fifteen minutes: 2017 → 2026
↗
4.
A bridge to the classics: taxonomy instead of a zoo
↗
5.
Two passports: the model and the system around it
↗
6.
Axis I up close: dense and MoE
↗
7.
Worked comparison: gpt-oss-120b and Llama 4 Scout
↗
8.
Axis II: context means length, quality, and cost
↗
9.
Axis III: modalities are directional
↗
10.
Axis IV: who controls the reasoning budget?
↗
11.
Axis V: access, licensing, and practical openness
↗
12.
Axis VI: the price of a token is not the price of a system
↗
13.
Dated reference: closed and API families
↗
14.
Dated reference: open weights and reproducible artefacts
↗
15.
From public signal to engineering evidence
↗
16.
openai/gpt-oss as the course’s table of contents
↗
17.
Code, level 1: a passport from a validated config.json
↗
18.
Level 2: what the notebook adds
↗
19.
The course map and the growing series
↗
20.
Self-check questions
↗
21.
Exercises with solutions
↗
22.
Summary cheat sheet
↗
23.
Notebook and sources
↗
Tokenization and Embeddings
17 уроков
1.
Tokenization is an architectural decision
↗
2.
From word vocabularies to byte-level subwords
↗
3.
Compression is the right analogy—with limits
↗
4.
The tokenizer as a deterministic program
↗
5.
Worked example: from UTF-8 to a compression curve
↗
6.
Level-1 code: a transparent BBPE trainer
↗
7.
What the notebook adds
↗
8.
Reading o200k_harmony correctly
↗
9.
A documented vocabulary landscape
↗
10.
Fertility, language, and domain
↗
11.
Embedding cost and weight tying
↗
12.
Where the token analogy extends to other modalities
↗
13.
Connections to the rest of the course
↗
14.
Self-check questions
↗
15.
Exercises
↗
16.
Key takeaways
↗
17.
Notebook and sources
↗
Positional Representations: From Sinusoids to Multidimensional…
18 уроков
1.
Motivation: content attention is not an ordering mechanism
↗
2.
The historical arc: one problem, several architectural answers
↗
3.
Fourier intuition, complex rotations, and what is exact
↗
4.
Formalism: the frequency ladder and the rotary identity
↗
5.
Worked example: three placements, one relative displacement
↗
6.
Reading the RoPE spectrum without inventing a hard horizon
↗
7.
Context extension changes a spectrum, not just one number
↗
8.
ALiBi: a distance prior in the logits
↗
9.
NoPE and hybrid position mechanisms
↗
10.
Reading gpt-oss positional fields as evidence—not biography
↗
11.
Level-one code: explicit shapes and explicit scaling placement
↗
12.
What the notebook validates
↗
13.
M-RoPE and time-aligned multimodal position
↗
14.
The 2026 landscape is a branching architecture map
↗
15.
Self-check questions
↗
16.
Exercises with solutions
↗
17.
Key takeaways
↗
18.
Notebook and primary sources
↗
Modern Attention Mechanisms
20 уроков
1.
Motivation: two bills hidden behind one decode step
↗
2.
The historical split: compress the cache or reorganize the…
↗
3.
Three useful classical views—and where they stop
↗
4.
SDPA, masks, and the square-root scale
↗
5.
Worked storage arithmetic: four head layouts and one hybrid…
↗
6.
GQA is a grouping invariant, not a quality guarantee
↗
7.
MLA stores a joint latent and absorbs linear maps
↗
8.
FlashAttention: dense attention without the quadratic HBM object
↗
9.
Windowed attention and layer schedules
↗
10.
“Attention sink” refers to two different objects
↗
11.
Sparse attention selects positions rather than shrinking each…
↗
12.
Level-one code: transparent SDPA, masks, and GQA
↗
13.
What the notebook verifies
↗
14.
Reading the gpt-oss attention implementation
↗
15.
A contemporary attention landscape with evidence labels
↗
16.
Multimodal attention: where the visual state is paid for
↗
17.
Self-check questions
↗
18.
Exercises
↗
19.
Key takeaways
↗
20.
Notebook and primary sources
↗
Normalization, activations, and the anatomy of a Transformer…
21 урок
1.
Why the quiet fields in a configuration matter
↗
2.
The historical problem was not “which norm wins?”
↗
3.
Three useful classical analogies
↗
4.
The residual stream and what the identity path actually…
↗
5.
LayerNorm and RMSNorm side by side
↗
6.
Four norm placements, four computation graphs
↗
7.
Query/key stabilization is a family, not one universal recipe
↗
8.
From a two-matrix FFN to gated multiplication
↗
9.
SwiGLU budgets and the gpt-oss variant
↗
10.
Reconstructing Llama-3.1-8B exactly
↗
11.
DyT and DyISRU: removing the feature reduction
↗
12.
mHC constrains residual mixing, not hidden-state trajectories
↗
13.
Level-1 code: the pieces we can inspect directly
↗
14.
What the notebook adds
↗
15.
Reading gpt-oss from code rather than from a guessed config…
↗
16.
A current block table with an evidence column
↗
17.
The same accounting applies to vision blocks
↗
18.
Self-check questions
↗
19.
Exercises with solutions
↗
20.
Key takeaways
↗
21.
Notebook and primary sources
↗
Mixture of Experts
20 уроков
1.
The bargain: capacity without activating the whole bank
↗
2.
From local specialists to trillion-parameter sparse models
↗
3.
Three older ideas hiding inside modern MoE
↗
4.
A typed contract for routing and parameter accounting
↗
5.
Rebuilding both gpt-oss models from open code
↗
6.
A numerical router trace
↗
7.
Load balancing without conflating three mechanisms
↗
8.
Fine-grained experts and always-on shared capacity
↗
9.
What expert specialization evidence can actually support
↗
10.
Inference: the expert bank, all-to-all, and batch union
↗
11.
LatentMoE and alternatives to fixed discrete top-k
↗
12.
Level-one code: policies should be visible
↗
13.
What the notebook verifies
↗
14.
Reading the gpt-oss implementation
↗
15.
A source-aware snapshot of current MoE architectures
↗
16.
Multimodal MoE is not one architecture
↗
17.
Self-check questions
↗
18.
Exercises
↗
19.
Key takeaways
↗
20.
Notebook and primary sources
↗
Beyond the Transformer: SSMs, Mamba, and Hybrid Sequence Models
21 урок
1.
Two costs of a long context
↗
2.
From control theory to hybrid language models
↗
3.
Three readings of the same recurrence
↗
4.
Discretization with Zero-Order Hold
↗
5.
Worked example: the cost of replacing exact ZOH by $\Delta B$
↗
6.
What selectivity changes in Mamba
↗
7.
Mamba-2 and the scope of State Space Duality
↗
8.
Delta-rule memory and adaptive forgetting
↗
9.
What a fixed state can and cannot preserve
↗
10.
Memory accounting and core-only arithmetic
↗
11.
Three hybrid patterns
↗
12.
Level-1 code: exact discretization, delta update, and scan…
↗
13.
What the notebook verifies
↗
14.
Reading the open Mamba implementation
↗
15.
A current architecture passport
↗
16.
Audio, video, and other long streams
↗
17.
Self-check questions
↗
18.
Exercises
↗
19.
Key takeaways
↗
20.
Notebook and primary sources
↗
21.
Supplementary module 7b: Diffusion LLMs (dLLM)
↗
Pretraining: Data, Scaling Laws, and Optimizers
20 уроков
1.
Motivation: three irreversible choices
↗
2.
The historical arc: the objective kept expanding
↗
3.
Three classical ideas hiding inside frontier pretraining
↗
4.
Formalism: compute, a fitted loss surface, and its analytic…
↗
5.
From raw web pages to a training distribution
↗
6.
Worked example A: two budgets and the cost of a mnemonic
↗
7.
The isoFLOP valley and the fitted cost of long training
↗
8.
Inference-aware allocation: when does the smaller model repay…
↗
9.
The data wall: token counts stop being interchangeable
↗
10.
Muon: change the spectrum of a matrix update
↗
11.
MuonClip: correct the parameter source of unstable attention…
↗
12.
Code (level 1): small functions that expose their assumptions
↗
13.
Level 2: how to use the module notebook
↗
14.
µP and MetaP: make small-model tuning informative
↗
15.
Parallelism: fitting the training state before optimizing…
↗
16.
Four directions in contemporary pretraining
↗
17.
Run passports: how to read public evidence
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
KV Caches and Efficient Inference
20 уроков
1.
Motivation: a generated token is a memory transaction
↗
2.
The historical arc: four successive bottlenecks
↗
3.
A bridge to classical systems: pages, caches, and queues
↗
4.
Formalism and worked example A: how many bytes does one token…
↗
5.
One request, two computational regimes
↗
6.
Worked example B: the bandwidth ceiling and the KV term
↗
7.
Batch economics: where weight amortization stops being the…
↗
8.
PagedAttention: decouple logical length from physical placement
↗
9.
Prefix caching: do not rebuild an identical beginning
↗
10.
Tiered placement: when HBM is only the first level
↗
11.
Prefill/decode disaggregation: give each phase its own pool
↗
12.
Code (level 1): geometry, bandwidth, and pages
↗
13.
Level 2: what the notebook makes reproducible
↗
14.
Compression I: eviction changes which history exists
↗
15.
Compression II: quantization, low rank, and the order of…
↗
16.
Engines and cache contracts: a dated map, not a permanent…
↗
17.
Reading a model’s inference passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Speculative Decoding and Quantization
20 уроков
1.
Two different ways to make a token cheaper
↗
2.
From exact sampling to deployable systems
↗
3.
The older ideas underneath the modern stack
↗
4.
The exact propose–verify–correct cycle
↗
5.
Worked example A: seeing the balance component by component
↗
6.
Expected progress per cycle
↗
7.
A cost model, an optimum, and an illustrative MTP point
↗
8.
A taxonomy of proposers
↗
9.
Native multi-token prediction at inference time
↗
10.
Verification as a short, wider decode operation
↗
11.
What low-bit quantization actually changes
↗
12.
Microscaled FP4, W4A4, and the role of hardware
↗
13.
Code: make the probability contract explicit
↗
14.
The notebook as an executable contract
↗
15.
Combining the levers without multiplying away the bottlenecks
↗
16.
A deployment workflow for current engines
↗
17.
Reading a quant/spec run passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
What to retain, notebook, and sources
↗
Supervised Fine-Tuning and Data Work
20 уроков
1.
Why next-token prediction is not yet an assistant
↗
2.
From instruction tuning to structured post-training
↗
3.
Three older ideas hiding inside SFT
↗
4.
Chat templates are executable contracts
↗
5.
Label masking: a 56-token example
↗
6.
Packing variable-length examples safely
↗
7.
Compute accounting and hyperparameters
↗
8.
Organizing SFT data by capability
↗
9.
Reading LIMA correctly
↗
10.
Synthetic data as a search-and-filter system
↗
11.
Reasoning demonstrations and the s1 compute scenario
↗
12.
Teaching a model to use tools
↗
13.
Level-one code: labels and traceable bins
↗
14.
What the notebook adds
↗
15.
Distillation channels
↗
16.
LoRA and a concrete parameter count
↗
17.
Reconstructing an SFT recipe from a checkpoint
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Learning from Feedback: RLHF, DPO, RLOO, and GRPO
20 уроков
1.
The signal changed: from imitation to judgment
↗
2.
From reward modelling to environment-scale RL
↗
3.
Three old ideas that organize the new vocabulary
↗
4.
Classical RLHF as an algorithmic contract
↗
5.
Worked example A: invert a Bradley–Terry model
↗
6.
DPO: eliminate the explicit reward model
↗
7.
Worked example B: read the DPO margin carefully
↗
8.
Direct preference losses differ by contract, not branding
↗
9.
RLOO and GRPO: construct a baseline from sibling rollouts
↗
10.
Worked example D: a deliberately simplified memory model
↗
11.
Worked example E: rollout generation can dwarf the update
↗
12.
What later GRPO variants are actually fixing
↗
13.
Level-one code: the numerical core
↗
14.
Level two: the executable notebook
↗
15.
RLVR still obeys Goodhart's law
↗
16.
Three open patterns: RLAIF, multi-stage reasoning, and…
↗
17.
How to read a post-training passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Reasoning Models and Test-Time Compute
20 уроков
1.
A second scaling axis
↗
2.
From longer traces to explicit controls
↗
3.
Three older ideas hiding inside test-time compute
↗
4.
What counts as a reasoning model?
↗
5.
What DeepSeek-R1 teaches us about training reasoning
↗
6.
Parallel scaling, worked example A: voting can get worse first
↗
7.
Worked example B: coverage is not selection accuracy
↗
8.
Sequential scaling and budget forcing
↗
9.
Failure modes of longer reasoning
↗
10.
Worked example C: an illustrative budget ladder
↗
11.
Worked example D: the KV cost of a thought
↗
12.
Worked example E: why adaptive allocation matters
↗
13.
Level-1 code: make selection and cost assumptions explicit
↗
14.
Level 2: the module notebook
↗
15.
Verifier families
↗
16.
Distilling reasoning traces
↗
17.
Reading a model's reasoning passport
↗
18.
Self-check
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Multimodal LLMs: Vision, Audio, and Video
20 уроков
1.
Motivation: every modality creates a new budget
↗
2.
Four families, four places to draw the boundary
↗
3.
Three older ideas provide the mental model
↗
4.
The first design: encoder, connector, language decoder
↗
5.
CLIP, SigLIP, and worked example C: reading InfoNCE numerically
↗
6.
Worked example A: three ways to spend visual tokens
↗
7.
Worked example B: when images create ordinary decoder KV
↗
8.
Connectors are budget control surfaces
↗
9.
Dynamic resolution turns image content into variable compute
↗
10.
Unified output reveals a conflict between semantics and fidelity
↗
11.
Worked example E: video is an information-selection problem
↗
12.
Thinker–Talker: coordinating two clocks
↗
13.
Code level 1: geometry before frameworks
↗
14.
Level 2: what the notebook makes reproducible
↗
15.
Audio is a continuous evidence stream
↗
16.
Reading the public landscape by design axis
↗
17.
Follow one request through a multimodal passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Embeddings, Retrieval, and RAG
20 уроков
1.
Motivation: one query, four opportunities to lose the answer
↗
2.
Historical progression: each generation restored a signal…
↗
3.
A bridge to classical methods: three old problems inside RAG
↗
4.
BM25 and learned sparse retrieval: the branch that protects…
↗
5.
Dense bi-encoders: learning a semantic space
↗
6.
Matryoshka Representation Learning (MRL), quantization, and…
↗
7.
One vector, one pair, or a matrix: where query-document…
↗
8.
Worked example B: exact search and the arithmetic of ANN
↗
9.
ANN under the hood: three indexes for three resource constraints
↗
10.
Hybrid retrieval, reciprocal-rank fusion, and worked example F
↗
11.
Chunking: deciding what the index is allowed to retrieve
↗
12.
The full cascade and worked example D: where request latency…
↗
13.
Level-1 code: hybrid retrieval with explicit contracts
↗
14.
Level 2: what the notebook makes reproducible
↗
15.
The evolution of RAG: repair the measured failure, not the…
↗
16.
Worked example E: RAG or long context
↗
17.
Evaluation and the embedder passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Agents and Tool Use
20 уроков
1.
Motivation: a response becomes a trajectory
↗
2.
Historical arc: the control loop became a software platform
↗
3.
Classical foundations: planning, feedback control, and…
↗
4.
Anatomy of an agent loop
↗
5.
Function calling and guided generation: syntax is only the…
↗
6.
ReAct, Plan-and-Execute, and CodeAct: choosing an action…
↗
7.
Worked example A: context growth across the loop
↗
8.
Worked example B: what one agent step costs
↗
9.
MCP: standardizing connection without outsourcing trust
↗
10.
Production anatomy: the harness is part of the algorithm
↗
11.
Worked example C: reliability over a long trajectory
↗
12.
Subagents and Amdahl’s law
↗
13.
Level-1 code: a bounded protocol loop
↗
14.
Level 2: what the notebook makes reproducible
↗
15.
System landscape: compare action surfaces, not logos
↗
16.
Benchmarks: each measures a different agent–environment pair
↗
17.
Reading an agent-harness passport
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, notebook, and sources
↗
Evaluating Modern LLM Systems: From Benchmarks to Decisions
20 уроков
1.
Evaluation is an instrument for a decision
↗
2.
The historical arc: every evaluation regime repaired a failure
↗
3.
Classical measurement theory already knows most of the problems
↗
4.
How a benchmark loses measurement power
↗
5.
Build an evaluation map before choosing benchmark names
↗
6.
Worked example A: are 95.5% and 95.0% distinguishable?
↗
7.
Difficulty has several sources: HLE, FrontierMath, and ARC-AGI
↗
8.
pass@k: finding one solution is not the same as being reliable
↗
9.
Human arenas and composite indices answer someone else’s…
↗
10.
LLM judges: a synthetic 30-point illusion
↗
11.
Multimodal capability, safety, truthfulness, and honesty are…
↗
12.
Long context: accepted length versus usable evidence
↗
13.
A minimal evaluation harness: one call, one observed output
↗
14.
What the level-2 notebook makes observable
↗
15.
A private golden set in seven deliberate steps
↗
16.
Eight directions shaping evaluation as of August 2026
↗
17.
Read two passports: the benchmark and the reported run
↗
18.
Self-check questions
↗
19.
Exercises
↗
20.
Key takeaways, course finale, and sources
↗
(reference). The Basic Transformer
22 урока
1.
Why the Transformer was needed
↗
2.
Notation and tensor shapes
↗
3.
Step 1: tokens, embeddings, and positions
↗
4.
Step 2: scaled dot-product attention
↗
5.
Masks: who is allowed to see whom
↗
6.
Multi-head attention
↗
7.
The FFN: nonlinear processing at each position
↗
8.
Residual connections and LayerNorm
↗
9.
The encoder layer and encoder stack
↗
10.
The decoder layer and cross-attention
↗
11.
Training: teacher forcing and cross-entropy
↗
12.
Generation: the same forward pass in a different loop
↗
13.
A minimal scaled dot-product attention implementation
↗
14.
A minimal Transformer block
↗
15.
A complete shape trace for a toy model
↗
16.
The 2017 original and a modern LLM: what survived
↗
17.
Common confusions
↗
18.
Self-check
↗
19.
Exercises with solutions
↗
20.
The module notebook
↗
21.
Reading route and sources
↗
22.
Final cheat sheet and return to the core course
↗
Capstone: build a MiniLLM from data to ablations
6 уроков
1.
Project overview and environment
↗
2.
Pretraining: data, tokenizer, and training
↗
3.
Post-training: SFT and DPO
↗
4.
Inference: caching, speculative decoding, and quantization
↗
5.
Ablations and limits of the conclusions
↗
6.
Final report and self-assessment
↗