Содержание курса
16
разделов
204
урока
377
тестов
обн. 9 августа 2026
Introduction
11 уроков
1.
Information and meaning: why they are not the same thing
↗
2.
The Shannon frame: strength and limitations
↗
3.
Why logarithms appear: a short bridge to entropy
↗
4.
How the course is structured and how to read it
↗
5.
Course program: overview
↗
6.
Literature: where to start
↗
7.
Reference: detailed course program
↗
8.
Reference: complete annotated reading list
↗
9.
Optional mathematical deepening I: the uniform source
↗
10.
Optional mathematical deepening II: from grouping to Shannon…
↗
11.
Computational block
↗
Entropy
8 уроков
1.
Entropy: the average bill for uncertainty
↗
2.
The chain rule: splitting one bill across tokens
↗
3.
From a coin to language: entropy rate
↗
4.
How to read entropy in a language model
↗
5.
Exercises
↗
6.
References and primary sources
↗
7.
Computational block
↗
8.
Exercise solutions
↗
Cross-Entropy and KL Divergence
13 уроков
1.
The model returns a distribution; the world returns one outcome
↗
2.
Cross-entropy: the model's average logarithmic bill
↗
3.
KL divergence: the part of the loss attributable to the model
↗
4.
The wrong code: how KL becomes extra bits
↗
5.
One formula for classification and language modeling
↗
6.
Why CrossEntropyLoss is so convenient to optimize
↗
7.
Perplexity: returning from the logarithmic scale
↗
8.
Mathematical deepening: why KL direction changes the answer
↗
9.
A map of KL objectives in machine learning
↗
10.
Exercises
↗
11.
References and primary sources
↗
12.
Computational block
↗
13.
Exercise solutions
↗
Jensen's inequality: when averaging meets nonlinearity
11 уроков
1.
Average first, or evaluate the loss first?
↗
2.
Jensen’s inequality: curvature sets the direction
↗
3.
Ensembles: an exact guarantee for log loss
↗
4.
The ELBO: the logarithm sits outside hidden explanations
↗
5.
The log-sum inequality: what is lost when states are merged
↗
6.
One theorem, four working mechanisms
↗
7.
What to carry into ML practice
↗
8.
Exercises
↗
9.
References and primary sources
↗
10.
Computational block
↗
11.
Exercise solutions
↗
Mutual information: how much one object tells us about another
16 уроков
1.
A feature is useful only relative to a target
↗
2.
One quantity, three readings
↗
3.
PMI: the local log ratio of association
↗
4.
Conditional MI: what a new feature adds beyond known features
↗
5.
DPI: an encoder can repackage information, but it cannot…
↗
6.
MI, correlation, and causality answer different questions
↗
7.
Noisy labels: when 100% error still carries one bit
↗
8.
Feature selection: a useful filter, not an oracle
↗
9.
InfoNCE: finding the matched pair among random candidates
↗
10.
Why MI is hard to estimate from samples
↗
11.
Attention entropy is not mutual information
↗
12.
What to carry into ML practice
↗
13.
Exercises
↗
14.
Sources and further reading
↗
15.
Computational block
↗
16.
Exercise solutions
↗
Source coding: from entropy to real bits
16 уроков
1.
Engineering setup: the probabilistic model and the entropy coder
↗
2.
Prefix codes and the Kraft--McMillan inequality
↗
3.
The one-symbol source-coding theorem
↗
4.
Huffman coding, arithmetic coding, and range coding
↗
5.
Long sequences: block coding, the AEP, and entropy rate
↗
6.
Language-model metrics: PPL, BPC, and BPB
↗
7.
A language model as part of a compressor
↗
8.
Model size, scaling laws, and total description length
↗
9.
Mathematical deepening: universal coding, sequential log…
↗
10.
Compression quality and model capabilities
↗
11.
A practical protocol for evaluating an LLM as a compressor
↗
12.
Exercises
↗
13.
Conclusion
↗
14.
Primary sources
↗
15.
Computational block
↗
16.
Exercise solutions
↗
Channel and capacity
16 уроков
1.
Reliable communication through noise
↗
2.
A discrete memoryless channel and a block code
↗
3.
Capacity of a discrete channel
↗
4.
BSC, BEC, and the Z-channel
↗
5.
The Gaussian channel and a power constraint
↗
6.
Why random coding achieves \(R<I(X;Y)\)
↗
7.
Converse and finite blocklength
↗
8.
Noisy labels as a channel
↗
9.
Channel coding inside ML: ECOC and DeepJSCC
↗
10.
An LLM as a channel: a precise model and the limits of the…
↗
11.
Formalizing a channel in an ML system
↗
12.
Exercises
↗
13.
Conclusion
↗
14.
Primary sources and further reading
↗
15.
Computational block
↗
16.
Exercise solutions
↗
Maximum entropy, exponential models, and KL regularization
15 уроков
1.
From partial knowledge to a distribution
↗
2.
Exponential tilting and information projection
↗
3.
The state space and reference measure
↗
4.
Classical MaxEnt distributions and the AWGN channel
↗
5.
Conditional MaxEnt and log-linear models
↗
6.
Softmax as score-plus-entropy optimization
↗
7.
Energy-based models and the partition function
↗
8.
A KL-regularized policy and its Gibbs optimum
↗
9.
Reward overoptimization and the role of KL
↗
10.
Mathematical deepening: exponential families and sufficiency
↗
11.
Exercises
↗
12.
Conclusion
↗
13.
Primary sources
↗
14.
Computational block
↗
15.
Exercise solutions
↗
Rate–distortion and the Information Bottleneck: managing…
14 уроков
1.
Why a model may need to forget
↗
2.
Rate–distortion theory: how much information must be retained
↗
3.
Information Bottleneck: preserve what matters for \(Y\)
↗
4.
Variational Information Bottleneck: making IB trainable
↗
5.
The VAE as a rate–distortion system
↗
6.
When rate becomes actual bits
↗
7.
Building an honest rate–distortion curve
↗
8.
The information plane: an objective or a description of…
↗
9.
Mathematical extension: I–MMSE and Gaussian noising
↗
10.
Exercises
↗
11.
What to carry forward
↗
12.
Primary sources
↗
13.
Computational block
↗
14.
Exercise solutions
↗
Algorithmic Information Theory: Shortest Descriptions, MDL…
14 уроков
1.
From a distribution to one particular object
↗
2.
Kolmogorov complexity and the description language
↗
3.
Incompressibility, uncomputability, and real compressors
↗
4.
The bridge to Shannon entropy
↗
5.
Algorithmic probability and universal prediction
↗
6.
MDL in machine learning
↗
7.
Similarity through joint compression
↗
8.
LLMs and universal prediction
↗
9.
Mathematical deepening: AIXI
↗
10.
Optional physical bridge: Landauer's principle
↗
11.
Exercises
↗
12.
What to retain
↗
13.
Primary sources
↗
14.
Exercise solutions
↗
Comparing distributions: f-divergences, variational critics…
14 уроков
1.
From two sample sets to a distribution-comparison problem
↗
2.
f-divergences: one template, different penalties
↗
3.
What f-divergences can see
↗
4.
Classification as density-ratio estimation
↗
5.
Variational form: making a discrepancy trainable
↗
6.
GAN and f-GAN
↗
7.
MINE and InfoNCE
↗
8.
Integral probability metrics: Wasserstein distance and MMD
↗
9.
Choosing a discrepancy in practice
↗
10.
Mathematical deepening: general measures and exact duality
↗
11.
Exercises
↗
12.
Conclusion
↗
13.
Primary references
↗
14.
Exercise solutions
↗
Information-theoretic generalization bounds: PAC-Bayes and…
14 уроков
1.
Why the best training error is almost inevitably too good
↗
2.
From a fixed model to adaptive selection
↗
3.
PAC-Bayes: a distribution before data and a distribution…
↗
4.
PAC-Bayes-kl: a certificate for a realized sample
↗
5.
From a certificate to a trainable distribution
↗
6.
MI bounds: learning as a channel from sample to model
↗
7.
The exact decomposition connecting PAC-Bayes and MI
↗
8.
Connections to MDL, Information Bottleneck, and…
↗
9.
Building a practical PAC-Bayes certificate
↗
10.
Mathematical deepening: more local information costs
↗
11.
Exercises
↗
12.
What to take away
↗
13.
Main references
↗
14.
Exercise solutions
↗
Reasoning in LLMs: computation, search, and information
15 уроков
1.
What changes when a model “thinks longer”
↗
2.
Amortization, scratchpads, and search
↗
3.
Why intermediate tokens can help
↗
4.
New information or new computation?
↗
5.
What can be measured along a trajectory?
↗
6.
One long trajectory or many candidates?
↗
7.
In-context learning: when context really brings evidence
↗
8.
How reasoning models are trained
↗
9.
How to evaluate a reasoning system
↗
10.
What is proved, what is observed, and what remains a hypothesis
↗
11.
Mathematical deepening: stepwise information gain
↗
12.
Exercises
↗
13.
Conclusion: information is not computation
↗
14.
Primary sources
↗
15.
Exercise solutions
↗
Computer Vision Through Information Theory
13 уроков
1.
Why computer vision is a useful testbed for information theory
↗
2.
Learned image compression: entropy models and actual bitstreams
↗
3.
Distortion and perceptual quality
↗
4.
Diffusion models: Gaussian noising, denoising, and I–MMSE
↗
5.
CLIP and contrastive image–text alignment
↗
6.
Self-supervised visual representations: invariances and…
↗
7.
Compressing a CV model: quantization, pruning, and distillation
↗
8.
One CV pipeline, four different information budgets
↗
9.
Mathematical deepening: two exact calculations
↗
10.
Exercises
↗
11.
Conclusion
↗
12.
Primary references
↗
13.
Exercise solutions
↗
Information Geometry: Fisher, Natural Gradient, and…
13 уроков
1.
What Does a “Small Step” Mean for a Probabilistic Model?
↗
2.
Fisher Information and the Local Form of KL
↗
3.
Natural Gradient: Steepest Descent Under Small Model Change
↗
4.
What “Curvature” Is Actually Used in a Deep Network?
↗
5.
Policy Geometry: NPG, TRPO, PPO, and KL in RLHF
↗
6.
Wasserstein Geometry and Distribution Flows
↗
7.
Exponential Families, Dual Coordinates, and Mirror Descent
↗
8.
Practical Map: Choose the Object Before the Geometry
↗
9.
Mathematical Deepening: α-Geometry and Exact Examples
↗
10.
Exercises
↗
11.
Conclusion
↗
12.
Main Sources
↗
13.
Exercise solutions
↗
Course Reference Sheet
1 урок
1.
Course Reference Sheet: Key Ideas and Formulas
↗