Чему вы научитесь
- Distinguish surprisal, entropy, cross-entropy, negative log-likelihood, KL divergence, and perplexity—including which distribution defines the expectation and which units are being used.
- Interpret probabilistic prediction as a coding problem and connect log loss to expected code length without confusing compression with meaning or truth.
- Apply Jensen’s inequality, the log-sum inequality, and variational decompositions to analyze ensembles, the ELBO, and approximation gaps.
- Compute and interpret mutual and conditional mutual information, use the data-processing inequality, and understand the limitations of finite-sample MI estimation.
- Formulate source- and channel-coding problems, channel capacity, typical sets, and rate–distortion trade-offs.
- Derive maximum-entropy distributions and exponential tilts; understand the roles of softmax, energy-based models, and KL-regularized policies.
- Distinguish the Information Bottleneck, variational IB, VAE/ELBO, and rate–distortion formulations rather than treating them as interchangeable names.
- Compare KL, f-divergences, MMD, and Wasserstein distance and choose an appropriate discrepancy for a particular ML problem.
- Read PAC-Bayes and mutual-information generalization bounds and determine what they do—and do not—guarantee.
- Analyze LLM reasoning, test-time compute, computer-vision representations, neural compression, and natural gradient using precise information-theoretic models.
- Reproduce key numerical examples in Python and critically evaluate information-theoretic claims in current ML papers.
О курсе
Для кого этот курс
Начальные требования
Basic probability: random variables, conditional probability, Bayes’ rule, expectation, and variance.
Basic calculus: functions, derivatives, gradients, and elementary integrals.
Some linear algebra is helpful: vectors, matrices, inner products, and quadratic forms.
Familiarity with introductory machine learning: training and validation data, loss functions, probabilistic classification, and gradient-based optimization.
Python, NumPy, and notebook experience are useful for the computational exercises but are not required for the theoretical path.
Measure theory, functional analysis, advanced coding theory, and other specialist mathematical courses are not required; the necessary constructions are introduced as they are needed.
Преподаватели курса
1
Как проходит обучение
The course is open, free, self-paced, and primarily text-based, with formulas, illustrations, short assessments, and computational examples.
It consists of 13 core modules, two optional advanced modules, and a final reference sheet:
• Modules 1–13 form the core path, from entropy, KL, and mutual information to coding, the Information Bottleneck, generalization, and LLM reasoning.
• Module 14 studies computer vision through information theory.
• Module 15 introduces information geometry and natural gradient.
• Module 16 is a compact reference sheet covering the main ideas and formulas.
Modules 14 and 15 are optional. You can skip them on a first pass or choose them according to your professional interests.
Most modules follow the same learning route:
1. A practical or ML question: what problem are we trying to understand?
2. An intuitive model and a precise problem statement.
3. Formal definitions, key properties, and derivations of the main formulas.
4. Classical examples and applications to ML, NLP, LLMs, computer vision, or reinforcement learning.
5. Auto-graded questions after the main theoretical steps.
6. Analytical and computational exercises.
7. Detailed solutions and reproducible Python code where appropriate.
More mathematical sections distinguish between material intended for a first pass and material that can be revisited later. Full proofs and specialist topics can be postponed without losing the main ML route.
Exercises are ungraded and optional, but they are the main way to turn familiarity with a formula into a working skill. Solutions are published separately: it is best to attempt a problem first and then compare approaches.
A study load of roughly four to five hours per week is recommended, but there is no fixed schedule.
Что вы получите
- Full open and free access to all 15 topic modules and the final course reference sheet.
- Complete Russian and English versions of the course materials.
- Auto-graded questions covering key concepts, formulas, and limits of applicability.
- A collection of conceptual, analytical, and computational exercises.
- Detailed exercise solutions and tested Python code for computational experiments.
- More than one hundred bilingual diagrams and illustrations explaining formulas, algorithms, and relationships between problem settings.
- Annotated lists of primary sources and suggestions for further study.
- A final reference sheet covering the main quantities, formulas, and diagnostic questions used throughout the course.