Lectures: David McAllester (mcallester@ttic.edu)
This class is intended to provide students with an understanding of the technical content of current research in deep learning. Students successfully completing the class should be able to read and understand current deep learning research papers and posses the technical knowledge necessary to both reproduce research results and to do original research in deep learning. The course covers current methods in computer vision, natural language processing and reinforcement learning for games and robotics. One of the amazing aspects of deep learning is that much the conceptual knowledge needed for research in these areas is shared among the areas making such broad coverage possible.
Prerequisites: This class assumes vector calculus, basic linear algebra (matrices, eigenvectors, eigenvalues), and a strong command of probability theory (multivariate Gaussians, covariance matrices, central limit theorem, Markov chains, and stationary distributions). Information theory is covered and not assumed but prior familiarity with information theory is helpful. The course is overall quite mathemtical and general mathematical maturity is required. There are also machine problems and a class programming project and previous familiarity with programming, and Python in particular, is advised.
| A Timeline of Deep Leaning focusing on when and where current methods were introduced. | Slides | Lecture 1 |
| The Fundamental Equations of Deep Learning | Slides | Video |
| Problems | ||
| Solutions |
Frameworks and Back-Propagation:
| Deep Learning Frameworks | Slides | Video |
| Backpropagation for Scalar Source Code | Slides | Video |
| Framework Objects and Backpropagation for Tensor Source Code | Slides | Video |
| Minibatching: The Batch Index | Slides | Video |
| The Educational Framework (EDF) | Slides | Video |
| EDF and the MNIST Coding Problem | ||
| PyTorch tutorial | ||
| Problems | ||
| Solutions |
| Slides | Lecture 3 |
| Pytorch Convolution Functions | |
| Problems | |
| Solutions |
Initialization, Normalization and Residual Connections:
| Slides | Lecture 4 |
| Problems | |
| Solutions |
Language Modeling, Machine Translation, RNNs, and the Transformer:
| Slides | Lecture 5 |
| Problems | |
| Solutions |
SGD I: Temperature, Batch Size, Momentum and Adam
| Slides | Lectuer 6 |
| Problems | |
| Solutions |
| Slides | Lectuer 7 |
| Problems |
Information and Generalization
| Information Theory Slides | Lectuer 8a |
| Problems | |
| Solutions | |
| Generalization Theory Slides | Lectuer 8b |
| Problems | |
| Solutions |
| Slides | Lecture 9 |
| SytleGAN2 YouTube1 | |
| SytleGAN2 YouTube2 | |
| SytleGAN2 Paper | |
| Problems | |
| Solutions |
Variational Autoencoders:
| Slides | Lecture 10 |
| Problems | |
| Solutions |
Vector Quantization and Multimodal Transformer Models
| Slides | Lecture 14 |
The Mathematics of Diffusion Models:
| Slides | Lecture 11 |
| Problems |
The Practice of Diffusion Models:
| Slides | Lecture 12 |
Mutual Information Coding
| Slides | Lecture 13 |
| Problems | |
| Solutions |
Reinforcement Learning (RL):
| Slides | |
| Problems | |
| Solutions |
| Slides |
| Slides |
Theorem Proving:
| Slides |
The Foundations of Mathematics:
| Slides |
AI Safety:
| Slides |