Lectures: David McAllester (mcallester@ttic.edu)
This class is intended to provide students with an understanding of the technical content of current research in deep learning. Students successfully completing the class should be able to read and understand current deep learning research papers and posses the technical knowledge necessary to both reproduce research results and to do original research in deep learning. The course covers current methods in computer vision, natural language processing and reinforcement learning for games and robotics. One of the amazing aspects of deep learning is that much the conceptual knowledge needed for research in these areas is shared among the areas making such broad coverage possible.
Prerequisites: This class assumes vector calculus, basic linear algebra (matrices, eigenvectors, eigenvalues), and a strong command of probability theory (multivariate Gaussians, covariance matrices, central limit theorem, Markov chains, and stationary distributions). Information theory is covered and not assumed but prior familiarity with information theory is helpful. The course is overall quite mathemtical and general mathematical maturity is required. There are also machine problems and a class programming project and previous familiarity with programming, and Python in particular, is advised.
A Timeline of Deep Leaning focusing on when and where current methods were introduced. | Slides | Lecture 1 |
Frameworks and Back-Propagation:
The Fundamental Equations of Deep Learning | Slides | Video |
Deep Learning Frameworks | Slides | Video |
Backpropagation for Scalar Source Code | Slides | Video |
Framework Objects and Backpropagation for Tensor Source Code | Slides | Video |
Minibatching: The Batch Index | Slides | Video |
The Educational Framework (EDF) | Slides | Video |
EDF and the MNIST Coding Problem | ||
PyTorch tutorial | ||
Problems | ||
Solutions |
Slides | Lecture 3 |
Pytorch Convolution Functions | |
Problems | |
Solutions |
Initialization, Normalization and Residual Connections:
Slides | Lecture 4 |
Problems | |
Solutions |
Language Modeling, Machine Translation, RNNs, and the Transformer:
Slides | Lecture 5 |
Problems | |
Solutions |
SGD I: Temperature, Batch Size, Momentum and Adam
Slides |
Problems |
Solutions |
Slides |
Problems |
Information and Generalization
Some Information Theory | Slides |
Problems | |
Solutions | |
Generalization Theory | Slides |
Problems | |
Solutions | |
Problems | |
Solutions |
GANs | Slides |
SytleGAN2 YouTube1 | |
SytleGAN2 YouTube2 | |
SytleGAN2 Paper | |
Problems | |
Solutions |
Variational Autoencoders:
Variational Autoencoders (VAEs) | Slides |
Problems | |
Solutions |
The Mathematics of Diffusion Models:
Diffusion Mathematics | Slides |
DALLE-2 Development Timeline | Slides |
Exam3 2022 |
Adjusting Generation
Generation Temperature, Guidance, and Majority Voting | Slides |
Contrastive Coding, Vector Quantization, and Autoregressive Image Modeling:
Slides | |
Problems | |
Solutions |
Reinforcement Learning (RL):
Slides | |
Problems | |
Solutions |
Slides |
Artificial General Intelligence (AGI) and AI Safety:
Slides |