Lectures: David McAllester (mcallester@ttic.edu)

This class is intended to provide students with an understanding of the technical content of current research in deep learning. Students successfully completing the class should be able to read and understand current deep learning research papers and posses the technical knowledge necessary to both reproduce research results and to do original research in deep learning. The course covers current methods in computer vision, natural language processing and reinforcement learning for games and robotics. One of the amazing aspects of deep learning is that much the conceptual knowledge needed for research in these areas is shared among the areas making such broad coverage possible.

Prerequisites: This class assumes vector calculus, basic linear algebra (matrices, eigenvectors, eigenvalues), and a strong command of probability theory (multivariate Gaussians, covariance matrices, central limit theorem, Markov chains, and stationary distributions). Information theory is covered and not assumed but prior familiarity with information theory is helpful. The course is overall quite mathemtical and general mathematical maturity is required. There are also machine problems and a class programming project and previous familiarity with programming, and Python in particular, is advised.

1. Introduction:

   A Timeline of Deep Leaning focusing on when and where current methods were introduced.	Slides	Lecture 1
   The Fundamental Equations of Deep Learning	Slides	Video

2. Frameworks and Back-Propagation:

   Deep Learning Frameworks	Slides	Video
   Backpropagation for Scalar Source Code	Slides	Video
   Framework Objects and Backpropagation for Tensor Source Code	Slides	Video
   Minibatching: The Batch Index	Slides	Video
   The Educational Framework (EDF)	Slides	Video
   EDF and the MNIST Coding Problem
   PyTorch tutorial
   Problems
   Solutions

3. Einstein Notation and Convolutional Neural Networks (CNNs):

   Slides	Lecture 3
   Pytorch Convolution Functions
   Problems
   Solutions

4. Initialization, Normalization and Residual Connections:

   Slides	Lecture 4
   Problems
   Solutions

5. Language Modeling, Machine Translation, RNNs, and the Transformer:

   Slides	Lecture 5
   Problems
   Solutions

6. SGD I: Temperature, Batch Size, Momentum and Adam

   Slides	Lectuer 6
   Problems
   Solutions

7. SGD II: Gradient Flow, Langevin Dynamics and The SGD SDE

   Slides	Lectuer 7
   Problems

8. Information and Generalization

   Information Theory Slides	Lectuer 8a
   Problems
   Solutions
   Generalization Theory Slides	Lectuer 8b
   Problems
   Solutions
   Problems
   Solutions

9. Generative Adversarial Networks (GANs):

   Slides	Lecture 9
   SytleGAN2 YouTube1
   SytleGAN2 YouTube2
   SytleGAN2 Paper
   Problems
   Solutions

10. Variational Autoencoders:

    Slides	Lecture 10
    Problems
    Solutions

11. The Mathematics of Diffusion Models:

    Slides	Lecture 11
    Exam3 2022

12. The Practice of Diffusion Models:

    Slides

    Lecture 12

13. Contrastive Coding

    Slides	Lecture 13
    Problems
    Solutions

14. Vector Quantization and Multimodal Transformer Models

    Slides

    Lecture 14

15. Reinforcement Learning (RL):

    	Slides
    Problems
    Solutions

16. AlphaZero and AlphaStar:

    Slides

17. Artificial General Intelligence (AGI) and AI Safety:

    Slides