TTIC 31230: Fundamentals of Deep Learning

Lectures: David McAllester (mcallester@ttic.edu)

This class is intended to provide students with an understanding of the technical content of current research in deep learning. Students successfully completing the class should be able to read and understand current deep learning research papers and posses the technical knowledge necessary to both reproduce research results and to do original research in deep learning. The course covers current methods in computer vision, natural language processing and reinforcement learning for games and robotics. One of the amazing aspects of deep learning is that much the conceptual knowledge needed for research in these areas is shared among the areas making such broad coverage possible.

Prerequisites: This class assumes vector calculus, basic linear algebra (matrices, eigenvectors, eigenvalues), and a strong command of probability theory (multivariate Gaussians, covariance matrices, central limit theorem, Markov chains, and stationary distributions). Information theory is covered and not assumed but prior familiarity with information theory is helpful. The course is overall quite mathemtical and general mathematical maturity is required. There are also machine problems and a class programming project and previous familiarity with programming, and Python in particular, is advised.

  1. Introduction:

    A Timeline of Deep Leaning focusing on when and where current methods were introduced. Slides Lecture 1

  2. Frameworks and Back-Propagation:

    The Fundamental Equations of Deep Learning Slides Video
    Deep Learning Frameworks Slides Video
    Backpropagation for Scalar Source Code Slides Video
    Framework Objects and Backpropagation for Tensor Source Code Slides Video
    Minibatching: The Batch Index Slides Video
    The Educational Framework (EDF) Slides Video
    EDF and the MNIST Coding Problem
    PyTorch tutorial
    Problems
    Solutions

  3. Einstein Notation and Convolutional Neural Networks (CNNs):

    Slides Lecture 3
    Pytorch Convolution Functions
    Problems
    Solutions

  4. Initialization, Normalization and Residual Connections:

    Slides Lecture 4
    Problems
    Solutions

  5. Language Modeling, Machine Translation, RNNs, and the Transformer:

    Slides Lecture 5
    Problems
    Solutions

  6. SGD I: Temperature, Batch Size, Momentum and Adam

    Slides
    Problems
    Solutions

  7. SGD II: Gradient Flow, Langevin Dynamics and The SGD SDE

    Slides
    Problems

  8. Information and Generalization

    Some Information Theory Slides
    Problems
    Solutions
    Generalization Theory Slides
    Problems
    Solutions
    Problems
    Solutions

  9. Generative Adversarial Networks (GANs):

    GANs Slides
    SytleGAN2 YouTube1
    SytleGAN2 YouTube2
    SytleGAN2 Paper
    Problems
    Solutions

  10. Variational Autoencoders:

    Variational Autoencoders (VAEs) Slides
    Problems
    Solutions

  11. The Mathematics of Diffusion Models:

    Diffusion Mathematics Slides
    DALLE-2 Development Timeline Slides
    Exam3 2022

  12. Adjusting Generation

    Generation Temperature, Guidance, and Majority Voting Slides

  13. Contrastive Coding, Vector Quantization, and Autoregressive Image Modeling:

    Slides
    Problems
    Solutions

  14. Reinforcement Learning (RL):

    Slides
    Problems
    Solutions

  15. AlphaZero and AlphaStar:

    Slides

  16. Artificial General Intelligence (AGI) and AI Safety:

    Slides