0. Introduction | Slides | Notebook
Course content, a deliverable, and spam classification in PyTorch.
1. Basic linear Algebra in PyTorch | Slides | Notebook | Live Demo
Basic linear algebra operations in PyTorch.
2. Linear Regression: Direct Methods | Slides | Notebook
Direct methods for solving least squares problems, comparing LU and QR factorization.
3. Linear Regression: Gradient Descent | Slides | Notebook
Linear regression via gradient descent.
4. How to compute gradients in PyTorch | Slides | Notebook
Introduction to PyTorch’s automatic differentiation system.
5. How to think about derivatives through best linear approximation
How to think about derivatives through best linear approximation.
6. Stochastic gradient descent: A first look
A first look at stochastic gradient descent through the mean estimation problem.
7. Stochastic gradient descent: insights from the Noisy Quadratic Model
When should we use exponential moving averages, momentum, and preconditioning?
8. Stochastic Gradient Descent: The general problem and implementation details | Notebook
Stochastic optimization problems, SGD, tweaks, and implementation in PyTorch
9. Adaptive Optimization Methods | Notebook
Intro to adaptive optimization methods: Adagrad, Adam, and AdamW.
10. Benchmarking Optimizers: Challenges and Some Empirical Results
How do we compare optimizers for deep learning?
11. A Playbook for Tuning Deep Learning Models
A systematic process for tuning deep learning models