Detailed Syllabus and Lectures
Lecture 8: Attention and Transformers (slides)
content-based attention, location-based attention, soft vs. hard attention, self-attention, attention for image captioning, transformer networks, vision transformers
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
- Neural Machine Translation by Jointly Learning to Align and Translate, D. Bahdanau, K. Cho, Y. Bengio, ICLR 2015
- Sequence Modeling with CTC, Awni Hannun, Distill, 2017
- Recurrent Models of Visual Attention, V. Mnih, N. Heess, A. Graves, K. Kavukcuoglu, NIPS 2014
- DRAW: a Recurrent Neural Network for Image Generation, K. Gregor, I. Danihelka, A. Graves, DJ Rezende, D. Wierstra, ICML 2015
- Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, NIPS 2017
- [Blog post] What is DRAW (Deep Recurrent Attentive Writer)?, Kevin Frans
- [Blog post] The Transformer Family, Lilian Weng
Lecture 7: Recurrent Neural Networks (slides)
sequence modeling, recurrent neural networks (RNNs), RNN applications, vanilla RNN, training RNNs, long short-term memory (LSTM), LSTM variants, gated recurrent unit (GRU)
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
Lecture 6: Understanding and Visualizing Convolutional Neural Networks (slides)
transfer learning, interpretability, visualizing neuron activations, visualizing class activations, pre-images, adversarial examples, adversarial training
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
- [Blog post] Understanding Neural Networks Through Deep Visualization, Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.
- [Blog post] The Building Blocks of Interpretability, Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye and Alexander Mordvintsev.
- [Blog post] Feature Visualization, Chris Olah, Alexander Mordvintsev and Ludwin Schubert.
- [Blog post] An Overview of Early Vision in InceptionV1, Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter.
- [Blog post] OpenAI Microscope.
- [Blog post] Breaking Linear Classifiers on ImageNet, Andrej Karpathy.
- [Blog post] Attacking machine learning with adversarial examples, OpenAI.
Lecture 5: Convolutional Neural Networks (slides)
convolution layer, pooling layer, cnn architectures, design guidelines, semantic segmentation networks, addressing other tasks
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
Lecture 4: Training Deep Neural Networks (slides)
data preprocessing, weight initialization, normalization, regularization, model ensembles, dropout, optimization methods
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
- Stochastic Gradient Descent Tricks, Leon Bottou.
- Section 3 of Practical Recommendations for Gradient-Based Training of Deep Architectures, Yoshua Bengio.
- Troubleshooting Deep Neural Networks: A Field Guide to Fixing Your Model, Josh Tobin.
- [Blog post] Initializing neural networks, Katanforoosh & Kunin, deeplearning.ai.
- [Blog post] Parameter optimization in neural networks, Katanforoosh et al., deeplearning.ai.
- [Blog post] The Black Magic of Deep Learning - Tips and Tricks for the practitioner, Nikolas Markou.
- [Blog post] An overview of gradient descent optimization algorithms, Sebastian Ruder.
- [Blog post] Why Momentum Really Works, Gabriel Goh
Lecture 3: Multi-layer Perceptrons (slides)
feed-forward neural networks, activation functions, chain rule, backpropagation, computational graph, automatic differentiation, distributed word representations
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
Lecture 2: Machine Learning Overview (slides)
types of machine learning problems, linear models, loss functions, linear regression, gradient descent, overfitting and generalization, regularization, cross-validation, bias-variance tradeoff, maximum likelihood estimation
Please study the following material in preparation for the class:
Required Reading:
Suggested Video Material:
Additional Resources:
Lecture 1: Introduction to Deep Learning (slides)
course information, what is deep learning, a brief history of deep learning, compositionality, end-to-end learning, distributed representations
Please study the following material in preparation for the class:
Required Reading:
Additional Resources:
- The unreasonable effectiveness of deep learning in artificial intelligence, Terrence J. Sejnowski, PNAS, 2020.
- Deep Learning, Yann LeCun, Yoshio Bengio, Geoffrey Hinton. Nature, Vol. 521, 2015.
- Deep Learning in Neural Networks: An Overview, Juergen Schmidhuber. Neural Networks, Vol. 61, pp. 85–117, 2015.
- On the Origin of Deep Learning, Haohan Wang and Bhiksha Raj, arXiv preprint arXiv:1702.07800v4, 2017