Detailed Syllabus and Lectures

Topic 11: Deep Implicit Layers (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Multiscale Deep Equilibrium Models. Shaojie Bai, Vladlen Koltun, J. Zico Kolter. NeurIPS 2020.
• Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation. Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jaegul Choo, Joonseok Lee, Sookyung Kim, Edward Choi. AAAI 2021.

Additional Resources:

• Zico Kolter, David Duvenaud, Matt Johnson's tutorial on Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond, NeurIPS 2020.

Topic 10: Vision Transformers (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. ICLR 2021.
• Vision Transformers for Dense Prediction. Ren Ranftl, Alexey Bochkovskiy, Vladlen Koltun. ICCV 2021.

Additional Resources:

• Transformers in Vision: A Survey. Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, Mubarak Shah. ACM Computing Surveys, 2021.

Topic 9: Memory and Attention (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Learning to Simulate Dynamic Environments With GameGAN. Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, Sanja Fidler. CVPR 2020.
• Kanerva++: Extending the Kanerva Machine With Differentiable, Locally Block Allocated Latent Memory. Jason Ramapuram, Yan Wu, Alexandros Kalousis. ICLR 2021.

Additional Resources:

• On Memory in Human and Artificial Language Processing Systems. Aida Nematzadeh, Sebastian Ruder & Dani Yogatama, Workshop on Bridging AI and Cognitive Science, ICLR 2020
• Alex Graves' lecture on Attention and Memory in Deep Learning, DeepMind x UCL Deep Learning Lectures, 2020.

Topic 8: Neural rendering (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Understanding Object Dynamics for Interactive Image-to-Video Synthesis. Andreas Blattmann, Timo Milbich, Michael Dorkenwald, Björn Ommer. CVPR 2020.
• PixelSynth: Generating a 3D-Consistent Experience from a Single Image. Chris Rockwell, David F. Fouhey, Justin Johnson. ICCV 2021.

Additional Resources:

• Advances in Neural Rendering (Part 1) (Part 2). Michael Zollhöfer et al., SIGGRAPH 2021 Courses.

Topic 7: Dynamic networks (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Semantically-Guided Representation Learning for Self-Supervised Monocular Depth. Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, Adrien Gaidon. ICLR 2020.
• Spatially-Adaptive Pixelwise Networks for Fast Image Translation. Tamar Rott Shaham, Michael Gharbi, Richard Zhang, Eli Shechtman, Tomer Michaeli. CVPR 2021.

Additional Resources:

• Dynamic Neural Networks: A Survey. Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, appeared online.

Topic 6: Neural implicit representations (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Implicit neural representations with periodic activation functions. Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, Gordon Wetzstein. NeurIPS 2020.
• pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein. CVPR 2021.

Additional Resources:

• Awesome Implicit Neural Representations, A curated list of resources on implicit neural representations, Vincent Sitzmann.

Topic 5: Neuro-symbolic approaches (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning. Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B. Tenenbaum, Chuang Gan. ICLR 2021.
• PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World. Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, Yejin Choi. ACL 2021.

Additional Resources:

• Neurosymbolic AI. David Cox, MIT 6.S191, 2020.

Topic 4: Object-centric representation learning (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Learning Physical Graph Representations from Visual Scenes. Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F. Fei-Fei, Jiajun Wu, Josh Tenenbaum, Daniel L. Yamins. NeurIPS 2020.
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition. Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, Christopher P. Burgess. arXiv preprint arXiv:2106.03849, 2021.

Additional Resources:

• What are Objects?. Klaus Greff, ICML 2020 Workshop on Object-Oriented Learning: Perception, Representation, and Reasoning, 2020.
• What is an object?. Irina Higgins, NeurIPS 2020 Workshop on Object Representations for Learning and Reasoning, 2020.

Topic 3: Graph neural networks (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, Antonio Torralba. ICLR 2019.
• Causal Discovery in Physical Systems from Videos. Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, Animesh Garg. NeurIPS 2020.

Additional Resources:

• Principles and applications of relational inductive biases in deep learning. Kelsey Allen, MIT CBMM Talks, 2019.

Topic 2: Multimodal representation learning (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech. David Harwath, Wei-Ning Hsu, James Glass. ICLR 2020.
• Learning to Learn Words from Visual Scenes. Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, Carl Vondrick. ECCV 2020.

Additional Resources:

• Extending Machine Language Models toward Human-Level Language Understanding. James L. McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, Hinrich Schütze. arXiv preprint arXiv:1912.05877, 2020.

Topic 1: Compositionality and systematic generalization (paper 1 slides) (paper 1 video) · (paper 2 slides) (paper 2 video)

• A Benchmark for Systematic Generalization in Grounded Language Understanding. Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, Brenden M. Lake. NeurIPS 2020.
• Benchmark for Compositional Text-to-Image Synthesis. Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, Anna Rohrbach. NeurIPS 2021 Datasets and Benchmarks Track.

Additional Resources:

• Compositionality decomposed: how do neural networks generalise?. Hupkes D., Dankers V., Mul M. and Bruni E. JAIR, Vol 67, 220.
• Connectionism and cognitive architecture: A critical analysis. Jerry A Fodor and Zenon W Pylyshyn. Cognition, 28(1):3–71, 1988.

Lecture 2: A Look at Deep Learning Landscape (slides)

course information, unsupervised learning

Please study the following material in preparation for the class:

Required Reading:

• Building Machines That Learn and Think Like People. Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman. Behavioral and Brain Sciences, Vol. 40, 2016
• Deep Learning for AI. Yoshua Bengio, Yann Lecun, Geoffrey Hinton. Communications of the ACM, July 2021.
• Why AI is Harder Than We Think. Melanie Mitchell. arXiv preprint, 2021.
• Intelligence without Reason. Rodney Brooks. 1991.

Suggested Video Material:

• Eric Horvitz and Peter Norvig's discussion on The Challenge & Promise of Artificial Intelligence, 2012.

Lecture 1: Introduction to the course (video)

course information, coarse format, presentation roles