Course Information
About
This course is about computational photography, an emerging new research area which brings together the advancements in computer graphics, computer vision and image processing to overcome the limitations of conventional photography. The course is structured around basic topics such as cameras and image formation, high dynamic range imaging, edge-aware filtering, gradient-domain processing, deconvolution, blending and compositing, visual quality assessment, deep image enhancement, and neural rendering. The goal of this course is to introduce students to a number of different computational techniques to capture, manipulate and enrich visual media
The main goal of this course is to introduce students to a number of different computational techniques to capture, manipulate and enrich visual media. The students are expected to develop a foundational understanding and knowledge of concepts that underly computational photography. The students will also be expected to gain hands-on experience via a set of programming assignments and a course project.
The course is taught by Aykut Erdem. The teaching assistant is Canberk Baykal.
Time and Location
Lectures: Monday, Wednesday at 11:30-12:40 (SNA B152)
Reference Books
- Computer Vision: Algorithms and Applications, Richard Szeliski, Second Edition, 2022 (pdf available online).
- Photography (10th edition), Barbara London, Jim Stone, and John Upton, Pearson, 2010
- Computer Vision: A Modern Approach (2nd Edition), David Forsyth and Jean Ponce, Prentice Hall, 2012.
Policies: All work on assignments must be done individually unless stated otherwise. You are encouraged to discuss with your classmates about the given assignments, but these discussions should be carried out in an abstract way. That is, discussions related to a particular solution to a specific problem (either in actual code or in pseudocode) will not be tolerated.
In short, turning in someone else’s work, in whole or in part, as your own will be considered a violation of academic integrity. Please note that the former condition also holds for the material found on the web, as everything on the web has been written by someone else.
Communication
The course webpage will be updated regularly throughout the semester with lecture notes, presentations, assignments and important deadlines. All other course-related communications will be carried out through Blackboard.
Pre-requisites
Good math (calculus, linear algebra, statistics) and programming skills.
Course Requirements and Grading
Grading for COMP527/ELEC519 will be based on
- Five programming assignments (done individually) (9% each),
- Course project (done in pairs) (30%),
- A midterm exam (20%), and
- Class participation (5%)
Schedule
| Date | Topic | Notes |
| Feb 27, Mar 1 | Introduction, Digital photography (slides) | Brian Hayes, Computational Photography, American Scientist 96, 94-99, 2008 Michael Johnston, Your Camera Roll Contains A Masterpiece, New Yorker, March 31, 2022 |
| Mar 6, Mar 8 | Image formation (slides) | Szeliski, Chapter 2 Forsyth and Ponce, Chapter 1.1 Antonio Torralba and William T. Freeman, Accidental pinhole and pinspeck cameras, CVPR 2012 |
| Mar 13, Mar 15 | Noise and Color (slides) Assg1 out: Camera Pipeline |
Szeliski, Chapter 2.3, 3.1.2, 10.1 Forsyth and Ponce, Chapter 3 Michael S. Brown, Understanding the In-Camera Image Processing Pipeline for Computer Visions, CVPR 2016 Tutorial S.J. Gortler, Chapter 19 (Color) of Foundations of 3D Computer Graphics, MIT Press, 2012 |
| Mar 20, Mar 22 | Exposure and high-dynamic-range imaging (slides) | Szeliski, Chapter 10.1, 10.2 Paul E. Debevec and Jitendra Malik, Recovering High Dynamic Range Radiance Maps from Photographs, SIGGRAPH 1997 Tomoo Mitsunaga and Shree K. Nayar, Radiometric Self Calibration, CVPR 1999 Erik Reinhard et al., Photographic tone reproduction for digital images, CVPR 1999 |
| Mar 27, Mar 29 | Image filtering (slides) Assg1 in, Assg2 out: HDR Imaging and Tonemapping |
Szeliski, Chapter 3.2, 3.3 Carlo Tomasi and Roberto Manduchi, Bilateral Filtering for Gray and Color Images, ICCV 1998 Sylvain Paris et al., A Gentle Introduction to the Bilateral Filter and Its Applications, SIGGRAPH 2008 class Georg Petschnigg et al., Digital photography with flash and no-flash image pairs, SIGGRAPH 2004 Kaiming He et al., Guided image filtering, ECCV 2010 Antoni Buades et al., A non-local algorithm for image denoising, CVPR 2005 Levent Karacan et al., Structure Preserving Image Smoothing via Region Covariances, SIGGRAPH Asia 2013 Qi Zhang et al., Rolling Guidance Filter, ECCV 2014 |
| Apr 3, Apr 5 | Gradient-domain image processing (slides) Course project proposal due |
Szeliski, Chapter 3.1.3, 3.5.5, 10.4.3 Pérez et al., Poisson Image Editing, SIGGRAPH 2003 Agrawal et al., Removing Photography Artifacts Using Gradient Projection and Flash-Exposure Sampling, SIGGRAPH 2005 Agrawal and Raskar, Gradient Domain Manipulation Techniques in Vision and Graphics, ICCV 2007 tutorial Bhat et al., GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering, ACM Trans. Graphics, 2010 Tumblin et al., Why I want a gradient camera?, CVPR 2005 Hua et al., A Survey on Gradient-Domain Rendering, Eurographics 2019 State of the Art Reports (STAR) Gallego et al., Event-based Vision: A Survey, IEEE TPAMI, 2020 |
| Apr 10, Apr 12 | Focal stacks and lightfields (slides) Assg2 in, Assg3 out: Flash/No Flash Photography |
Szeliski, Chapter 12.1.3, 14.3 Suwajanakorn et al., Depth from Focus with Your Mobile Phone, CVPR 2015 Levoy and Hanrahan, Light Field Rendering, SIGGRAPH 1996 Gortler et al., The Lumigraph, SIGGRAPH 1996 Ng et al., Light field photography with a hand-held plenoptic camera, Stanford TR 2005-02 |
| Apr 17, Apr 19 | No class - Spring Break | |
| Apr 24, Apr 26 | Deconvolution, Coded photography (slides) | Szeliski, Chapter 3.4.3, 3.4.4, 10.1.4, 10.3 R. Fergus et al., Removing camera shake from a single image, SIGGRAPH 2006 Cho and Lee, Fast motion deblurring, SIGGRAPH Asia 2009 Levin et al., Understanding and evaluating blind deconvolution algorithms,CVPR 2009 Levin et al., Image and depth from a conventional camera with a coded aperture, SIGGRAPH 2007 Levin et al., Motion-Invariant Photography, SIGGRAPH 2008 Nagahara et al., Flexible depth of field photography, ECCV 2008 |
| May 3 | Deep Neural Networks (slides) Assg3 in, Assg4 out: Deep Low-light Image Enhancement |
Szeliski, Chapter 5.1-5.3 |
| May 8, May 10 | Deep Neural Networks (cont'd.) (slides) Project progress reports due |
Szeliski, Chapter 5.4 |
| May 15, May 17 | Deep Generative Models (slides) Assg4 in, Assg5 out: Text-driven Image Manipulation |
Szeliski, Chapter 5.5.4 Goodfellow, Generative Adversarial Networks, NeurIPS 2016 Tutorial |
| May 22, May 24 | Visual quality assessment | Ma and Fanf, Image Quality Assessment in the Modern Age, ACM MM 2021 tutorial Sheikh et al., A statistical evaluation of recent full reference image quality assessment algorithms, IEEE TIP 2006 Ding et al., Comparison of full-reference image quality assessment models for optimization of image processing systemsIJCV 2021 Wang and Bovik, Reduced-and no-reference image quality assessment, IEEE SPM 2011 Talebi and Milanfar, NIMA: Neural Image Assessment, IEEE TIP 2018 |
| TBA | Midterm Exam | |
| May 29, May 31 | Advanced topics, Course wrap-up Assg5 in |
|
| June 12, June 14 | Project presentations Final project reports due |
Resources
Related Conferences
- IEEE International Conference on Computer Vision (ICCV)
- European Conference on Computer Vision (ECCV)
- IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- SIGGRAPH
- SIGGRAPH Asia
- IEEE International Conference on Computational Photography (ICCP)
- Advances in Neural Information Processing Systems (NeurIPS)
- International Conference on Learning Representations (ICLR)
Reference Journals
- ACM Transactions on Graphics (ACM TOG)
- IEEE Transactions on Image Processing (IEEE TIP)
- IEEE Transactions on Multimedia (IEEE TMM)
- IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI)
- International Journal of Computer Vision (IJCV)
- Computer Vision and Image Understanding (CVIU)
- Image and Vision Computing (IMAVIS)
Python Resources
- Python/numpy Tutorial.
- scikit-learn: Machine learning in Python
Linear Algebra
- A Geometric Review of Linear Algebra, by Eero Simoncelli
- An Introduction to Linear Algebra in Parallel Distributed Processing, by M.I. Jordan
Resources for scientific writing and talks
- Notes on writing, by Fredo Durand
- How to write a great research paper, by Simon Peyton Jones (video)
- Small Guide To Giving Presentations, by Markus Püschel
- Giving an effective presentation: Using Powerpoint and structuring a scientific talk, by Susan McConnell (video)
- Writing papers and giving talks, by Bill Freeman (notes)