Course Description
Digitization of reality is at the cusp of widespread adoption; it also embodies a unique convergence of techniques that span the gamut of electrical and computer engineering. This course introduces students to augmented/virtual/mixed reality systems, or what is broadly referred to as extended reality (XR). The course covers hardware and software topics central to XR systems, namely, content capture, streaming, and display. The course also aims to make students comfortable with some of the basic tools required to digitize, optimize, render, control, and visualize content.
In addition to lectures spanning the eco-system of topics central to XR systems, the course will feature labs where students will implement techniques to digitize 3D content using RGBD cameras, importing and manipulating digital content in modern XR development workflows, building controllers with 6 DOF, and finally building a 3D display. By the end of the course, all of these components will be connected to complete the cycle between reality to digital reality and back.
The course will have one project that will require implementing and demonstrating a feature on an XR display. The project will have multiple deliverables, including a mid-semester presentation that will involve a deep dive into related technical articles.
Course Goal
At the culmination of the course, the student should be able to understand of gamut of techniques underlying XR systems, including different flavors of XR systems that are in vogue. The course will provide hands-on experience in building different components of an XR system, and the ability to critically analyze the tradeoffs in each.
Selected Final Project
Recitation and Office Hours
Recitation: Fridays 11:00 am - 12:20 pm ET, WEH 5409
Office hours will be held during the semester at the following times:
- Mondays 4:30 - 5:30 pm, Anthony, CIC 2312
- Mondays 10:00 - 11:30 am, Aswin, PH B24
- Thursdays 4:00 - 5:00 pm, Tao, CIC 2300
- Tuesdays 4:00 - 5:00 pm, Edward, CIC 2300
Pre-requisites
18-290 Signals and Systems OR 18-213 Computer Systems
Class Schedule
| Date | Topics | Homework |
|---|---|---|
| Tu, Aug 29 | Introduction to XR | |
| Th, Aug 31 | XR Landscape | |
| Tu, Sept 5 | Human Perception I | |
| Th, Sept 7 | Human Perception II | Lab 1 Checkpoint 1 Due |
| Tu, Sept 12 | XR Tools | |
| Th, Sept 14 | Software Tools and Real-time Rendering | Lab 1 Final Due |
| Tu, Sept 19 | Stereoscopic Displays | |
| Th, Sept 21 | Integral Displays | |
| Tu, Sept 26 | Focal Planes - Varifocal / Multifocal Systems | Lab 2 Due |
| Th, Sept 28 | Optical Tools and Holographic Displays | |
| Tu, Oct 3 | Exotic Displays | |
| Th, Oct 5 | Localization and Tracking I | Lab 3 Due |
| Tu, Oct 10 | Localization and Tracking II | |
| Th, Oct 12 | Basics of 3D Scanning | |
| Tu, Oct 17 | Fall Break No Class | |
| Th, Oct 19 | Fall Break No Class | |
| Tu, Oct 24 | Appearance / BRDFs / Light Stages | |
| Th, Oct 26 | Scene Reconstruction I | Lab 4 Due |
| Tu, Oct 31 | Scene Reconstruction II | Final Project Out |
| Th, Nov 2 | Guest Lecture (Larry Shea, CMU School of Drama) | |
| Tu, Nov 7 | Democracy Day No Class | |
| Th, Nov 9 | Digitizing the Body - Human Capture | |
| Tu, Nov 14 | Guest Lecture (David Lindlbauer, CMU HCII) | |
| Th, Nov 16 | Collaborative Virtual Environments | |
| Tu, Nov 21 | Guest Lecture (Mallesh Dasari, NEU) | |
| Th, Nov 23 | Thanksgiving No Class | |
| Tu, Nov 28 | Guest Lecture (Doug Lanman, Meta) | |
| Th, Nov 30 | Frontiers of XR | |
| Tu, Dec 5 | Final Project Presentation | Final Project Due |
| Th, Dec 7 | Final Project Presentation |
Grading
The final grade is consisted of:- Four two-week lab assignments (40%)
- Experience session participation (5%)
- Weekly in-class quizzes (15%)
- Final project (40%)
Labs
The labs comprises of:- Lab 1: Digitization and View in XR
- Lab 2: Unity and Programming XR
- Lab 3: Porting and 3D Displays
- Lab 4: Localization and Tracking
Experience Sessions
Experience sessions lets you to get familiar with various state-of-the-art technologies and demos and will be sporadically scheduled during the semester.- Virtual Reality Headsets
- Augmented Reality Headsets
- Experimental Displays
- Tracking and SLAM