Pioneering a New Virtual Reality Course

Merging the digital and physical worlds, the technologies of virtual reality (VR) and augmented reality (AR) offer immersive ways of experiencing and interacting with digital content.

Jipeng Sun, a former computer science PhD student at Northwestern Engineering, sees profound potential in the seamless integration of the virtual and the tangible.

“Even though the road is difficult, I truly believe that this technology could replace the smartphones we're currently using one day,” Sun said.

During a 10-week summer session, 12 undergraduate, master’s, and PhD students learned how to build standalone VR hardware and software prototypes from scratch in Sun’s COMP_SCI 396: Advanced Graphics Seminar: VR/AR Systems course.

Designed and taught by Sun, the course is the first of its kind to focus specifically on building standalone VR systems. Unlike PC VR headsets, which require a connection to a computer to function — either wirelessly or tethered by a cable — standalone VR headsets are self-contained systems that include a full operating system and built-in processors, sensors, batteries, storage, memory, and displays. A camera module can also be built into standalone VR systems, allowing them to extend into AR.

Sun developed the course with the expectation that the computer science, computer engineering, and electrical engineering students would have varied — or no – experience with the VR/AR discipline. He was also mindful of the goals of students interested in continuing in academia versus those planning on pursuing industry positions. Motivation was the only course prerequisite.

“The people and things in this beautiful 3D world leave us memories we treasure so much, but with current imaging and display technology, they can only be recorded and revisited in a compromised 2D way. I have a passion about changing this status quo through VR/AR systems,” Sun said. “And I wanted Northwestern students to have the ability to really understand this technology.”

The course reflects the ethos of the maker movement: the best way to understand a system is to build one.

Jack Tumblin, associate professor emeritus of computer science at the McCormick School of Engineering, explained that the recent availability of affordable constructive technology — like the Raspberry Pi single-board computers used in the course — has inspired a renewed interest in hobby electronics.

“One of the things that I find really exciting and gratifying about what Jipeng is doing is that he's really pioneering the concept of taking that newly lowered entry barrier and applying it to state-of-the-art wearable VR/AR headsets,” said Tumblin, who served as the course coordinator and Sun’s co-adviser. “It's a tour de force of optics, visual perception, real-time rendering, communication, inertial measurements, and 3D transformations all put together into one system. This is the sort of course I would have loved as an undergraduate.”

Sun organized the class into small teams, intentionally mixing and matching skill sets so students could collaborate productively to build a sophisticated, complete computing environment stage-by-stage, aided by starter code and modules Sun developed.

One student team included Ruixiang Chai, a computer science PhD student; Ziyang Guo, a PhD student in computer science advised by Jessica Hullman, Ginni Rometty Professor and associate professor of computer science; and Yinmiao Li, a PhD student in computer science and learning sciences, a joint program through Northwestern Engineering and Northwestern’s School of Education and Social Policy.

The team successfully extended the VR prototype with a self-built stereo camera module, which is the fundamental building block for advanced VR/AR interaction methods such as eye tracking and hand tracking. The camera module cost less than $20.

Chai’s research centers around high precision camera calibration methods utilizing accessible consumer cameras and tools. His interest in the AR/VR course was piqued when he saw Sun organizing the breadboard, wiring, embedded computer, and other hardware pieces in their shared office.

“I actually have 3D sickness,” Chai said. “I've never tried wearing a VR system before this course, so this field is very new to me. We touched on quite a lot of topics — graphics, display, optics, machine learning, and embedded systems programming — and that's the essence of this course, providing a platform for students coming from very different backgrounds to learn the basics and fundamentals.”

With a background in human-computer interaction, interactive media, and artistic computing, Li’s research interests involve building new media for educational use.

“VR/AR is a very huge thing in education right now,” Li said. “People are trying to maximize learning experiences with VR and AR, and to be more immersive.”

Three teaching assistants aided the course, including Ashwin Baluja, an undergraduate student pursuing bachelor’s degrees in computer science and mathematics from Weinberg; Jiahui Li, a master’s degree student in computer science at Northwestern Engineering with a diverse background in software engineering, design, education, and product management; and Yiran Zhang (MS ’23), who is currently pursuing a PhD in computer science at Purdue University on VR/AR topics.

“I am passionate about exploring the potential of VR/AR technology to benefit various domains, including learning assessment, product design and development, enhanced situational awareness, and B2C use cases," Li said. "Furthermore, I am enthusiastic about contributing to the development of a supportive learning community that facilitates people who want to venture into the VR/AR field in academia or in the industry.”

Sun acknowledged the support of the Department of Computer Science, in particular Samir Khuller, Peter and Adrienne Barris Chair of Computer Science at Northwestern Engineering, for his mentorship on teaching summer courses; and program assistant Julia Blend for her assistance with procuring components. He is also grateful to Royce Morris, electronic specialist in the Department of Electrical and Computer Engineering, for his lab assistance. Branden Ghena, assistant professor of instruction, and Ilya Mikhelson, associate professor of instruction, provided guidance on embedded systems. In addition, Karl Guttag, the president of KgonTech, presented a guest lecture to the students.

Sun transferred to the Princeton Computational Imaging Lab this month following Tumblin’s retirement. He made the course website publicly available to help guide other students and instructors interested in this field.

“This course is my gift to this place that I really love, to show my passion and respect for this area of research and my love for the Northwestern community,” Sun said.