Dolby Laboratories Hosts Northwestern Team at ‘Future of Cinema’ Summit

Dolby Laboratories welcomed a multidisciplinary team of Northwestern researchers to its headquarters in San Francisco in January to imagine the future of cinema.

A leader in immersive entertainment experiences, Dolby Laboratories has partnered with faculty from Northwestern Engineering’s Department of Computer Science and Northwestern’s School of Communication to provide gift funding for seed projects that could advance the state-of-the-art in audio, visual, and voice technologies.

“This is the first time that Dolby has worked closely with Northwestern in such a critical area for our business,” said Brett Crockett, vice president of sound technology research and development at Dolby. “With that, the researchers from Northwestern leveraged incredible insight and deep expertise that provided us with a new perspective on the many opportunities we have in front of us to significantly impact the future of cinema for consumers across the world.”

Following a prior workshop in January 2023 in California, a delegation from Dolby Laboratories visited the Evanston campus in May to continue building relationships, converging around shared research priorities, and sparking potential collaboration opportunities.

During this most recent summit, scientists at Dolby Laboratories described innovations in cinema sound and imaging and new approaches to push the frontier of the cinematic experience. Northwestern faculty members shared their investigations in fields including audio accessibility in gaming, computational imaging, event cinema, generative music modeling, neurocinematics, and speech audio watermarking.

“Visiting Dolby Laboratories was an exhilarating experience,” said Samir Khuller, Peter and Adrienne Barris Chair of Computer Science at the McCormick School of Engineering. “Dolby is an epicenter of amazing entertainment experiences. Rigorous research underpins the development of their products and they imagine the possibilities of new technology every single day.”

Bio-Inspired Computational Imaging

With the support of Dolby Laboratories, Northwestern Engineering’s Emma Alexander and her Bio Inspired Vision Lab team are tackling challenges in the domain of nontraditional color sensing.

Alexander, assistant professor of computer science and (by courtesy) electrical and computer engineering at the Northwestern Engineering, and computer science PhD student M. Kerem Aydin presented their joint work with Qi Guo (Purdue University) to develop a colorization algorithm framework that reconstructs a hyperspectral image from a grayscale guide image and spatially sparse spectral clues. Their paper “HyperColorization: Propagating Spatially Sparse Noisy Spectral Clues for Reconstructing Hyperspectral Images” was published in Optics Express in March. The team’s method reduces compute time and the impact of shot noise while enhancing robustness by incorporating guided sampling, edge-aware filtering and dimensionality estimation techniques — surpassing previous algorithms in various performance metrics.

“We hope to use this work to develop a bio-inspired camera that will computationally recover color information from a colorblind camera, by shifting where light lands rather than blocking it out,” Alexander said. “This improved light efficiency will enable color imaging in very dim and also very fast-moving scenes.”

At the summit, Alexander also discussed her lab’s blending of techniques from the physical, computational, and biological sciences in the development of 3D cameras built to mimic jumping spiders' efficient and unusual eyes and motion sensing principles from the larval zebrafish.

Generative modeling of music and speech audio watermarking

Bryan Pardo, professor of computer science at Northwestern Engineering, presented two recent projects developed by his team in the Interactive Audio Lab.

VampNet is a masked acoustic token modeling approach to audio generation built by computer science PhD student Hugo Flores García, Pardo, and two collaborators currently at Adobe Research — Prem Seetharaman (PhD ’19) and Rithesh Kumar.

A powerful music co-creation tool, sound artists can apply VampNet to tasks including music compression, inpainting, outpainting, continuation, and looping with variation while maintaining a specific style, genre, and instrumentation.

Pardo also discussed MaskMark, a neural network-based digital audio watermarking technique optimized for speech. Designed to help combat issues of misinformation and fraudulent impersonation in the domain of high-quality speech synthesis models, the tool embeds a traceable signature in generated audio.

The paper “MaskMark: Robust Neural Watermarking for Real and Synthetic Speech” — developed by Pardo, computer science doctoral student Patrick O'Reilly, and Zeyu Jin and Jiaqi Su (Adobe Research) — was accepted at the IEEE International Conference on Acoustics, Speech and Signal Processing, held April 14 -19 in Seoul, South Korea.

Northwestern contributors to the summit also included the following faculty members:

• Bharath Chandrasekaran, Ralph and Jean Sundin Endowed Professor and chair of communication sciences and disorders at Northwestern’s School of Communication
• Derrick Fields, assistant professor of instruction in radio, television, and film at Northwestern’s School of Communication
• Manling Li, incoming assistant professor of computer science at Northwestern Engineering
• Stephan Moore, professor of instruction in radio, television, and film at Northwestern’s School of Communication
• Jacob Smith, professor and associate chair of radio, television, film in the School of Communication and director of the MA in SAI graduate program

“Working with Dolby has been an amazing experience for our faculty, and we hope to host the Dolby team visit back to Northwestern in early fall,” Khuller said.