Professor Jack Tumblin Retires After 22 Years at Northwestern

John E. (Jack) Tumblin is retiring as an associate professor of computer science on August 31 after 22 years at Northwestern. He joined the department as an assistant professor in September 2001.

A staunch proponent of innovating cameras for use as measuring devices, Tumblin’s research currently centers around combining computers, cameras, and displays to help museum curators, conservators, and archivists better preserve, study, and understand their collections.

"I remember very vividly Jack's first talk here as a faculty candidate. We'd seen a lot of great candidates in graphics, all working, basically, on geometry of the underlying objects in the scene,” said Larry Birnbaum, professor of computer science at Northwestern Engineering. “Jack gave a completely different kind of talk, focusing on photometry — in other words, on light. We all thought, ‘this person is doing something completely different, we should hire him!’ This unique way of thinking made Jack one of the true pioneers of computational photography."

Tumblin characterized the last six months of his appointment as a “working sabbatical” to tie up loose ends on some long-standing passion projects.

“Some old-fashioned backwaters have been overlooked in computer vision and cameras,” Tumblin said. “Our cameras are getting steadily smaller, cheaper, and better. Camera sensors are exquisitely precise in pixel spacing and layout, and displays are another set of fixed grids of phenomenally precisely placed elements. It's always grated on me that these things are not commonly and widely used as measuring instruments.”

Ahead of his retirement, Tumblin is revisiting multiple research questions related to the application of cameras and displays as measuring instruments, including the use of repeatable photographs and machine learning to extract information from and detect minute changes in an artifact.

“Imagine if we could take a photograph now, and then, 100 years from now, we take that exact same photograph. If it's perfectly aligned, you can take the difference and amplify it mathematically, and it'll show you every little change that ever happened,” Tumblin said. “A museum could simply run a program to warn of changes larger than a certain threshold in collected items that are slowly deteriorating despite our best conservation efforts.”

One of the early members of the Center for Scientific Studies in the Arts (NU-ACCESS), Tumblin has collaborated extensively on projects with former faculty members Oliver Cossairt, Mark Walton, and Florian Willomitzer, with strong support from NU-ACCESS co-director Francesca Casadio, associate vice president and Grainger Executive Director of Conservation and Science at the Art Institute of Chicago; co-director Aggelos Katsaggelos, Joseph Cummings Professor of Electrical and Computer Engineering and (by courtesy) computer science at Northwestern Engineering; and Kenneth Shull, professor of materials science and engineering at Northwestern Engineering. The NU-ACCESS teams have also partnered with the Rijksmuseum in Amsterdam; the Georgia O’Keeffe Museum in Santa Fe, New Mexico; and the MIT Museum in Boston.

Contrast microscope

One ‘loose end’ research interest Tumblin is revisiting is the development of a so-called ‘contrast microscope’ to capture subtle, very-low contrast features.

“The human eye and current solid-state cameras can’t detect light changes smaller than about one percent, in part because digital sensors are corrupted by three different ‘flavors’ of noise,” Tumblin said. “If we can set up the camera to keep it and the lighting absolutely steady, and then take a few thousand digital photos, then we can combine them to detect changes finer than the quantization levels of the camera by about a factor of 10.”

Tumblin aims to achieve contrast thresholds of 0.1 percent or smaller.

The contrast microscope method might enable some other oddly novel observations, including measuring changes in interreflections when occluded objects move, ‘un-erasing’ a white-board, or predicting fall foliage colors.

“Chlorophyll levels in each tree leaf will change throughout the day; if we capture those changes in photos, we can extrapolate their zero-chlorophyll state to predict peak autumn colors,” Tumblin said.

In the sphere of museum conservation, Tumblin hopes to deploy a small contrast microscope unobtrusively to monitor items vulnerable to damage by light exposure. Detecting changes well before human eyes can see them could provide a “fading alarm.”

Next-generation calibration

To overcome a primary challenge in using cameras for geometric measurements, Tumblin seeks to improve on the Camera Calibration Toolbox for Matlab developed more than 20 years ago by Jean-Yves Bouguet at the California Institute of Technology and Zhengyou Zhang at Microsoft Research.

The original toolkit uses a printed checkerboard pattern and software to aid a photographer in measuring where the camera is positioned in 3D space, where it's aiming, and what distortions are being applied by the lens systems.

“It's easy and good, but not all that accurate unless you are really accurate in the setup. Now we can do far better,” Tumblin said.

Tumblin proposes replacing the printed checkerboard with a uniform, modern AMOLED display and computing a closed-loop calibration to iteratively amplify the errors and obtain at least another order of magnitude increase in accuracy.

“Even though that old code is cherished and widely acclaimed, it seems no one is bothering to try and make something better, and I think we're about 15 years overdue to do that,” Tumblin said.

Particularly for the task of examining and depicting the inside of amphorae and other pottery, Tumblin hopes to apply this next generation calibration software to paired cameras, long rigid sticks, and small displays.

“Ideally, a conservator could take a long stick with a small camera on one end and a phone-like display at the other, and thread the camera through the vessel’s openings to explore and photograph its interior,” Tumblin said. “While exploring, a fixed second camera captures both the vessel’s exterior and the display at the long stick’s other end. Together, the exterior camera and display let us compute the 3D pose of the interior camera. Transforming interior viewpoints to match the exterior camera then lets us create accurate cutaway pictures to see everything we photographed inside.”

Computational photography

Tumblin is known for his work to popularize computational photography, which aims to extend digital equipment well beyond the limits of ordinary film-like cameras.

“Jack Tumblin is extremely creative and an intrepid explorer, who got into computational photography in its infancy when digital photography was beginning to take off in the mid 90s,” said Samir Khuller, Peter and Adrienne Barris Chair of Computer Science at the McCormick School of Engineering.

“Interactive computing and digital optical devices used in new ways can break old established limits on temporal, spatial, and spectral capture performance,” Tumblin said. “It can capture many more useful descriptors of the 3D scene we wish to capture.”

Tumblin is also known for his work in HDR (High Dynamic Range) tone mapping and light field capture, a technique that imagines a two-dimensional sphere around an object. If a camera takes a picture of that object looking inwards from every point on the sphere, each pixel would be measuring a tiny cone of light coming off the surface and defining its appearance as a 4D collection of rays.

“By controlling the 4D rays that illuminate the object, we can measure complex reflectance and all optical effects of glass and translucent materials,” Tumblin said. “The full 8D transfer function — or the ratio of each 4D incoming ray to each 4D outgoing ray — completely describes the object’s visual appearance. The full 8D function is too vast and boring to capture directly; what subset will be the most useful?”

Tumblin frequently collaborated with Ramesh Raskar (previously at Mitsubishi Electric Research Laboratories and now at the MIT Media Lab) and colleagues to advance computational photography. Tumblin met Raskar in 2002, shortly after joining the then twelve-member faculty team at Northwestern Computer Science. Raskar and Tumblin’s collaborations drew attention to coded aperture methods, including a motion-blur decoding method popularized as “flutter-shutter,” the use of variable binary aperture masks to create invertible point-spread functions, an ‘agile spectrum’ camera method that enables cameras to adapt the spectrum of their color primaries, and a way to achieve light-field capture within cameras by simple masks that induce a novel heterodyning effect.

“One of the things I'm most proud of is popularizing this idea of computational photography and doing new things in odd ways,” Tumblin said. “I like to come up with crazy ideas, including wildly impractical ones, just to shake loose people's thinking about how to solve problems.”

Over the course of his career, Tumblin has mentored approximately 60 undergraduate students and 51 master’s students. He collaborated with many PhD students, serving as primary adviser for Amy Ashurst Gooch (PhD '06), Ankit Mohan (PhD '08), Xiang Huang (PhD '13), and Paul Olczak (PhD '15). Bingjie (Jenny) Xu, Tumblin’s final advisee, defended her thesis last month and plans to start a position with a research group at Apple.

“When I joined Northwestern, Jack said he wanted to help mentor graduate students,” Khuller said. “This illustrated his true nature to want to be helpful to students and see them succeed. What else can a graduate student hope for?”

Prior to joining Northwestern, Tumblin completed a post-doctoral position in computer graphics at Cornell University, advised by Donald Greenberg. Previously, he was a research engineer and consultant at the IVEX Corporation, leading the development of a Level-C FAA-certifiable computer graphics image generator for flight simulation. Tumblin was awarded five of the 11 total patents he holds while working with IVEX. He was also a television engineer at KTLA-TV5 in Los Angeles after a one-year stint in graduate school at the University of Southern California School of Cinematic Arts.

Tumblin earned a PhD in computer science advised by Greg Turk and Jessica Hodgins
(1999), a master’s degree in electrical engineering (1990), and a bachelor’s degree in electrical engineering (1978), all from the Georgia Institute of Technology.

Tumblin will re-join the department as an adjunct faculty member in winter 2024 to teach the COMP_SCI 351-1: Intro to Computer Graphics course, which focuses on core principles and mathematics behind computer-assisted picture-making. Students learn to program interactive 2D or 3D shapes and implement them in browsers using OpenGL.

The Department of Computer Science is hosting a retirement celebration honoring Tumblin on May 31 in the Norris Center.