Bringing Texture to Your Flat Touchscreen

What if the touchscreen of your smartphone or tablet could touch you back? What if touch was as integrated into our ubiquitous technology as sight and sound?

Northwestern University and Carnegie Mellon University researchers now report a fascinating discovery that provides insight into how the brain makes sense of data from fingers.

In a study of people drawing their fingers over a flat surface that has two “virtual bumps,” the research team is the first to find that, under certain circumstances, the subjects feel only one bump when there really are two. Better yet, the researchers can explain why the brain comes to this conclusion.

Their new mathematical model and experimental results on “haptic illusions” could one day lead to flat-screen displays featuring active touch-back technology, such as making your touchscreen’s keyboard actually feel like a keyboard. Tactile information also could benefit the blind, users of dashboard technology in cars, players of video games and more.

“Touch is so important in our real world, but it is neglected in the digital world,” said J. Edward Colgate, an expert in touch-based (haptic) systems. He is the Allen and Johnnie Breed University Professor of Design at Northwestern’s McCormick School of Engineering and Applied Science. “We want to create something that will make touch a reality for people interacting with their screens, and this work is a step in that direction.”

Forces felt by the fingers as they travel along a flat surface can lead to the illusion that the surface actually contains bumps. This so-called “virtual bump illusion” is well known in the haptics field, Colgate said, and the researchers were able to make use of it.

“By leveraging the virtual bump illusion, we were able to design a meaningful experiment that shed light on the way the brain integrates information from multiple fingers,” Colgate said. “Our big finding was ‘collapse ‘ -- the idea that separate bumps felt in separate fingers are nonetheless experienced as one bump if their separation happens to match that of the fingers.”

The study, which will be published the week of Feb. 9 by the Proceedings of the National Academy of Sciences (PNAS), is about how the brain makes sense of data from the fingers.

Colgate, the paper’s corresponding author, and longtime Northwestern haptics collaborator Michael A. Peshkin joined forces with Carnegie Mellon’s Roberta Klatzky to work on filling the digital world’s functional gap by enabling flat screens to engage the haptic perceptual system. This is known as “surface haptic” technology.

The research team’s experiment presented two virtual bumps, with the distance between them varying across trials, to subjects participating in the study. When bump and finger spacing were identical, subjects reported feeling two bumps as one. In this case, the brain thinks it is too coincidental that there should be two bumps at the same time, so it registers the bumps as one.

“How does your body and mind interpret something flat and ‘see’ it as having shape and texture?” said Klatzky, a world-renowned expert in cognitive psychology and haptic perception. “An important step toward effective surface haptics is to understand what kinds of stimulation might lead you to feel something other than uniform flatness when you touch the surface of your device. Our study contributes to this understanding.”

Klatzky is the Charles J. Queenan Jr. Professor of Psychology and Human-Computer Interaction at Carnegie Mellon.

“Our findings will help us and other researchers figure out how to design haptic technology to produce certain tactile effects,” said Peshkin, a professor of mechanical engineering at the McCormick School. “Haptics -- giving a feel to objects -- just enhances the physicality of a person’s experience.”

Steven G. Manuel, the study’s first author and a Northwestern alumnus, developed the model of where the “illusion of protrusion” comes from. It describes how the brain constructs a mental depiction of the surface using sensory signals from two fingers as they explore a surface over time and space.

A critical feature of the model, and one found in theories of perception more generally, is that it assumes the brain is biased toward inferring causes rather than registering coincidences. In essence, as the fingers encounter forces while they explore a flat surface, the brain creates virtual bumpiness that is most consistent with the physical bumps that would produce the same sensations.

The work was supported by a National Science Foundation Division of Information and Intelligent Systems grant, Surface Haptics via Tractive Forces (IIS0854100). 

The title of the paper is “A Coincidence-Avoidance Principle in Surface Haptic Interpretation.” Manuel, Colgate, Peshkin and Klatzky are authors of the paper.