UC San Diego Researchers Collaborate to Fool the Eye and the Mind through Touch

Article Content

Concept circuitry layout of flexible nano-material. Illustrates target zones of interest. Photo courtesy of Jon Paden

UC San Diego researchers are using a Chancellor’s Interdisciplinary Collaboratories Fellowship to explore Augmented Reality. Department of Visual Arts Professor Sheldon Brown and M.F.A. candidate Jon Paden are partnering with scholars in the departments of Neuroscience, Computer Science and Engineering, Electrical and Computer Engineering, and NanoEngineering on a project entitled, “Tromper soi-même: Fooling the senses with new techniques of combined visual and afferent Augmented Reality rendering of sight and touch.”

Augmented Reality places virtual 3D objects and virtual 2D graphics within the real world through holographic imaging techniques, blurring the lines between what is real and what is enhanced with computers. For instance, according to Forbes, AR and other gaming technology has made its way into the world’s museums. At UC San Diego, researchers inspired by the visual illusion “trompe l’oeil,” or “fooling the eye,” are bringing together computer-based visual rendering techniques with nano-material physical sensation to take this notion a step further—“Tromper soi-même”—to fool the mind. Their AR research explores the implications and boundaries of communicating through touch.

“Our research brings new sensations in haptics (relating to the sense of touch) to bear with Augmented Reality headsets to change perception of the real,” explains Paden. “The end goal is an embedded system that allows the virtual world to interact with you in a blended reality. A device of experiences in new somatic (relating to the body) fidelity has potential far-reaching social, psychological and medical applications.”

According to Brown, the John D. and Catherine T. MacArthur Foundation Endowed Chair in Digital Media and Learning and director of the Arthur C. Clarke Center for Human Imagination, this field is gaining traction. “There are several researchers on campus investigating meaning-making with new display and interface,” he said, clarifying that there are limitations of the kinds of meaning-making that can occur. “Jon Paden has been making connections between various researchers on campus that are working on different aspects of this problem, and this project brings some of those threads together.”

Paden, a media artist and student of Brown’s, notes the deliberate engagement of artists with STEM researchers on this project. “We hope to bring divergent fields together at the onset to offer a broader spectrum of use.”

As an artist, Paden’s complex art practice merges new technology with traditional media to investigate memory and contextual relationships in time. Therefore, he offers a unique research contribution and opportunity to develop new creative relationships. “This collaboration, when successful, will give additional fidelity of material to my cause by allowing me to include more somatic relationships to information as material for artistic creation,” said Paden.

Visualization of proposal goal. Creating a virtual butterfly and trying to convince the user it is real based on replicating the force created by the butterfly’s movement, while rendering the virtual insect with ambient lighting conditions. Photo courtesy of Jon Paden 

The project is also fitting for a department on the cutting edge of art and technology. UC San Diego Visual Arts, celebrating 50 years at the university, was on the forefront of integrating computational and media practices in the curriculum. The department houses both an Interdisciplinary Computing and the Arts undergraduate major and a new undergraduate major in Speculative Design. Cross-pollination with engineering is also prevalent through visual arts engagement in the Structural and Materials Engineering Building and efforts like the EnVision Arts and Engineering Maker Studio.

“Tromper soi-même: Fooling the senses with new techniques of combined visual and afferent Augmented Reality rendering of sight and touch” includes the following faculty participants: Brown, visual arts; Jurgen P. Schulze, computer science and engineering; and Darren J. Lipomi, nanoengineering. Graduate student participants include: Charles Dickey, neuroscience; Kamran Alipour, computer science and engineering; Cody Carpenter, nanoengineering; Paden, visual arts; and Jared Pham, electrical and computer engineering.

UC San Diego’s Department of Visual Arts is ranked 13th in the nation for fine arts, according to U.S. News & World Report, with high-ranking specialties in multimedia and sculpture. The Division of Arts and Humanities jumped 10 points this year to #23, according to U.S. News & World Report’s 2017 Best Global Universities rankings.