Dual Multimedia Installations Address Sonification of Endangered Coral Reefs

Article Content

In spring 2017, UC San Diego music professor and former Qualcomm Institute (QI) composer in residence Lei Liang, and Falko Kuester, the institute’s professor of visualization and virtual reality, organized a unique seminar course on “Hearing Seascapes: A Collaborative Seminar on the Sonification of Coral Reefs.” It provided graduate students, primarily from the Music department, with an opportunity to develop multimedia projects to highlight the dangers facing coral reefs in many parts of the world.

Out of that seminar course emerged two art installations accepted into the Qualcomm Institute’s Initiative for Digital Exploration of Arts & Sciences (IDEAS) 2017-2018 performance season. The two works will premiere simultaneously in QI’s Atkinson Hall on the UC San Diego campus. The immersive works include:

• Erasure, an ambitious multimedia installation produced by a robust collaboration among three UC San Diego Music Ph.D. students: Jacob Sundstrom in Computer Music, Fiona Digney in Music Performance, and Anthony Vine in Musical Composition, together with Computer Science and Engineering Ph.D. student Vid Petrovic; and
• Hearing Seascapes, which combines coral-reef imagery and audio data to generate sound based on the location and viewpoints of endangered coral reefs. The work was co-developed by Lauren Jones, a Music graduate student in Vocal Performance, and Music Ph.D. student Eunjeong Stella Ko, both at UC San Diego.

Hearing Seascapes will be staged in the SunCAVE virtual-reality (VR) room in the Immersive Visualization Lab, and Erasure in the Reconfigurable Media Lab, both on the first floor of QI's Atkinson Hall. The works will premiere Thursday, February 8, 2018 from 5-7 p.m., followed by a public reception. Starting at 5:30pm, Music professor Lei Liang will introduce a series of short talks by the artists behind both works.

“Over the last several decades, coral-reef ecosystems have suffered significant impacts from both local and global factors,” said Ko. “We aim to convey important messaging to the audience and illuminate data that show the declining health of ocean coral reefs.”

ERASURE

Through an interconnected network of three-dimensional (3D) photomosaic models of coral reefs and spatially as well as electronically processed percussion sounds, a metaphorical ecosystem forms and responds directly to human presence and the temporal history of that presence throughout the work’s existence.

Erasure responds negatively to human presence in the installation environment. As more and more people enter the space, the installation begins to break down: the sonic tapestry of percussion sounds contort and particulate, the synthetic biome of coral visualizations begin to morph into unnatural forms, and the entire system mutates – all with the presence of the audience – and members of that audience are aesthetically confronted with their impact on these remote and fragile ecosystems.  The transformation, however, is neither irreversible nor unidirectional. “As viewers leave, the piece rebounds, but more slowly than the rate at which it broke down,” explained Ph.D. student Sundstrom. “The system bounces back from the immediate and long-term human impact, and it reflects the resilience of the reefs to withstand and adapt to global shifts in climate and the ecosystem.”

The visual component of Erasure consists of 3D photomosaic models of coral reefs taken from the 100 Island Challenge, based in the Scripps Institution of Oceanography. These digital reproductions were created by the Challenge’s technical visualization advisors from the Qualcomm Institute’s Cultural Heritage Engineering Initiative (CHEI), jointly with Scripps Oceanography. By rigorously photographing and collecting data from reef sites and rendering the data into 3D computer models using custom software developed by CHEI’s Falko Kuester and Vid Petrovic, students from the Music department were able to observe the reefs from various angles, light levels, and distances.

“These coral reef constructions ebb and flow between their natural state—meticulously constructed synthetic ecosystems—and transformative states: from granulations of the stony corals and polyps into whirling cascades of particles, to fissions of vast reef colonies into splintered slabs that recede in and out of focus,” said  Music’s Anthony Vine. “Lingering traces of the piece in its untouched state float among the remains, and the metaphorical ecosystem appears to be dreaming and longing to return to an undisturbed state.”

The sound-space of Erasure is created from a reservoir of percussion improvisations that both reflect the sounds that might be found in and around a coral-reef environment, as well as poetic expansions that reach beyond the palette of oceanic utterances: scraped and struck limestone tiles, sweeping washes of hands streaked across a bass drum, and the murky drones of rolled bell plates. By manipulating these samples in a simple causal network, emergent behavior materializes to constitute a lush atmosphere of sound. In this way, the behavior of the sound world is not unlike the behavior found in reef ecology: masses of small units combining to create a complex and rich environment.

HEARING SEASCAPES

“Music can bring an image to life, and by giving a voice to the coral reefs, we can help the audience make an emotional connection to ecology and realize the fragility of these reefs,” said Lauren Jones. “The goal of this project is to create an immersive experience for the viewer that allows them to submerge themselves in the world of the coral.”

Viewers use a joystick to ‘dive’ and explore the reef, controlling the location, viewpoint, depth and speed of navigation. Audience members hear different sounds that represent different species in the data set. Each species has its own specific personality (represented by different sounds). “Coral reefs are living, breathing organisms that are vulnerable to small changes of the surrounding environment and climate,” explained Stella Ko. “We assign the coral reefs a distinct personality by examining certain characteristics such as texture, habit, origin and growth. To convey messages through music, we’ve recorded an underwater dialogue of voices.” Indeed, the installation was designed to induce conversation between coral reefs and a fish maneuvering through the reefs, and the sound will become louder or softer depending on how far away it is from the coral.

Jones and Ko set for themselves three goals with Hearing Seascapes: to display experiments with different aspects of sound and innovative graphic design to create an enjoyable environment for the audiences; to tell an effective and interactive story invoking concepts of adventure, imagination and humor to motivate people to recognize environment health; and to create an inviting seascape with a synergy of voices, images, synthesized sounds and human emotion.

“Based on the notion of acoustic ecology, we want to bring out the positive aspects of sound in the ocean environment,” they noted. “We hope to highlight the importance of engaging in the soundscapes of coral reefs in hopes that our musical voice can make scientific results more accessible to society.”