Making Sound Decisions For Immersive Experiences

At SXSW, all eyes were on AI, ML, AR and VR. Our ears, however, were more likely checking out the many bands jamming in the music festival showcases. At a time when hearables have risen to prominence at CES, it’s a little curious how few Interactive sessions were dedicated to the sense of hearing and audio innovation.

As anyone sporting a SXSW Film badge will tell you, audio is crucial to filmmaking. They may even go so far as to quote George Lucas, who once opined, “I feel that sound is half the experience.” He should know. Kids don’t run around shouting “bang bang” anymore but rather, “pew pew pew.”

The same Lucas-quoting filmmaker will also tell you that the emotional impact of sound is huge in any film. In many panels, the word “emotion” was tossed around like a beach ball at a concert. “Design emotionally intelligent machines.” “Create emotional bonds with customers.” “Shopping decisions are emotional.” Yet, most VR panels focused on visual and tech elements including 3D modeling and photogrammetry, and not the emotional connections sound makes with listeners.

Throughout a panel dedicated to VR interface design, not one word was uttered about the use of audio, until an audience member questioned the panel on it. The three panelists excitedly lit up. Timoni West  (Unity Labs) said, “Sound is more important than other element in VR. If you close your eyes you should know where you are.” Isaac Cohen (Cabbibo) followed that with, “Sound is a great way to make emotion.” And Ken Perlin (NYU Media Research Lab) tied both thoughts together with: “Sounds is more fundamental to our emotional engagement than vision.“

In a panel that focused on UX and storytelling, Luxloop’s cofounder Mandy Mandelstein took a welcome detour into elaborating on the power of audio. Both a filmmaker and visual interactive artist, Mandy believes that “narrative has always been immersive.” She also believes that technology shouldn’t get in the way of telling a story. Feeling encumbered by VR headsets, she set out to create a modern storytelling experience in the form of an audio narrative for the Minneapolis institute of Art.

The project is called Overheard, and you essentially get to follow and eavesdrop on virtual characters you never see, and hear their stories as you make your way around the cultural institution. Museum goers ended up visiting different parts of the museum they never would have thought to explore before.

There’s more going on in the aural space than just the innovative use of headphones. At another panel, the belief that “hearables will be embedded in our lives” was confidently expressed. Stuart Karten, president of Karten:Design, a product innovation consultancy, even went further to say that, “that hearables will eventually be hidden in our environment and not things that we wear.” Something to keep in mind when designing interactive/experiential events for retail, entertainment, public art and so on.

Sometimes the sound you hear – the sound that’s creating a playful experience for you – is the one you make: that is, your voice. In a panel on innovation in public spaces, Eric Gradman, cofounder of Two Bit Circus, talked about creating smart, interactive environments that are social, physical and fantastical. One example includes recreating the classic Atari video game inside a 150 square-foot field of play.

The game uses a laser projector to display game elements on the floor. A Kinect sensor tracks your movement as you navigate around the game area on a rollable stool and avoid getting hit by asteroids. The kicker, you blast incoming pieces of space rock by shouting “pew pew pew” at them. (George Lucas will like his royalty check please.)

So, is anybody innovating in the realm of audio for VR experiences instead of treating it as an afterthought? The folks behind the world’s first music video mixed in 3D sound were on hand to explain and demonstrate just how pivotal sound is to the viewer’s journey in the immersive VR space. In a 360 music video for renowned beatboxer, Reeps One, Aurelia Soundworks partnered with The Mill to test how changing environments affect the sonic qualities in a VR experience. The end result was a video that plays around with the concept of where sounds come from.

In the 2D world, audio doesn’t necessarily hold importance to the content consumer. According to Digiday, 85% of Facebook video is watched without sound. But whether you’re creating a physical game installation, a virtual reality experience or any sort of narrative production, it’s important to know that pulling things out of the real world like dialogue, effects, ambient noises and other elements of sound design makes 3D experiences, virtual or otherwise, more magical.