A new Facebook experiment reveals how sound could play a key role in augmented reality. Chief scientist Michael Abrash and his team at FRL Research (formerly Facebook Reality Labs) published information today about what the group calls “ perceptual superpowers ” — AR devices that find out what you're trying to hear, then enhance it and dampen background noise. In conjunction with spatial audio features, the interface produces an aural equivalent of a hologram overlay in a pair of glasses.
For years, Facebook has been pursuing high-quality virtual sound, primarily through its Oculus virtual reality headsets. FRL Research's new work focuses on applications for AR. For example, "imagine being able to hold a conversation in a crowded restaurant or bar without having to lift your voice to be understood or strained to understand what others are saying," the company explains.
AR glasses could do this by collecting audio with microphones, using contextual clues to gage which sounds are significant, and feeding those sounds through a noise-canceling earpiece. Conversely, if you're on a phone or conference call, enhanced spatial sound might project participants' voices or other audio to specific areas of the room, enhancing the feeling that you're actually with someone else—or "audio presence" in FRL Research's words.
As Facebook admits, the "perceptual superpowers" pitch of the lab is somewhat close to the features of current hearing aids, enhancing sound and reducing background noise. (In fact, one experimental device uses brain implants to concentrate on individual voices.)
However, AR glasses provide unique opportunities for people with and without hearing difficulties. Directional sensors and outward-facing or eye-tracking cameras may collect detailed contextual information, including your body orientation and where your gaze is focused. This allows the earpieces to find an "acoustic spotlight" to intensify. And, of course, the AR system can combine its sound landscape with the amplified audio.
Facebook may also supplement the scheme with other AR initiatives. LiveMaps, for example, aims to create rich maps full of knowledge about people's surroundings. If LiveMaps detects that someone is wearing glasses in a restaurant, the glasses could automatically flag and cancel the clinking of the silverware sounds.
Facebook posted a photo of a prototype in-ear monitor on the dummy's head, as well as an image of team members wearing headphones that look more like Valve's off-ear Index speakers. The Oculus Quest VR headset uses guided speakers, a comfortable one-size-fits-all solution for projecting sound into the environment. Nevertheless, leakage sound to bystanders — while in-ear monitors may be better suited for adjusting overall sound levels while wearing AR glasses in public. "What type factor to use to solve a problem essentially depends on the application," says FRL Research Audio Chief Ravish Mehra.
Other companies have experimented with an aural increase — most notably Bose, which unveiled audio-only AR glasses in 2018. Bose's glasses used space sensors to sense where the wearers were looking and deliver features such as virtual street tours. However, the company struggled to make them economically viable, and this June, it abandoned the project.
Unlike in previous years, Facebook doesn't hold conferences to show off new technologies in 2020, so we haven't heard the audio in person. FRL Research provided reporters with a demonstration of their filtering mechanism on a remote call. And any spatial audio enhancements could soon be added to VR headsets, including a reported redesign of Oculus Quest during this month's Facebook Connect meeting. "The work we're doing and the work we've discussed today certainly have applications for our VR line of products," Mehra says.
The complete implementation of these capabilities could be years away. We don't know anything about Facebook's User AR Glass Roadmap, even though we're increasingly learning more about what the organization feels they can do. A few pictures display plastic lenses, but they are designed to carry the microphone array, not to function as full-length glasses. Facebook has previously said that several prototype variants have been developed, including a recent combined AR / VR headset concept that looks like a pair of sunglasses.
To market a real commercial product, Facebook would need to allay fears that go beyond technological considerations. This involves persuading people that "perceptual superpowers" do not pose a danger to privacy.
Without limitations, sufficiently robust microphones and filtering technologies could allow people to listen to conversations in public spaces imperceptibly. AI audio analysis poses far stranger and more disturbing possibilities, such as the ability to flag individual voices or conversation keywords in a crowded room. (It may also potentially perform more harmless, beneficial tasks such as real-time translation.) And, of course, the glasses will also record your conversations.
There is no suggestion that FRL Research is contemplating intentional surveillance. But many people still fear — probably incorrectly — that Facebook apps will record them to target advertising. That is why the mere possibility could scare consumers. FRL Research says that the audio recorded is encrypted and stored on servers that a limited number of researchers can access.
Abrash says that Facebook is trying to create privacy on all aspects of its glass architecture, repeating its overall AR / VR plans. Among other items, the device might ask for permission from someone else's glasses before amplifying a conversation, or it might have a restricted range."Instead of thinking of this as a magic flashlight, we can point to someone and hear what they're saying, think about it more like people can take part in the discussion they're in any way," he told reporters.
And for now, Abrash says, the tiny microphone arrays aren't powerful enough to handle long-range espionage. "If you ever see someone walking with glasses that are two feet wide," he warned, "you should be suspicious."