While Mark Zuckerberg continually espouses his metaverse vision for an immersive social world, researchers at Facebook Reality Labs (FRL) have been working on a prototype to ensure “social co-presence.” Once you put on a virtual reality (VR) headset you are partially disconnected from those physically around you because your eyes can’t be seen, so the reverse passthrough VR project aims to solve that.

Facebook has made great strides with its Oculus Quest platform and the Passthrough+ feature, instantly allowing a VR user to see the outside world and interact with it. This is being further pushed by the experimental Passthrough API allowing developers to play with mixed reality (MR) features. But while you can see and talk to someone in front of you whilst inside VR, most importantly seeing their facial responses, the same can’t be said the other way around.

So a team led by research scientist Nathan Matsuda has been developing reverse passthrough VR over the past couple of years, the latest prototype an imposing, cable-laden device that displays eyes in 3D, a stepping stone to what could be a more natural way to talk to some whilst they’re in VR. Rather than seeing a user’s actual eyes, the system displays a 3D model of the user’s eyes with eye-tracking data making the model move accordingly. The team initially started with a 2D screen that produced an unnatural effect so they moved onto a 3D light field display.

As demonstrated in the above demo, on the far right this provided depth and allowed facial features to appear in the correct position no matter how the user turned their head. This worked for any number of people in the same room. The central image showcases what it looks like on a 2D display.

When initially presented to FRL Chief Scientist Michael Abrash he wasn’t convinced at its usefulness: “My first reaction was that it was kind of a goofy idea, a novelty at best,” said Abrash. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things, and that’s a good thing, because now it’s clearly a unique idea with genuine promise.”

The current 3D modelling might look a bit rough but FRL’s Codec Avatars research could provide a more effective display solution, with real-time expressions that behave and look natural. An example can be seen below on the reverse passthrough prototype.

As with all early prototypes, the reverse passthrough VR system may never be more than a research project as Facebook looks to advance its VR and AR vision. For further updates on FRL’s latest tech, keep reading VRFocus.

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://www.vrfocus.com/2021/08/facebook-research-reveals-reverse-passthrough-vr-prototype/