Connect with us

AR/VR

Facebook Researchers Develop Bleeding-edge Facial Reconstruction Tech So You Can Make Goofy Faces in VR

Avatar

Published

on


Facebook Reality Labs, the company’s R&D division, has been leading the charge on making virtual reality avatars realistic enough to cross the dreaded ‘uncanney valley’. New research from the group aims to support novel facial expressions so that your friends will accurately see your silly faces VR.

Most avatars used in virtual reality today are more cartoon than human, largely as a way to avoid the ‘uncanny valley’ problem—where more ‘realistic’ avatars become increasingly visually off-putting as they get near, but not near enough, to how a human actually looks and moves.

The Predecessor: Codec Avatars

The ‘Codec Avatar’ project at Facebook Reality Labs aims to cross the uncanny valley by using a combination of machine learning and computer vision to create hyper-realistic representations of users. By training the system to understand what a person’s face looks like and then tasking it with recreating that look based on inputs from cameras inside of a VR headset, the project has demonstrated some truly impressive results.

Recreating typical facial poses with enough accuracy to be convincing is already a challenge, but then there’s a myriad of edge-cases to deal with, any of which can throw the whole system off and dive the avatar right back into the uncanny valley.

The big challenge, Facebook researchers say, is that it’s “impractical to have a uniform sample of all possible [facial] expressions” because there’s simply so many different ways that one can contort their face. Ultimately this means there’s a gap in the system’s example data, leaving it confused when it sees something new.

The Successor: Modular Codec Avatars

Image courtesy Facebook Reality Labs

Researchers Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh from the University of Toronto, Vector Institute, and Facebook Reality Labs, propose a solution in a newly published research paper titled Expressive Telepresence via Modular Codec Avatars.

While the original Codec Avatar system looks to match an entire facial expression from its dataset to the input that it sees, the Modular Codec Avatar system divides the task by individual facial features—like each eye and the mouth—allowing it to synthesize the most accurate pose by fusing the best match from several different poses in its knowledge.

In Modular Codec Avatars, a modular encoder first extracts information inside each single headset-mounted camera view. This is followed by a modular synthesizer that estimates a full face expression along with its blending weights from the information extracted within the same modular branch. Finally, multiple estimated 3D faces are aggregated from different modules and blended together to form the final face output.

The goal is to improve the range of expressions that can be accurately represented without needing to feed the system more training data. You could say that the Modular Codec Avatar system is designed to be better at making inferences about what a face should look like compared to the original Codec Avatar system which relied more on direct comparison.

The Challenge of Representing Goofy Faces

One of the major benefits of this approach is improving the system’s ability to recreate novel facial expressions which it wasn’t trained against in the first place—like when people intentionally contort their faces in ways which are funny specifically because people don’t normally make such faces. The researchers called out this particular benefit in their paper, saying that “making funny expressions is part of social interaction. The Modular Codec Avatar model can naturally better facilitate this task due to stronger expressiveness.”

They tested this by making ‘artificial’ funny faces by randomly shuffling face features from completely different poses (ie: left eye from {pose A}, right eye from {pose B}, and mouth from {pose C}) and looked to see if the system could produce realistic results given the unexpectedly dissimilar feature input.

Image courtesy Facebook Reality Labs

“It can be seen [in the figure above] that Modular Codec Avatars produce natural flexible expressions, even though such expressions have never been seen holistically in the training set,” the researchers say.

As the ultimate challenge for this aspect of the system, I’d love to see its attempt at recreating the incredible facial contortions of Jim Carrey.

Eye Amplification

Beyond making funny faces, the researchers found that the Modular Codec Avatar system can also improve facial realism by negating the difference in eye-pose that is inherent with wearing a headset.

In practical VR telepresence, we observe users often do not open their eyes to the full natural extend. This maybe due to muscle pressure from the headset wearing, and display light sources near the eyes. We introduce an eye amplification control knob to address this issue.

This allows the system to subtly modify the eyes to be closer to how they would actually look if the user wasn’t wearing a headset.

Image courtesy Facebook Reality Labs

– – – – –

While the idea of recreating faces by fusing together features from disparate pieces of example data isn’t itself entirely new, the researchers say that “instead of using linear or shallow features on the 3D mesh [like prior methods], our modules take place in latent spaces learned by deep neural networks. This enables capturing of complex non-linear effects, and producing facial animation with a new level of realism.”

The approach is also an effort to make this kind of avatar representation a bit more practical. The training data necessary to achieve good results with Codec Avatars requires first capturing the real user’s face across many complex facial poses. Modular Codec Avatars achieve similar results with greater expressiveness on less training data.

It’ll still be a while before anyone without access to a face-scanning lightstage will be able to be represented so accurately in VR, but with continued progress it seems plausible that one day users could capture their own face model quickly and easily through a smartphone app and then upload it as the basis for an avatar which crosses the uncanny valley.

Source: https://www.roadtovr.com/facebook-reality-labs-modular-codec-avatar-research-goofy-face-vr/

AR/VR

Microsoft to Acquire Bethesda’s Parent Company ZeniMax for $7.5B

Avatar

Published

on


Microsoft today announced that it’s entered into an agreement to acquire ZeniMax Media, the parent company of game studio Bethesda Softworks. According to TechCrunch, the price was set at $7.5 billion.

The acquisition is slated to include all of ZeniMax’s properties including Bethesda Game Studios, id Software, ZeniMax Online Studios, Arkane, MachineGames, Tango Gameworks, Alpha Dog, and Roundhouse Studios.

Once approved, this will make Microsoft the owners of some of the most influential titles in modern gaming, including franchises such as The Elder Scrolls, Fallout, Wolfenstein, DOOM, Dishonored, Prey, Quake, and Starfield.

ZeniMax’s fleet of studios have also been responsible for a number of VR titles such as DOOM VFR, Fallout 4 VR, Wolfenstein: Cyberpilot, and Skyrim VR.

Image courtesy Bethesda Softworks

Microsoft says in a blog post that the inclusion of ZeniMax’s studios will help continue the company’s commitment “to deliver a breadth of amazing games to discover and play on Xbox.”

Although Microsoft hasn’t said as much, it’s likely Bethesda will be narrowing its focus to develop their long-standing franchises for Xbox—an expensive weapon in the coming battle with Sony’s PlayStation 5.

“One of the things that has me most excited is seeing the roadmap with Bethesda’s future games, some announced and many unannounced, to Xbox console and PC including Starfield, the highly anticipated, new space epic currently in development by Bethesda Game Studios,” says Phil Spencer, head of Xbox.

Although it’s still unclear how the acquisition will affect Bethesda’s future VR aspirations—Microsoft still has no clear plan for VR headset support on its Xbox platform—Oculus’ Consulting CTO John Carmack says the acquisition may be a good thing for him at least.

Carmack co-founded id Software and was lead programmer of titles such as Wolfenstein 3D, Doom, and Quake. Ever since ZeniMax and Facebook’s Oculus were engaged in a lengthy legal battle over the alleged theft of intellectual property developed by Carmack back when he was employed by id Software, he was understandably unable to publicly engage with the games he developed.

Source: https://www.roadtovr.com/microsoft-acquire-bethesdas-zenimax-7-5-billion/

Continue Reading

AR/VR

Fallout & Doom Creator Bethesda is Being Acquired by Microsoft

Avatar

Published

on

Microsoft is in the habit of buying developers to bring under its Xbox brand, doing so in grand fashion today by announcing the acquisition of Zenimax Media and its videogame publisher Bethesda Softworks, which is behind big IP’s like Fallout, DOOM, Skyrim, Wolfenstein and many more.

DOOM VFR screenshot
Doom VFR

The agreement sees Microsoft purchase Zenimax Media for $7.5 billion USD (£5.85bn GBP) in cash. This will mean Microsoft has 23 studio teams under its banner – up from 15 – with Bethesda’s franchises now guaranteed to be added to Xbox Game Pass. Microsoft has stated plans to bring: “Bethesda’s future games into Xbox Game Pass the same day they launch on Xbox or PC.”

This may make quite the difference to fans of Bethesda’s franchises planning their next console purchase, with both Xbox Series X and PlayStation 5 on their way this holiday season. “Generations of gamers have been captivated by the renowned franchises in the Bethesda portfolio and will continue to be so for years to come as part of Xbox,”  said Phil Spencer, executive vice president, Gaming at Microsoft in a statement.

So what does this mean for virtual reality (VR) gamers? Bethesda has supported the VR community with a range of ports and original content over the years, from Fallout 4 VR to Wolfenstein: Cyberpilot. At this stage, there’s no reason to believe that support for VR will stop, even if Xbox Series X doesn’t support the technology. Microsoft previously acquired inXile Entertainment in 2018 and by the end of 2020, the studio will be releasing multiplayer shooter Frostpoint VR: Proving Grounds.

Wolfenstein: Cyberpilot
Wolfenstein: Cyberpilot

There is a greater possibility going forward that Bethesda and its subsidiaries won’t be releasing any content for PlayStation 5 and therefore future PlayStation VR headsets. But this won’t be happening right away, as this type of process isn’t quick with Microsoft expecting the acquisition to be completed in the second half of 2021. “We’re still working on the same games we were yesterday, made by the same studios we’ve worked with for years, and those games will be published by us,” adds SVP of PR & Marketing Pete Hines in a blog posting.

The announcement continues a selection of acquisitions affecting the VR gaming landscape, with Koch Media purchasing Vertigo Games this month, the team behind Arizona SunshineSkyworld, Ghost Patrol VR and A Fisherman’s Tale. Whilst Facebook picked up Ready at Dawn, the creator of Lone Echo and Echo VR, currently working on Lone Echo 2.

VRFocus will continue its coverage of Microsoft’s acquisition of Zenimax Media, reporting back with further updates.

Source: https://www.vrfocus.com/2020/09/fallout-doom-creator-bethesda-is-being-acquired-by-microsoft/

Continue Reading

AR/VR

Play Puzzle Title Tsuro: The Game of the Path on Oculus Quest in October

Avatar

Published

on

Tabletop board game Tsuro: The Game of the Path arrived for Oculus Rift last year thanks to Thunderbox Entertainment, offering a tactile yet zen-like experience. Today, the developer has announced the videogame is coming to Oculus Quest next month.

Tsuro: The Game of the Path

Playable solo or as a multiplayer, Tsuro: The Game of the Path provides a simple gameplay mechanic, place tiles on a board to move your stone, keeping it in play for as long as possible. As the number of tiles increase so do the chances of your stone following another player’s path which could very well lead to your stone leaving the table and ending your game.

Tsuro: The Game of the Path is set in a tranquil Japanese garden with animals wandering through and the idyllic sounds of nature in the air. So you can take a stroll whilst the other players take their turns – the videogame supports up to 8 across platforms – or to get a better viewpoint of the evolving board, teleporting up to a roof for example.

The aim of the puzzle title is to make sure your stone counter is the last on the board, not an easy task when their are many paths for it to follow.

Tsuro: The Game of the Path

Thunderbox Entertainment will release Tsuro: The Game of the Path for Oculus Quest on the 23rd October, priced at £7.99 GBP/$9.99 USD.

October is set to be a big month for Oculus Quest content thanks to the launch of Oculus Quest 2 on 13th. Title confirmed during Facebook Connect last week include The Walking Dead: Saints & Sinners from Skydance Interactive, Resolution Games’ BlastonLittle Witch Academia: VR Broom Racing and Enhance Games’ Rez Infinite. As further VR videogames are announced, VRFocus will keep you updated. 

Source: https://www.vrfocus.com/2020/09/play-puzzler-title-tsuro-the-game-of-the-path-on-oculus-quest-in-october/

Continue Reading
Big Data4 hours ago

New Poll: What Python IDE / Editor you used the most in 2020?

Big Data6 hours ago

Machine Learning from Scratch: Free Online Textbook

Big Data8 hours ago

Facebook vows to restrict users if U.S. election descends into chaos: FT

Automotive8 hours ago

The Potential of Predictive Analytics in Labor Industries

Big Data8 hours ago

TikTok proposes social media coalition to curb harmful content

Big Data8 hours ago

Tesla warns on challenges of scaling up production

Big Data8 hours ago

TikTok’s promise of 25,000 new U.S. jobs sets lofty goal

Blockchain9 hours ago

Crypto.com Blocks Withdrawals After Alleging Users Attempted Illegitimate Trades

Blockchain10 hours ago

DeFi Tokens Take a Beating Amid Cryptocurrency Market Hemorrhaging

Blockchain10 hours ago

7 Ways to Build Your Brand with Blockchain Marketing

Blockchain10 hours ago

Litecoin short-term Price Analysis: 22 September

Forex10 hours ago

The SEC made the award public without naming the whistleblower

Forex10 hours ago

New Zealand regulator imposes new guidelines on CLSA’s local license

Blockchain10 hours ago

Non-Fungible Token Craze Explodes Despite DeFi Market Woes

Blockchain11 hours ago

Bitcoin Gets Knocked Back After Fib Rejection

CNBC12 hours ago

BlizzCon will return as an online-only event in February 2021

Blockchain12 hours ago

Bitcoin Cash, VeChain, Dogecoin Price Analysis: 22 September

Coinpedia13 hours ago

YAM Finance Attempts A Comeback: YAMv2 Price Plunges

Techcrunch13 hours ago

Europe’s Point Nine outs new ~€100M fund to back early-stage SaaS and digital marketplaces

Blockchain13 hours ago

ECB President Christine Lagarde Says Digital Euro Could Complement Cash

CNBC13 hours ago

‘The Dark Overlord’ hacking group member sentenced to five years in prison

Blockchain14 hours ago

Phil Anderson Accepts Crypto Donations For His Assembly Campaign

Fintech14 hours ago

5 Key Highlights From Huawei’s Developer Conference

Automotive15 hours ago

China’s electric carmaker WM Motor pulls in $1.47 billion Series D

CNBC15 hours ago

Royole’s FlexPai 2 5G foldable phone costs under $1,500

Fintech17 hours ago

CoinJar builds world-class cryptocurrency exchange on AWS

Fintech17 hours ago

Gen Z Versus Millennials – Who Fared Better at Saving Money and Budgeting?

Fintech18 hours ago

New $280 million ASX tech share listing Wednesday

CNBC18 hours ago

Microsoft keeps the same price for its new wireless Xbox controllers

CNBC20 hours ago

Elon Musk warns that Tesla’s ‘Battery Day’ tech is two years away

Big Data20 hours ago

PayBito Experiences Record Sign-ups Post Referral Program Launch in…

Techcrunch21 hours ago

Impossible Foods nabs some Canadian fast food franchises as it expands in North America

Blockchain21 hours ago

U.S SEC, OCC issue first regulatory guidance for stablecoins 

Techcrunch22 hours ago

Daily Crunch: This TikTok deal is pretty confusing

Fintech22 hours ago

Australian FinTech company profile #102 – mx51

Fintech23 hours ago

Nucleus Wealth and Arrow Financial Advice join forces to offer Robo white label advice

Techcrunch23 hours ago

CDC removes updated guidelines around COVID-19 aerosol transmission, but this expert explains why it should reverse the reversal

Crunchbase23 hours ago

Illumina Buys Out Big Investors With $8B Acquisition Of Grail

Fintech24 hours ago

Neobank Volt, LAB3, and Microsoft partner to build “Volt 2.0”

Visual Capitalist1 day ago

29 Psychological Tricks To Make You Buy More

Trending