Five days ago, on a train traveling home for Christmas, I was thinking about my personal highlights of 2019. While a lot of good things happened in the past 12 months (and I’m not going to talk about private matters here), from a professional point of view, there’s a clear winner: Giving a talk about mixed reality at the ACM Conference on Human Factors in Computing Systems (a.k.a. CHI) in Glasgow.
The talk was based on research I conducted together with friends from the University of Michigan (where I was a post-doc from 2017‒18), Michael Nebeling and Brian Hall. We had noticed that a lot of people we talked to had differing and partly competing understandings of what mixed reality (or MR) is. For instance, some relied on the original definition by Milgram and Kishino from 1994, which defines MR as a continuum (see below), while others adhered to a newer notion pushed by Microsoft, which also applies to experiences that are clearly VR.
Hence, we concluded that — even though it might seem the question What is Mixed Reality? should have a relatively simple answer — it would be worthwhile to discover and investigate all the different notions of mixed reality that are out there. And we were right, the situation wasn’t as easy as you’d think.
1. How VR could bring transhumanism to the masses
2. How Augmented Reality (AR) is Reshaping the Food Service Industry
3. ExpiCulture — Developing an Original World-Traveling VR Experience
As we hypothesized, there is indeed not a single, “best” definition of mixed reality. Instead, we found six distinct and widely used working definitions:
- MR according to Milgram et al.’s continuum (see above)
- MR as a synonym for AR
- MR as a type of collaboration (interaction between AR and VR users that are potentially physically separated)
- MR as a combination of AR & VR (a system combining distinct AR and VR parts)
- MR as an alignment of environments (e.g., synchronization between a physical and virtual environment)
- MR as a “stronger” version of AR (e.g., HoloLens)
These can be classified based on a conceptual framework (some would call it a taxonomy) with seven dimensions:
- number of environments
- number of users
- level of immersion (e.g., not immersive ‒ partly immersive ‒ fully immersive)
- level of virtuality (e.g., not virtual ‒ partly virtual ‒ fully virtual)
- degree of interaction (e.g., implicit ‒ explicit)
- input (e.g., motion, location)
- output (e.g., visual, audio)
I have also distilled our findings into an infographic:
To discover the six working definitions as well as the seven dimensions of the conceptual framework, we conducted expert interviews that were augmented (clever wordplay, huh?) by an extensive literature review. First, we interviewed a total of ten experts working on augmented and/or virtual reality, from both, academia and industry (occupations ranged from professor to R&D executive to CEO of an AR company). These interviews yielded a preliminary set of four working definitions. Subsequently, we reviewed a total of 68 sources, mainly from the CHI, CHI PLAY, UIST, and ISMAR conferences from 2014‒18 (inclusive). These confirmed the four preliminary notions while we also discovered two more that were added to the set.
Ultimately, we derived the conceptual framework by identifying the minimum number of dimensions that still allowed us to classify all of the working definitions unambiguously.
To give just one example (from our paper), let’s have a look at how Pokémon GO would fit into the conceptual framework. First of all, the viral game constitutes MR according to notion № 4: a combination of AR and VR in a single system.
- It comprises one environment since everything happens on the same device.
- It can be played by one user on that device.
- The level of immersion lies between not immersive and partly immersive.
- The level of virtuality lies between partly virtual (the game’s AR view) and fully virtual(the game’s map view).
- Interaction is implicit (the player moves in the real world, all explicit interaction happens via a HUD).
- It uses the user’s geolocation as input and provides visual and auditory output.
Now, why is this important? Mixed reality is a trending topic. Many people are talking about it nowadays and the number of papers, research artifacts, hardware, and apps is steadily increasing. MR has the potential to become omnipresent in our everyday lives. Therefore, it is important to put one’s words into context. With our research, we hope to provide researchers, students, and professionals with a tool that lets them better communicate what they mean when talking about MR, and to reduce misunderstandings in a rapidly evolving field. We are also proud that our paper received an 🏅 Honorable Mention Award, which stresses the importance of the question at hand.
As further reading, I recommend Milgram and Kishino’s original article about the Reality‒Virtuality Continuum, our own paper from CHI 2019 (of course), as well as my article What is augmented reality, anyway?
PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.
