Immersive In Situ Visualizations for Monitoring Architectural-Scale Multiuser MR Experiences
Zhongyuan Yu, Daniel Zeidler, Krishnan Chandran, Lars Engeln, Kelsang Mende, Matthew McGinity · 2024
Read Immersive In Situ Visualizations for Monitoring Architectural-Scale Multiuser MR Experiences by Zhongyuan Yu, Daniel Zeidler, Krishnan Chandran, Lars Engeln, Kelsang Mende, Matthew McGinity (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15918.txt.
Introduction
Zhongyuan Yu and co-authors give the hosts of shared mixed-reality events a way to see what their visitors are experiencing in a 2024 paper. Co-located multiuser mixed reality fills whole buildings with people who walk naturally inside one shared virtual world. Exhibitions, presentations, and experiments all ride on it. The host's problem is blindness: engagement, device health, and system hiccups unfold invisibly.
Their answer is immersive in situ visualization, dashboards that live inside the experience itself rather than on a side monitor. Real-time streams about visitor engagement, device performance, and system events become visible in place, where the host can act on them without leaving the world being hosted. The design grows out of the team's own hard lessons delivering such events at architectural scale, where a struggling visitor or a dying headset can quietly sink the show.
Catalog record
| CREATOR | Zhongyuan Yu, Daniel Zeidler, Krishnan Chandran, Lars Engeln, Kelsang Mende, Matthew McGinity |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15918
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Immersive In Situ Visualizations for Monitoring Architectural-Scale Multiuser MR Experiences Zhongyuan Yu, Daniel Zeidler, Krishnan Chandran, Lars Engeln, Kelsang Mende, Matthew McGinity Abstract Mixed reality (MR) environments provide great value in displaying 3D virtual content. Systems facilitating co-located multiuser MR (Co-MUMR) experiences allow multiple users to co-present in a shared immersive virtual environment with natural locomotion. They can be used to support a broad spectrum of applications such as immersive presentations, public exhibitions, psychological experiments, etc. However, based on our experiences in delivering Co-MUMR experiences in large architectures and our reflections, we noticed that the crucial challenge for hosts to ensure the visitors’ quality of experience is their lack of insight into the real-time information regarding visitor engagement, device performance, and system events. This work facilitates the display of such information by introducing immersive in situ visualizations. Keywords Information Visualization, In Situ Visualization, Multiuser Mixed Reality 1 Introduction Over recent years, multiuser MR (MUMR) systems, in which groups of people share a single, coherent world composed of virtual and real elements, have been shown to hold great promise, with applications spanning education, gaming, and data analysis. By enabling direct perception of both the physical world and fellow users, MUMR fosters collective, collaborative behavior, and contributes to overcoming isolation experienced with pure VR Guo et al. 2024 . Recently, low-cost mobile HMDs with video see-through capabilities, such as Meta Quest 3s, allow for seamless and flexible adjustment in the integration between real and virtual environments across the entire reality-virtuality continuum Milgram et al. 1995 . Besides, with the emergence of robust markerless inside-out tracking and on-device computing, such devices can now be effectively used in large spaces without depending on any external tracking or computing infrastructure. Hence MUMR systems built on such devices become highly scalable, allowing visitors to explore expansive virtual scenarios by naturally walking through large architectural spaces, making them ideally suited for creating immersive presentations. We use the term Co-MUMR to refer to Co-Located Multiuser Mixed Reality experiences. Further, we define a Co-MUMR experience to be “architectural scale” when it enables viewers to navigate freely within spaces that exceed the dimensions of typical room-sized environments and involves structures such as hallways, staircases, mezzanines, elevators, escalators, etc Schier et al. 2023 . In our lab, we have developed and deployed many Architectural-scale Co-MUMR experiences, in artistic, educational, museum, research studies or prototypes, gaming, and psychotherapy domains, gaining many hundreds of hours of experience with hundreds of subjects. In all cases, we observe roles are typically differentiated into “host” and “visitors”. Depending on the context, the “host” may be a tour guide, teacher, instructor, moderator, experiment coordinator, or support technician, while the “visitors” may be museum or exhibition visitors, trainees or students, or trial subjects, for example. The “host” typically has experience with the system, and is responsible for ensuring a smooth experience for everyone involved. Sometimes multiple hosts are active, or the roles change dynamically. Very often, the hosts themselves are wearing an HMD, because they have an active role as guiding experts in the experience. In this case, they face the challenge of performing their “in-experience” diegetic role (e.g. teacher), while simultaneously performing the role of host. However, even in cases where the host has no active diegetic role, our experience demonstrates that the performing of the duties of a host can be greatly enhanced when the host themselves is wearing an HMD. In this paper, we therefore explore the possibilities and investigate the usefulness of in situ mixed reality visualizations for hosting architectural-scale Co-MUMR experiences. We facilitate the display of real-time visitor and device information with immersive in situ visualizations, empowering hosts with direct information access. The visualized information serves as an initial step towards enabling informed decision-making, assisting in resolving technical issues, or potentially offering targeted assistance and explanations to visitors during the experience. Our main contributions are: (1) The design of immersive in situ visualizations for monitoring visitor engagement and system performance towards aiding hosts in enhancing visitor experiences. (2) The implementation of the proposed in situ visualizations in an operational system. Figure 1: Exemplary immersive in situ visualizations. (A) A sketch of a collection of in situ visualizations helping hosts access real-time information regarding the multiuser MR experience. (B) Visitor engagement visualization displays their rendered view by side. (C) Embedded visualization of real-time device metrics, including rendering framerate and battery status. (D) Visualization of past calibration events with a color-coded circular-shaped chart around a calibration station. 2 Related Work A few approaches have investigated in situ visualizations of visitors’ engagement, such as view directions, directly in the immersive environment. For instance, Wang et al. proposed a system to enhance transparency and integrative transition between the realities of HMD users sharing the same physical environment Wang et al. 2020 . It allows guests to observe all other HMD users’ interactions contextualized in their own MR environment. However, this approach primarily aims at enabling guests to perceive each other’s perspectives without distinguishing between hosts and visitors and it is not specifically tailored to address the challenge of understanding visitor engagement in large architectural spaces. Thanyadit et al. developed a system to enable instructors to observe and guide students in the context of VR classrooms Thanyadit et al. 2019 . They proposed immersive visualizations to enhance the instructor’s ability to monitor students’ activities and focus. However, the system primarily caters to educational settings where the virtual environment for each visitor is consistently synchronized, which is considered as a “guided” experience, visualizations to offer insights into visitor engagement in an “explorative” setting are not included in their work. 3 Information Expectations and Design Considerations Figure 2: Photos and screenshots of our previously organized and hosted Co-MUMR experiences. 3.1 Co-MUMR Hosting Experiences Throughout the past years, our lab has developed and exhibited many Co-MUMR experiences, in a variety of public and research domains, ranging from public exhibitions in museums to multi-user educational experiences and large-scale MUMR scientific studies in psychology and psychotherapy (as shown in Fig. 2 ). Collectively, this constitutes many hundreds of hours of hands-on MUMR experience with many hundreds of visitors. We present four such cases here: E1: Hybrid Spaces. In February 2023, we developed and hosted a Co-MUMR experience that blended real and virtual elements, transforming a 30m x 20m space into an immersive multi-layered parkour environment. This explorative experience spanned four days, with 180 visitors, each on average staying 30 minutes in MR (see Fig. 2 (A)). The hosts were responsible for introducing the concept to participants, aiding the tracking alignment, and resolving issues for unexpected disruptions. Hosts did not wear MR headsets themselves. E2: Virtual Leap of Faith. Around the same period, we conducted a second Co-MUMR experience, augmenting a four-story building with a large space (20m x 30m) by superimposing it with an artistic version of the space. This Co-MUMR experience spanned one day, engaging around 100 visitors. Each session accommodated a minimum of two participants, managed by three individuals (see Fig. 2 (B)). E3: Future Mobility. In mid-2023, we organized an additional Co-MUMR experience in the foyer and on the terrace of the second floor (around 15m x 5m each), with a focus on future climate-neutral mobility concepts. This one-day event