GIST: Group Interaction Sensing Toolkit for Mixed Reality

Diana Romero, Yasra Chandio, Fatima M. Anwar, Salma Elmalaki · 2025

Read GIST: Group Interaction Sensing Toolkit for Mixed Reality by Diana Romero, Yasra Chandio, Fatima M. Anwar, Salma Elmalaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11797.txt.

Introduction

Diana Romero, Yasra Chandio, Fatima Anwar, and Salma Elmalaki wrote a 2025 paper on a sensing toolkit that reads group behavior inside mixed reality headsets. When teams work in immersive spaces, collaboration can break down silently. Existing analysis leans on outside observers and hand labeling, which miss moment to moment dynamics.

Their pipeline passively captures gaze, audio, and position from ordinary headsets, with no external gear. One module builds social network maps of who interacts with whom, both per session and within it. A second uses unsupervised clustering to find behavioral phases between pairs as they unfold. A trial with 48 people sorting objects together showed within session structure that session averages erase. The authors offer the toolkit for studying teamwork where it happens.

Catalog record

CREATORDiana Romero, Yasra Chandio, Fatima M. Anwar, Salma Elmalaki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-11797

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GIST: Group Interaction Sensing Toolkit for Mixed Reality Diana Romero, Yasra Chandio, Fatima M. Anwar, Salma Elmalaki Abstract . Understanding how teams coordinate, share work, and negotiate roles in immersive environments is critical for designing effective mixed-reality (MR) applications that support real-time collaboration. However, existing methods either rely on external cameras and offline annotation or focus narrowly on single modalities, limiting their validity and applicability. To address this, we present a novel group interaction sensing toolkit ( GIST ), a deployable system that passively captures multi-modal interaction data, such as speech, gaze, and spatial proximity from commodity MR headset’s sensors and automatically derives both overall static interaction networks and dynamic moment-by-moment behavior patterns . We evaluate GIST with a human subject study with 48 participants across 12 four-person groups performing an open-ended image-sorting task in MR. Our analysis shows strong alignment between the identified behavior modes and shifts in interaction network structure, confirming that momentary changes in speech, gaze, and proximity data are observable through the sensor data. 1. Introduction Collaborative Mixed Reality (MR) applications are transforming fields such as surgical planning and training ( Gerup et al. 2020 ) , shared 3D architectural walkthroughs ( Du et al. 2016 ) , remote industrial equipment maintenance guidance ( Gonzalez-Franco et al. 2017 ) , and co-creative product prototyping ( Cascini et al. 2020 ) , each relying on seamless coordination among multiple users (in real time). However, despite these rich, group-based scenarios, we still lack scalable, real-time methods to observe and analyze how groups interact in immersive technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) 1 1 1 VR immerses users in a digital world, AR adds digital information onto the real world, and MR blends the two, allowing digital and physical objects to interact in real-time. . Effective collaboration depends on participants’ communication patterns, decision-making, turn-taking, social norm setting, and authority negotiation ( Leavitt 1951 ; Kerr and Tindale 2004 ; Lapinski and Rimal 2005 ; Gençer 2019 ) , all of which vary with group size, composition and task complexity ( Hackman and Vidmar 1970 ; Halfhill et al. 2005 ; Higgs et al. 2005 ) and ultimately shape productivity, creativity and performance ( Stogdill 1972 ; Paulus 2000 ; Podsakoff et al. 1997 ) . While traditional research has examined group behavior in physical settings, the emergence of immersive technologies (AR/VR/MR) offers new frontiers for collaborative work. MR, in particular, seamlessly integrates digital objects into the physical environment, enabling interaction as if they were physically present. This seamless integration creates collaborative scenarios distinct from VR (fully immersive) or AR (digital overlay), introducing novel social and spatial variables such as spatial awareness ( Luo et al. 2022 ; McGee et al. 2024 ) , presence ( Irlitti et al. 2023 ; Tran et al. 2024 ) , and multimodal communication ( Szczurek et al. 2023 ; Mathis et al. 2024 ) . Moreover, inter-, intra-, and multi-user variability ( Zhao et al. 2024 ; Elmalaki 2021 ) demands methods that can capture and interpret these complex, evolving interactions. Previous research has significantly advanced our understanding of collaboration through various methods, including VR/AR studies ( Zhang et al. 2018 ; Chaffin et al. 2017 ; Olguin and Gloor 2009 ) , sociometric badges ( Kim et al. 2012 ) , and external techniques such as motion capture cameras, intrusive wearable sensors, or manual video analysis ( Zhang et al. 2022 ) . However, these approaches cannot be directly applied to the on-device environment of Mixed Reality (MR) headsets ( Ens et al. 2019 ) . MR uniquely blends physical and virtual worlds, leading to challenges such as occlusions and spatial constraints. In this context, the aforementioned methods are unable to capture crucial in situ cues such as gaze, speech, and motion, which are essential to understand group behavior ( Bai et al. 2020 ) . In this paper, we address the need for a deeper understanding of group behavior in MR by answering the following research questions ( RQ ): How can the sensory systems in MR headsets effectively capture group behavior during collaborative tasks? (RQ1) and What algorithms can process and interpret the data to infer group behavior? (RQ2) . To answer these, we introduce Group Interaction Sensing Toolkit ( GIST ) , the first end-to-end framework that passively senses, aggregates, and clusters multimodal signals from commodity MR headsets to infer group behavior. Using embedded headset sensors, we present GIST and make four main contributions: (1) Passive Multimodal Sensing Pipeline. We capture and preprocess raw gaze, binaural audio, and motion data using embedded sensors in commodity MR headsets, preserving natural interactions without extra hardware. (2) Sociogram Aggregation Module. We construct instantaneous interaction graphs (sociograms) to capture graph-based aggregation of group behavior using domain-informed thresholds to compute structural metrics of group behavior. (3) Temporal Clustering Module. We implement unsupervised time-series clustering of dyadic interactions to reveal evolving behavioral phases, without manual annotation. (4) Empirical Validation. We deployed GIST prototype in a 48-participant (12-group) unconstrained image-sorting study. Our evaluation demonstrates the system’s stability, interpretability, and ability to uncover meaningful group-level insights through structural and temporal analysis. 2. Related Work 2.1. Group Behavior Sensing in MR/VR Decades of VR research have revealed how virtual contexts shape social presence, interpersonal dynamics, and collaboration ( Bailenson et al. 2008 ; Hauber et al. 2006 ; Gatica-Perez et al. 2005 ) , with proteus effects and other frameworks ( Tajfel and Turner 2004 ) , explaining the emergence of group norms in digital spaces. MR adds further complexity by merging real and virtual worlds, introducing novel factors such as altered spatial awareness ( Luo et al. 2022 ) , shifts in presence ( Irlitti et al. 2023 ) , multimodal communication channels ( Szczurek et al. 2023 ) , and even changing basic behaviors such as eye contact compared to face-to-face settings ( Prytz et al. 2010 ) . But most MR studies focus on individual-user tasks, designing overlays for walking ( Lages and Bowman 2019 ) , biking ( Kosch et al. 2022 ; Matviienko et al. 2022 ) or driving ( Stefanidi et al. 2024 ; Rusch et al. 2013 ) rather than multi-user collaboration. Prior work in immersive environments has utilized multi-modal sensors to instrument group interactions, but often for offline or lab-based analysis. TeamSense, for example, used badges to log speaking time and proximity for VR team cohesion ( Zhang et al. 2018 ) , gaze and controller motion have characterized turn-taking in small VR groups ( Chaffin et al. 2017 ; Olguin and Gloor 2009 ) , and MR prototypes have recorded head pose and hand gestures under lab cameras for task coordination ( Ens et al. 2019 ) . These approaches yield static, offline snapshots and require extra hardware or manual annotation, leaving a gap in fully automated, in-situ inference of evolving group behavior using only the sensors built into MR headsets. 2.2. Social Signal Processing & Sociometric tools Ubiquitous computing has shown that simple audio, location, and motion cues can reveal rich social dynamics in co-located groups. Sociometric Badges tracked speaking patterns, turn-taking, and interpersonal distance to predict team cohesion and performance

Read the complete text