Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-engaging in Immersive VR Meetings
Geonsun Lee, Yue Yang, Jennifer Healey, Dinesh Manocha · 2025
Read Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-engaging in Immersive VR Meetings by Geonsun Lee, Yue Yang, Jennifer Healey, Dinesh Manocha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16739.txt.
Introduction
Geonsun Lee, Yue Yang, Jennifer Healey, and Dinesh Manocha design a way to rejoin virtual meetings in 2025. People who step away from a live virtual session struggle to catch up. Fixed panels pull the eyes from the group, while brief captions give no past context.
Their tool, EngageSync, pins live text to each avatar and adapts to how engaged the user is. It adds short machine summaries of what was missed. In studies with small and mid-sized groups, users kept more social presence than with fixed panels. They also rejoined faster and recalled more than with plain captions.
Catalog record
| CREATOR | Geonsun Lee, Yue Yang, Jennifer Healey, Dinesh Manocha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-engaging in Immersive VR Meetings Geonsun Lee, Yue Yang, Jennifer Healey, Dinesh Manocha Abstract . Maintaining engagement in immersive meetings is challenging, particularly when users must catch up on missed content after disruptions. While transcription interfaces can help, table-fixed panels have the potential to distract users from the group, diminishing social presence, while avatar-fixed captions fail to provide past context. We present EngageSync, a context-aware avatar-fixed transcription interface that adapts based on user engagement, offering live transcriptions and LLM-generated summaries to enhance catching up while preserving social presence. We implemented a live VR meeting setup for a 12-participant formative study and elicited design considerations. In two user studies with small (3 avatars) and mid-sized (7 avatars) groups, EngageSync significantly improved social presence ( p < .05 p<.05 ) and time spent gazing at others in the group instead of the interface over table-fixed panels. Also, it reduced re-engagement time and increased information recall ( p < .05 p<.05 ) over avatar-fixed interfaces, with stronger effects in mid-sized groups ( p < .01 p<.01 ). Conference: Proceedings of the CHI Conference on Human Factors in Computing Systems; Apr 26–May 1; Yokohama, Japan Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’25), Apr 26–May 1, 2025, Yokohama, Japan ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Collaborative and social computing CCS: Human-centered computing Virtual reality Figure 1. EngageSync is a context-aware transcription panel designed to help users in immersive VR meetings catch up on conversations while maintaining social presence. It operates in two modes: Engagement Mode and Re-engagement Mode . In Engagement Mode, users can view live transcriptions of the speaker or a summary of the previous utterance of a listener by gazing at the person of interest and performing a pinch gesture. In Re-engagement Mode, summaries of conversations that occurred during the user’s absence are displayed for all relevant participants. Once all summaries are read, the user is considered "re-engaged" and caught up with missed context, and the system returns to Engagement Mode. A visual comparison of EngageSync's two operational modes: Engagement Mode and Re-engagement Mode. In Engagement Mode, a user gazes at avatars and performs a pinch gesture to view live transcriptions of the current speaker (default text panel) or a summary of a listener's previous utterance (text panel with a green circle on the bottom right). In Re-engagement Mode, summaries of conversations that occurred during the user's absence are displayed for all relevant participants (text panel with an orange circle on the bottom right). Once all summaries are read, the user is considered "re-engaged" and the system returns to Engagement Mode. The figure illustrates how EngageSync adapts based on user engagement, helping maintain social presence while providing efficient ways to catch up on missed content. 1. Introduction Remote work and virtual collaboration have become integral to modern communication with video-mediated meetings seeing unprecedented growth following the COVID-19 pandemic ( Karl et al., 2022 ; Lee et al., 2022 ) . As the demand for seamless remote interaction continues to grow, researchers and developers have turned their attention not only to video conferencing platforms but also to more immersive technologies like Virtual Reality (VR) and Mixed Reality (MR) ( McVeigh-Schultz and Isbister, 2022 ) . These technologies offer new opportunities to reimagine how we engage in virtual environments (VEs), providing enhanced social interaction and presence that video conferencing alone cannot achieve. ( McVeigh-Schultz and Isbister, 2021 ; McVeigh-Schultz and Isbister, 2022 ; Park et al., 2024 ) While remote meetings offer significant convenience, they also make users more susceptible to distractions, both internal (e.g., multitasking or loss of focus) and external (e.g., notifications, people entering the room) ( Cao et al., 2021 ) . Research in remote video meeting interfaces has made strides in keeping users engaged through meeting summaries, post-meeting transcripts ( Samrose et al., 2021 ) , and more recently, live interactive transcription tools designed to maintain participation during discussions ( Son et al., 2023 ; Chen et al., 2023 ) . These tools not only help users stay on track but also enable them to catch up after disruptions, which can occur frequently in remote work settings. Although immersive VR platforms promise heightened focus and social presence through their immersive nature, users in VR environments are not immune to external distractions. Interruptions such as device notifications, someone physically entering the room, or technical difficulties can pull users out of their virtual space. Previous work has attempted to address these challenges in VR by mitigating disruptions during experiences ( Gottsacker et al., 2021 ; Ghosh et al., 2018 ) . However, questions arise about whether video meeting transcription interfaces can be directly applied to the spatial, immersive nature of VR meetings. One possible solution is to replicate video meeting interfaces by providing a spatially fixed transcription panel in front of the user, allowing them to scroll through transcripts. However, this approach often forces users to look away from other meeting participants, breaking social engagement and reducing the sense of presence in the group. Current social VR platforms, such as VRChat ( VRChat Inc., 2014 ) , handle this issue by attaching captions directly to the speaking avatar, maintaining eye contact and social interaction. Yet, these avatar-fixed captions, do not provide missed content, particularly when users return from a disruption and need to catch up. This raises the question of how we can bring the advantages of transcription interfaces from video meetings into immersive VR environments while balancing the need for both social presence and effective re-engagement after interruptions. To address this question, we first conducted a formative study examining how users interact with live transcriptions in immersive VR meetings when experiencing disruptions. Using a conventional table-fixed interface, we observed how participants dropped out and rejoined conversations, analyzing their catch-up strategies and social connection maintenance. This investigation revealed key challenges in designing transcription interfaces that support both efficient re-engagement and social presence in VR environments. Based on these insights, we developed EngageSync , a novel, context-aware avatar-fixed transcription interface that dynamically adapts to the user’s engagement state. EngageSync supports two key use cases: (1) providing real-time transcription and summaries during active participation, and (2) offering context-aware summaries to support re-engagement after disruptions. By adapting based on whether users are engaged or re-engaging, EngageSync maintains social presence while ensuring efficient catch-up on missed content. We evaluate EngageSync through two user studies, comparing it to existing interfaces such as table-fixed and always-on avatar-fixed transcription panels. Our evaluation investigates its effectiveness in enhancing social presence and information recall across different group sizes and scenarios. This work makes the following contributions: • Design insights for context-aware transcription in immersive VR environments: Based on a formative study with 12 participants, we identify key design challenges for supporting re-engagement and enhancing social presence in immersive group discussions. • EngageSync, a novel adaptive avatar-fixed transcription interface: EngageSync provides live transcriptions for active conversations and context-aware summaries for re-engagement supporting users to balance social presence with effective conversation catch-up. • Comprehensive evaluation across group sizes: Through two user studies, we evaluate EngageSync’s performance compared to table-fixed and avatar-fixed interfaces, demonstrating significant