Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18

Riccardo Bovo, Frederik Brudy, George Fitzmaurice, Fraser Anderson · 2025

Read Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18 by Riccardo Bovo, Frederik Brudy, George Fitzmaurice, Fraser Anderson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08689.txt.

Introduction

Riccardo Bovo and 3 co-authors present a system that enriches VR speech transcripts with gaze and pointing data in a 2025 paper. In virtual reality, words like it or there often point to things only seen in the scene. Plain transcripts lose that context, making coreference hard.

Eye tracking, laser pointing, and scene data are turned into text notes about non-verbal cues and objects of interest. Data come from twelve participants in pairs doing a design critique of a 3D apartment model. A language model resolves references far better with the enriched transcript than with speech alone. The gain supports better tools for immersive meetings.

Catalog record

CREATORRiccardo Bovo, Frederik Brudy, George Fitzmaurice, Fraser Anderson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human-centered computing CCS: Human-centered computing Natural language interfaces CCS: Human-centered computing Virtual reality CCS: Human-centered computing Collaborative interaction Riccardo Bovo, Frederik Brudy, George Fitzmaurice, Fraser Anderson Abstract . Understanding transcripts of immersive multimodal conversations is challenging because speakers frequently rely on visual context and non-verbal cues, such as gestures and visual attention, which are not captured in speech alone. This lack of information makes coreferences resolution-the task of linking ambiguous expressions like “it” or “there” to their intended referents-particularly challenging. In this paper we present a system that augments VR speech transcript with eye-tracking laser pointing data, and scene metadata to generate textual descriptions of non-verbal communication and the corresponding objects of interest. To evaluate the system, we collected gaze, gesture, and voice data from 12 participants (6 pairs) engaged in an open-ended design critique of a 3D model of an apartment. Our results show a 26.5% improvement in coreference resolution accuracy by a GPT model when using our multimodal transcript compared to a speech-only baseline. Figure 1. Depiction of our system performing coreference resolution by leveraging non verbal cues such as pointing and visual attention. The system uses user’s pointing behaviour and eye-gaze to determine the referent of ambiguous referring expressions, by correlating the location of eye gaze and pointing targets with pronoun "this" used by users. 1. Introduction The use of VR and AR for collaborative applications is rapidly growing, enabling immersive recordings of activities, such as design reviews Liu et al. 2020 ; Liu et al. 2014 ; López-Tarruella et al. 2018 ; Autodesk 2023 ; Autodesk 2023a ; Arkio ehf. 2023 , remote support RealWear, Inc. 2023 ; Kim et al. 2020 , and social interactions VRChat Inc. 2023 . These recordings are increasingly revisited to support collaboration, analysis, and follow-up tasks. As adoption grows, machines will need to better understand such recordings in order to summarize discussions, extract insights, and provide accessibility support. However, immersive conversations (both, in real-world or mixed reality environments) are inherently multi-modal. Besides speech, conversation participants frequently use non-verbal cues–such as gaze and pointing–to establish shared understanding ( Figure 1 ). This poses a difficulty for machine comprehension. Referring expressions (REs), such as “this” or “that,” are common in natural conversation, yet their meaning often depends on visual or gestural context. For example, in the utterance “I like it!” the RE “it” can only be understood when paired with a gesture or gaze toward the referent. Current Large Language Models (LLMs) and Visual Large Language Models (VLMs) can process transcripts of meetings to generate summaries and insights, but they struggle with these ambiguous Referring Expressions, leading to incorrect or incomplete identification of referenced objects. This limits the ability of intelligent systems to support tasks such as summarizing VR conversations Zhang et al. 2023 , extracting information from design review meetings Mahadevan et al. 2023 , and offering real-time accessibility support. Prior research has explored grounding conversation in shared visual contexts by linking dialogue to entities in a scene using neural models Yu et al. 2019 ; Goel et al. 2023 ; Kong et al. 2014 ; Guo et al. 2022 ; Kottur et al. 2021 . However, these approaches are ineffective when speakers never explicitly name the object they are referring to (exophora). For example, in the dialogue ‘P1: "What do you think about this?"‘ ‘P2: "It does not look comfortable!"‘, the referent remains unclear without additional cues. Even when entities are explicitly named (endophora), coreference resolution is often unreliable in natural conversations Zhang et al. 2023 . For instance, after ‘P1: "There is a funny coffee table."‘ a response like ‘P2: "We should move this."‘ could refer either to the table or another object introduced later. While seminal HCI work such as “Put That There” Bolt 1980 and follow-ups Romaniak et al. 2020 ; Miniotas et al. 2006 ; Bovo et al. 2023 leveraged non-verbal cues in human–machine interaction, they did not address multi-speaker conversations or use such cues for transcript comprehension. Similarly, studies on gaze synchrony in collaboration Vrzakova et al. 2019 ; Villamor & Rodrigo 2018 ; Moulder et al. 2023 ; Pietinen et al. 2008 highlight its role in establishing shared focus, but have not quantified how effectively it identifies the precise object under discussion. This paper proposes a system ( Figure 2 ) that augments VR speech transcripts to improve coreference resolution, by leveraging non-verbal cues. By inferring speakers’ attention through their gaze and pointing behavior, the system disambiguates spatial implicit referring expressions (REs) such as ‘P1: "What do you think about this?"‘ or ‘P2: "It does not look comfortable."’ The system augments the transcript with contextual cues, linking these expressions to the intended object in the scene (e.g., the living room sofa), enabling more accurate downstream processing, such as summarization and information extraction. Our primary contribution is a system that integrates gaze and pointing into VR speech transcripts to address the fundamental problem of coreference resolution. We built a multi-user VR application that captures verbal interactions, eye-tracking data, and laser-pointing behavior. VR provides a controlled environment where pre-segmented 3D objects simplify linking verbal references and non-verbal behavior, avoiding the challenges of object detection and segmentation that arise in AR or real-world settings. VR is increasingly adopted for collaborative design review in both academia Liu et al. 2020 ; Liu et al. 2014 ; López-Tarruella et al. 2018 and commercial applications Autodesk 2023 ; Autodesk 2023a ; Arkio ehf. 2023 , making it a natural setting to systematically study how non-verbal cues can be integrated into transcript comprehension. To evaluate our system, we conducted a study with 12 participants (six pairs) performing a collaborative design review in VR. We compared coreference resolution performance performed by Chat GPT4 on our augmented transcripts against a baseline using only speech transcripts. The primary contributions of this work are twofold: • A novel system that augments speech transcripts from collaborative VR sessions with non-verbal cues (gaze and pointing) to resolve ambiguous referring expressions, validated through a 12-participant study that demonstrates a 26.5% improvement in coreference resolution accuracy over a speech-only baseline. • A quantitative analysis of how different non-verbal behaviors—gaze and pointing—and their synergies (individual, concurrent, and recurrent patterns) contribute to identifying the object of interest, establishing a clear hierarchy of cues for resolving ambiguity in immersive collaborative dialogue. 2. Related Work We review previous work on coreference resolution , visual attention as well as pointing based communication in collaborative settings, and how HCI research leveraged comprehending deictic behaviour such as speech+gaze and speech+gestures through non verbal and multimodal interaction . 2.1. Coreference Resolution Coreference resolution involves identifying words and phrases in a text that refer to the same entity, a crucial task in natural language processing Goel et al. 2023 . This task is particularly challenging in conversational contexts due to their fluid structure, dynamic topic shifts, ambiguous pronoun use, and implicit shared knowledge, which can obscure clear reference resolution Yu et al. 2019 . For example, speakers often

Read the complete text