MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue

2026

Read MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21796.txt.

Introduction

Anna Deichler and co-authors introduce MM-Conv, a dataset and benchmark for grounding language in dynamic 3D dialogue. Vision-language models handle static images well but stumble on ambiguous phrases in multi-turn conversation. The benchmark draws on six point seven hours of egocentric virtual-reality interaction with synced speech, motion, gaze, and scene geometry, holding over four thousand two hundred hand-checked referring expressions. A two-stage pipeline resolving chat ambiguity before visual localization lifts grounding by eleven to twenty-two percentage points on average.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47687
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue Abstract Grounding language in the physical world requires AI systems to interpret references that emerge dynamically during conversation. While current vision-language models (VLMs) excel at static image tasks, they struggle to resolve ambiguous expressions in spontaneous, multi-turn dialogue. We address this gap by introducing (1) a benchmark for referential communication in dynamic 3D environments, built from 6.7 hours of egocentric VR interaction with synchronized speech, motion, gaze, and 3D scene geometry, and (2) a two-stage grounding pipeline that explicitly resolves conversational ambiguity before visual localization. The benchmark includes over 4,200 manually verified referring expressions spanning full, partitive, and pronominal types. Our contextual rewriting approach improves grounding performance by 11–22 percentage points on average, with a pure detector (GroundingDINO) reaching 56.7% on pronominals after rewriting—nearly double the best end-to-end baseline. Results demonstrate that decoupling linguistic reasoning from visual perception is more effective than end-to-end approaches for conversational grounding. Keywords: vision-language models, referential communication, multimodal grounding, egocentric dialogue, embodied AI MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue Anna Deichler, Jim O’Regan, Fethiye Irmak Dogan, Lubos Marcinek, Anna Klezovich, Iolanda Leite, and Jonas Beskow KTH Royal Institute of Technology, Stockholm, Sweden {deichler, joregan, fidogan, lubosm, annkle, iolanda, beskow}@kth.se Abstract content 1. Introduction Understanding and resolving referring expressions in situated, real-world contexts remains a core challenge for multimodal AI. While recent progress in vision-language models (VLMs) has brought significant advances in grounding natural language to images and videos, these models often fall short when it comes to reference resolution in dynamic, embodied environments. Here, language is rarely disambiguated by words alone; instead, meaning is distributed across multiple modalities, including speech, gesture, gaze, and spatial context, unfolding over time and grounded in interaction 11 . These capabilities are critical for applications such as home-assistive robots or AR-based guides, where agents must interpret ambiguous phrases like ‘that one’ in context. Existing benchmarks for referring expression grounding, such as ScanRefer 5 or ReferIt3D 1 , have driven progress but are limited by their reliance on scripted, single-turn textual descriptions and static 3D scenes. This leaves a critical gap in evaluating models on the spontaneous, messy, and multimodal nature of real human dialogue. To address this, we introduce a new benchmark that shifts the focus to the challenge of resolving spontaneous, conversational references as they emerge naturally in rich, situated dialogues. These expressions, such as “this one here”, “the yellowish thing”, or simply “it”, are inherently ambiguous unless grounded in a shared perceptual and interactional context. In realistic, situated dialogue, language is often insufficient on its own. The problem of referential understanding has been approached using descriptive features in language 28 ; 10 ; 15 , but in spontaneous interaction, non-verbal cues are crucial for resolving ambiguity, with studies showing that referential gaze is tightly synchronized with speech 29 and that gestures provide effective guidance for disambiguating objects 35 ; 27 . Recent work has explored learning to generate such pointing behaviors for embodied agents 8 . Early benchmarks miss core conversational phenomena—ellipsis, anaphora, and multimodal deixis, common in situated dialogue. Current VLMs, largely trained on text–image pairs, struggle to resolve such ambiguities without nonverbal cues (e.g., gaze, gesture) and often lack explicit, structured reasoning for highly underspecified references. Our contribution is the first benchmark that synchronizes these signals with conversational language, setting the stage for truly multimodal grounding. To support this, we built a 6.7-hour dataset of immersive, multimodal interactions with synchronized speech, motion, gaze, and 3D scene geometry. We annotated 4,211 naturally occurring referring expressions and designed an evaluation framework spanning explicit nouns to highly contextual pronouns. Results reveal critical limitations of state-of-the-art VLMs for temporally grounded references and demonstrate the effectiveness of a modular, ambiguity-first pipeline for context-aware grounding. Our main contributions are: • A new benchmark for spontaneous, multimodal referential grounding , with 4,211 annotated expressions from 6.7 hours of conversational data in dynamic 3D environments. • A two-stage grounding pipeline that separates conversational disambiguation from visual localization, improving performance by 11–22 percentage points across reference types. • Systematic evaluation revealing that (1) current VLMs struggle with conversational references despite native context support, and (2) modular architectures outperform end-to-end approaches for situated grounding. • A rich, reusable dataset with synchronized full-body motion, gaze, facial expression, and egocentric video to support downstream research in embodied interaction. The rest of the paper is structured as follows: Section 2 describes related work in referential understanding and grounding in vision-language models. Section 3 describes the dataset and the annotations, and Section 4 presents the human and VLM experiments. 2. Related Work 2.1. Referential Understanding Benchmarks Table 1: Comparison with prior multimodal referential benchmarks. Dataset Modalities Scale Embodied Dialogue ScanRefer (2020) Text, 3D 51.6k, 800 scenes No One-shot YouRefIt (2021) Text, RGB, Gesture 4k, 432 scenes Yes Multi-turn TEACh (2022) Text, Sim3D, Actions 3k dialogues Nav/Manip Multi-turn Ours Speech, RGB, Depth, Motion, Gaze 4.2k, 5 scenes Full VR Spontaneous Foundational datasets like ScanRefer 5 and ReferIt3D 1 established 3D referring expression grounding but rely on single-turn, text-only descriptions. While YouRefIt 6 added multi-turn dialogue with gesture, it uses third-person video. TEACh 16 includes embodied dialogue but lacks continuous multimodal streams (gaze, motion). Our work addresses this gap by integrating temporally synchronized speech, full-body motion, gaze, and egocentric vision from immersive VR, enabling the investigation of reference resolution with all modalities naturally co-occurring in interaction. Table 1 summarizes key differences. 2.2. Vision–Language Models for Grounding Vision-language models (VLMs) for grounding have evolved significantly, moving from dual-encoder architectures to more flexible generative models. This progression is critical for handling the spontaneous, conversational language, which stands in contrast to the clean, template-like queries found in traditional referring expression benchmarks. Models like GroundingDINO 18 , built on CLIP 25 embeddings, perform well on short to medium-length prompts; however, their ability to generalize to longer, more conversational inputs is limited. In contrast, Florence-2 32 integrates a co-trained causal language model that supports prompt-driven decoding over both visual and textual inputs. This makes Florence-2 particularly well-suited for grounding tasks in ecologically valid settings where referring expressions are embedded in natural dialogue. Recent advances in GPT-style vision-language models, such as GroundingGPT 17 , further push this boundary by leveraging large-scale, decoder-only architectures that unify visual and textual modalities in a generative framework. Table 2 provides a comprehensive comparison of these vision-language grounding models, highlighting their architectural differences and capabilities. 3. Dataset Table 2: Comparison of vision-language grounding models, including text/visual encoders, training status, and notable capabilities. Model Text Enc. Visual Enc. Visual In. Token Notable Features Grounding DINO 18 BERT DETR (Swin) RGB (2D) 256 Referring expression grounding via cross-attention; zero-shot localization with box output. Florence-2 32 Transformer (BART) DaViT RGB (2D) 1024 Foundation model with prompt-driven generation; supports grounding using polygon or box output. KOSMOS-2 23 Transformer (causal LM) ViT (patch14-224) RGB (2D) 2048 Grounds text spans to boxes via “location tokens”; outputs grounded captions/VQA with box coordinates (open-vocab grounding). Ferret 34 Vicuna-style LLM CLIP ViT-L/14 (region features) RGB (2D) 4096 Referring and region-level understanding via explicit region features; supports referring/grounded dialogue with box-based region specification. GroundingGPT 17 Vicuna-v1.5 CLIP ViT-L/14, Q-former (video) RGB (2D), Video 4096 Multi-modal

Read the complete text