Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (

Wentao Mo, Qingchao Chen, Yuxin Peng, Siyuan Huang, Yang Liu · 2025

Read Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia ( by Wentao Mo, Qingchao Chen, Yuxin Peng, Siyuan Huang, Yang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11058.txt.

Introduction

Wentao Mo and co-authors push 3D scene question answering beyond single views in a 2025 paper. Older datasets rarely force a model to combine looks from several angles, and their labels tie text to lone objects. Their new MV-ScanQA set makes sixty-eight percent of questions need multi-view reasoning, far above past sets. A companion corpus, TripAlign, holds one million triplets linking 2D views, object groups, and text.

Their baseline model, LEGO, borrows strength from big 2D vision-language models and pre-trains on TripAlign. It tops the new benchmark and also leads older captioning and question sets, they report. Data and code are public.

Catalog record

CREATORWentao Mo, Qingchao Chen, Yuxin Peng, Siyuan Huang, Yang Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11058

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3758244 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Scene understanding CCS: Computing methodologies Spatial and physical reasoning Wentao Mo, Qingchao Chen, Yuxin Peng, Siyuan Huang, Yang Liu Abstract . The advancement of 3D vision-language (3D VL) learning is hindered by several limitations in existing 3D VL datasets: they rarely necessitate reasoning beyond a close range of objects in single viewpoint, and annotations often link instructions to single objects, missing richer contextual alignments between multiple objects. This significantly curtails the development of models capable of deep, multi-view 3D scene understanding over distant objects. To address these challenges, we introduce MV-ScanQA, a novel 3D question answering dataset where 68% of questions explicitly require integrating information from multiple views (compared to less than 7% in existing datasets), thereby rigorously testing multi-view compositional reasoning. To facilitate the training of models for such demanding scenarios, we present TripAlign dataset, a large-scale and low-cost 2D-3D-language pre-training corpus containing 1M ⟨ \langle 2D view, set of 3D objects, text ⟩ \rangle triplets that explicitly aligns groups of contextually related objects with text, providing richer, view-grounded multi-object multimodal alignment signals than previous single-object annotations. We further develop LEGO, a baseline method for the multi-view reasoning challenge in MV-ScanQA, transferring knowledge from pre-trained 2D LVLMs to 3D domain with TripAlign. Empirically, LEGO pre-trained on TripAlign achieves state-of-the-art performance not only on the proposed MV-ScanQA, but also on existing benchmarks for 3D dense captioning and question answering. Datasets and code are available at https://matthewdm0816.github.io/tripalign-mvscanqa . 1. Introduction Figure 1 . (a). MV-ScanQA tests multi-view reasoning, with 10× more multi-view questions than existing datasets. (b). TripAlign: A large-scale 2D-3D-language dataset that aligns object groups with captions for richer multi-modal supervision. (c). LEGO: Our 2D-3D LVLM leverages TripAlign to transfer 2D LVLM knowledge to 3D tasks. Recent advancements in Large Language Models (LLMs) ( OpenAI 2023 ; Touvron et al. 2023 ) and their visual integration have led to powerful 2D Large Vision-Language Models (2D LVLMs) ( Li et al. 2023 ; Liu et al. 2024 ; Liu et al. 2023 ) . However, extending these to 3D environments faces critical challenges stemming from fundamental limitations in existing 3D vision-language datasets ( Chen et al. 2021 ; Chen et al. 2020 ; Achlioptas et al. 2020 ; Azuma et al. 2022 ) : First, existing evaluation benchmarks ( Chen et al. 2021 ; Chen et al. 2020 ; Achlioptas et al. 2020 ; Azuma et al. 2022 ) do not adequately test true 3D spatial understanding. Our studies reveal that more than 93% of tasks can be effectively solved 1 1 1 94%, 96% and 93% for ScanQA, ScanRefer and Nr3D, respectively. Please refer to Section 3 for solvability explanation. using information from just a single egocentric view. This reliance on single viewpoints means that models are not compelled to develop deeper, multi-view spatial reasoning capabilities essential for genuine 3D understanding. Second, current training datasets suffer from sparse annotations. Existing datasets ( Chen et al. 2021 ; Chen et al. 2020 ; Achlioptas et al. 2020 ; Azuma et al. 2022 ) only link instructional text to single objects, missing the rich contextual relationships between multiple objects that naturally occur in scene descriptions, limiting models’ ability to learn from dense and complex multi-object-to-text correspondences. To address limited evaluation of multi-view 3D scene understanding, we introduce a new challenging evaluation dataset MV-ScanQA . MV-ScanQA is specifically designed to assess multi-view compositional reasoning by explicitly requiring information integration across multiple viewpoints. We define an instruction’s "solvability" based on whether all relevant objects are sufficiently visible within a set of "solvable views". In stark contrast to existing datasets where less than 7% of questions require multiple views, 68% of questions in MV-ScanQA necessitate at least two distinct views to answer correctly, and 13% require three or more views. This design pushes models beyond single-view comprehension, creating a substantially more challenging benchmark for true 3D scene understanding. To address the sparse annotation problem, we introduce pre-training dataset TripAlign , a large-scale 2D-3D-language corpus with over 1M triplets. Unlike previous datasets that only provide single object-text pairs, TripAlign achieves denser multi-object annotations by leveraging 2D views as an intermediary. Specifically, we form \langle 2D view, set of 3D objects, instruction text \rangle triplets, where the 2D view serves as a bridge to naturally group multiple contextually related objects that appear together. This approach enables us to automatically extract rich multi-object relationships from scene descriptions, providing richer cross-modal alignment signals for training. TripAlign is constructed via a fully-automated, dual pipeline: (1) generating instruction text from the original 2D views of the 3D scenes by image captioners, naturally capturing multi-object relationships visible in the view, and (2) extending existing 3D VL datasets by selecting informative views that contain multiple relevant objects. Finally, to demonstrate the utility of our proposed datasets, we present LEGO ( L arge 3D Vision-Language Model Learning from E gocentric G rounded O bservations) as a baseline solution. LEGO is a novel 2D-3D LVLM that uniquely builds upon pre-trained 2D LVLMs and effectively transfers their knowledge to the 3D domain. It leverages the instruction-relative egocentric views and the view-dependent multi-object alignment mechanism trained on TripAlign to process information from multiple viewpoints, showcasing strong performance on MV-ScanQA and other established 3D vision-language tasks. We make the following contributions: 1) MV-ScanQA : A novel and challenging 3D question answering benchmark designed to evaluate multi-view scene understanding and compositional reasoning capabilities, where 68% of questions require integrating information from multiple views. 2) In-depth Solvability Analysis : A quantitative analysis demonstrating the limitations of existing benchmarks in testing multi-view reasoning and highlighting the significant step forward provided by MV-ScanQA. 3) TripAlign : A large-scale 2D-3D-language tri-modal aligned corpus of over 1M triplets that explicitly aligns object groups and egocentric views with instruction texts, addressing data scarcity and providing richer view-centric multimodal correspondence signals than previous single-object annotations. 4) LEGO : A 2D-3D LVLM that, when trained on our TripAlign dataset, achieves state-of-the-art performance on existing 3D VL benchmarks and on our new MV-ScanQA evaluation set. Its success highlights both the critical role of high-quality multi-view training data and the limitations of current 3D LVLMs in multi-view reasoning, validating the necessity of TripAlign and MV-ScanQA in advancing the 3D VL field. 2. Related Work 3D Vision-Language (3D VL) Datasets. Existing 3D VL evaluation datasets ( Chen et al. 2020 ; Azuma et al. 2022 ; Achlioptas et al. 2020 ) primarily test single-view understanding, with over 93% instructions solvable from one viewpoint. This fails to evaluate true 3D spatial reasoning capabilities. MV-ScanQA addresses this gap by requiring enforced multi-view integration, with 68% of questions needing at least two views, creating a substantially more challenging benchmark that pushes models beyond single-view comprehension. Existing 3D VL pre-training datasets focus on combining data across multiple tasks to improve 3D VLMs’ understanding abilities. Previous work ( Zheng et al. 2024 ; Zhu et al. 2023 ; Jia et al. 2024 ; Huang

Read the complete text