Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views
Jiho Choi, Seonho Lee ⋆ {}^{\star}\lx@orcidlink{0009-0006-4183-4214}{\orcidlogo}, Seojeong Park, Hyunjung Shim · 2026
Read Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views by Jiho Choi, Seonho Lee ⋆ {}^{\star}\lx@orcidlink{0009-0006-4183-4214}{\orcidlogo}, Seojeong Park, Hyunjung Shim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23557.txt.
Introduction
Choi and colleagues train multimodal models to reason about three-dimensional scenes from several photos, in a 2026 computer vision paper. Answering questions about a room from multiple views demands a coherent mental map. Models trained only on final answers often build inconsistent maps and pick poor viewpoints. Their reasoning then sounds plausible while resting on the wrong view of the scene.
The framework, DR-MV3D, splits the job into three supervised steps. First the model builds an allocentric global map, a map from a neutral overhead frame. Then it plans a trajectory of viewpoints suited to the question. Finally it grounds the answer in an egocentric view, the scene as seen from a position inside it. Two dense rewards grade the map and the viewpoint order against geometry from frozen vision models. Tests on MindCube, VSI-Bench, and BLINK show steady gains over strong multi-image baselines.
Catalog record
| CREATOR | Jiho Choi, Seonho Lee ⋆ {}^{\star}\lx@orcidlink{0009-0006-4183-4214}{\orcidlogo}, Seojeong Park, Hyunjung Shim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views Jiho Choi, Seonho Lee ⋆ {}^{\star}\lx@orcidlink{0009-0006-4183-4214}{\orcidlogo}, Seojeong Park, Hyunjung Shim Abstract Multi-view 3D Visual Question Answering (MV3D-VQA) requires integrating partial observations into a coherent 3D scene representation and selecting informative viewpoints for multi-step spatial reasoning. However, current multimodal LLMs are typically trained with sparse, answer-level supervision, which often yields inconsistent cross-view reasoning and brittle view selection. We present DR-MV3D (Dense Reward for MV3D-VQA), a map-grounded learning framework that provides dense, verifiable rewards to supervise the reasoning process. Our approach decomposes MV3D-VQA into (i) allocentric global map construction, (ii) question-conditioned view-trajectory planning, and (iii) egocentric grounding for answer prediction. To make intermediate steps learnable without manual annotations, we introduce two rewards: a global consistency reward that aligns the predicted map with geometry-consistent pseudo targets from frozen 3D vision foundation models (e.g., VGGT + SAM3), and a local trajectory reward that supervises ordered viewpoint selection. We optimize the full pipeline with trajectory-level policy optimization (GRPO). Experiments on MindCube, VSI-Bench, and BLINK (MV) show that DR-MV3D consistently improves over strong multi-image baselines, supporting the effectiveness of process-level dense supervision for multi-view 3D reasoning. Code is available at: https://github.com/kaist-cvml/dr-mv3d 1 Introduction Figure 1 : Overview of DR-MV3D. Existing MLLMs trained with sparse , answer-level supervision (left) often build inconsistent cognitive maps and misread egocentric directions, leading to wrong answers despite plausible-looking reasoning. In contrast, DR-MV3D (right) supervises the reasoning process with dense , verifiable rewards: a global reward aligns the predicted allocentric cognitive map with a geometry-consistent target from a frozen 3D vision foundation model (VFM), and a local trajectory reward guides ordered viewpoint selection so that the model grounds the query in the correct view (e.g., resolving the ego-direction “left” via the appropriate image) and arrives at the correct answer. Recent advances in multimodal large language models (MLLMs) [ 20 , 30 , 36 , 51 , 46 ] have achieved remarkable progress on visual perception, visual question answering, and multimodal reasoning [ 6 , 46 , 14 ] . However, many real-world environments are inherently three-dimensional and can only be observed through multiple partial viewpoints. This has motivated growing interest in multi-view 3D visual question answering (MV3D-VQA) [ 17 , 59 ] , where a model must integrate observations from multiple viewpoints and reason about spatial relationships within a scene. Unlike conventional visual question answering that relies on a single image, MV3D-VQA requires aggregating fragmented visual evidence across views to form a coherent understanding of the underlying 3D scene [ 17 , 12 , 44 , 59 ] . Such spatial reasoning is essential for embodied and physical intelligence [ 10 , 22 , 62 ] , where agents must operate under partial observability and reason about object relations, occlusions, and hypothetical movements. Despite these advances, existing MLLMs still struggle to construct and utilize coherent three-dimensional spatial representations from partially observable multi-view inputs [ 44 , 12 , 18 , 59 , 52 ] . Empirical studies on multi-view reasoning benchmarks reveal several systematic failure modes: models frequently produce inconsistent predictions across viewpoints, exhibit unstable reasoning under occlusion, and perform close to random on compositional spatial queries that require integrating evidence from multiple views [ 17 , 12 , 44 , 59 ] . These observations suggest that the core challenge is not merely the availability of visual inputs, but the difficulty of constructing and maintaining a consistent scene-level representation that supports structured spatial reasoning. Although MLLMs may implicitly encode partial spatial knowledge, they often fail to align, retrieve, and update this information coherently across viewpoints, preventing fragmented observations from being organized into a unified representation of the underlying 3D scene. Recent work has begun to address this limitation by introducing explicit intermediate representations. MindCube [ 44 ] and related studies [ 17 , 15 , 59 ] demonstrate that providing additional views alone yields limited gains; instead, they show that performance improves when models first construct an intermediate representation such as a cognitive map and then reason over it. These findings establish the map-then-reason paradigm as a promising direction: the cognitive map functions as a persistent spatial workspace that organizes fragmented observations into a coherent scene-level structure for downstream tasks such as global scene summarization (spatial memory) and question-conditioned view planning. Despite this progress, several key challenges remain. First, most existing approaches rely solely on allocentric (world-centered) representations. While allocentric maps provide a stable global reference frame, reasoning tasks often involve egocentric queries such as viewpoint-dependent relations or hypothetical movements, creating a representation misalignment between the constructed map and the reasoning process. Second, MLLMs often struggle to produce geometrically consistent 3D structures even when generating intermediate maps. In contrast, specialized 3D Vision Foundation Models (VFMs) [ 43 , 45 ] , such as VGGT, exhibit strong capabilities in multi-view geometric reasoning, including depth estimation, pose consistency, and object-level spatial layout reconstruction. This reveals a clear representation quality gap between language-driven map generation and geometry-grounded vision models. As illustrated in Figure 2 , cognitive maps derived from VGGT [ 43 ] are more similar to the ground-truth cognitive map than those generated by Qwen2.5-3B [ 36 ] , suggesting that VFM-based representations can provide more reliable structural signals for spatial reasoning. Third, most existing reinforcement learning approaches rely primarily on sparse rewards defined by final answer correctness, which provides limited guidance for learning spatially structured reasoning processes. Figure 2 : Cognitive map and metric comparison. Top: multi-view observations, VFM-based scene reconstruction (VGGT+SAM3), and similarity scores to the GT cognitive map. The similarity is reported from three aspects: Dir. measures inter-object directional relations (relative spatial ordering between objects), Face. measures the facing orientation of the viewpoint, and Overall is a weighted combination of Dir. and Face. We compute Dir. as pairwise directional agreement and Face. as viewpoint-facing agreement; Overall is their weighted average. Bottom: GT cognitive map, an MLLM-generated cognitive map (Qwen2.5-3B), and a VFM-derived cognitive map. The VFM-derived cognitive map achieves higher similarity to GT than the MLLM-generated map, suggesting that VGGT+SAM3 can provide a reliable pseudo-structural signal when GT cognitive maps are unavailable. 1 (Forward) 2 3 4 (a) Multi-view Observation (b) Scene Reconstruction 0 20 40 60 80 100 Dir. Face. Overall MLLM VFM SFT w/ VFM (c) Similarity Chart window and blue curtain (forward) pink bottle (forward) closet and door (back) white wall (forward-left) pink plush toy and headboard (forward-right) 1 selected 2 3 4 R L F B (d) GT Cognitive Map white wall (back) pink bottle (forward) closet and door (forward-right) window and blue curtain (forward-left) pink plush toy and headboard (forward) 2 selected (incorrect) 3 4 1 R L F B (e) CogMap from Qwen2.5-3B window and blue curtain (forward) pink bottle (forward) closet and door (back) white wall (left) pink plush toy and headboard (forward-right) 1 selected (correct) 2 3 4 R L F B (f) CogMap from VGGT In this work, we propose DR-MV3D (Dense Reward for Multi-View 3D reasoning), a map-grounded multi-view reasoning framework that explicitly addresses these challenges as in Figure 1 . To resolve the allocentric–egocentric representation misalignment (Challenge 1), our approach constructs a global allocentric cognitive map summarizing scene-level spatial structure from multi-view observations, and then grounds the