Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

Chanyoung Gwak *, Yoonwoo Jeong *, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin †, Minsu Cho † · 2026

Read Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps by Chanyoung Gwak *, Yoonwoo Jeong *, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin †, Minsu Cho † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23023.txt.

Introduction

Gwak and co-authors present Cog3DMap, a framework that gives vision-language models an explicit 3D map. Multi-view spatial reasoning is hard because model features carry meaning but little geometry. Cog3DMap builds a 3D memory step by step from many views. Each token holds both semantic and geometric data, grounded in space. The model then reasons over that structured map directly. Tests on spatial reasoning benchmarks reach state-of-the-art scores, the authors report. Code is set for public release.

Catalog record

CREATORChanyoung Gwak *, Yoonwoo Jeong *, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin †, Minsu Cho †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55508
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps Chanyoung Gwak *, Yoonwoo Jeong *, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin †, Minsu Cho † Abstract Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap , a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available. * * footnotetext: Equal contribution. † † footnotetext: Corresponding authors. 1 Introduction Multimodal Large Language Models (MLLMs) [ 21 , 8 , 7 , 3 , 2 ] have achieved remarkable success in general visual understanding by leveraging the reasoning capabilities of Large Language Models (LLMs). Despite their success, precise spatial reasoning from multi-view images remains a fundamental challenge, as their visual representations are predominantly semantic and lack explicit geometric grounding. To address this limitation, recent studies [ 42 , 12 ] incorporate spatially-annotated training data constructed from ground-truth 3D annotations, while another line of work [ 52 , 38 , 10 ] further augments visual tokens with spatial features extracted from external models such as pointmap estimators [ 35 ] . Although these approaches enhance the spatial reasoning of MLLMs, they still rely on the MLLM to implicitly infer the underlying 3D structure, without providing an explicit geometric representation of the scene. A promising direction for overcoming this challenge lies in how humans represent spatial environments. In cognitive science, this ability is broadly studied under the concept of spatial memory , which concerns encoding, storing, and retrieving information about spatial layouts and object locations [ 5 , 26 ] . A central framework in this line of research is the cognitive map , an internal representation that encodes spatial relationships among locations in the environment [ 34 , 29 ] . Such representations are thought to be acquired incrementally, progressing from recognizing individual landmarks, to learning routes between them, and finally to forming survey-level knowledge that captures metric spatial relationships [ 33 ] . Drawing from this principle, we introduce Cog3DMap , a framework that incrementally constructs a computational counterpart of cognitive maps from multi-view images. Following the incremental nature of cognitive map formation, Cog3DMap introduces a recurrent framework that progressively integrates multi-view observations into a unified 3D map, where each spatial coordinate is associated with a token carrying both semantic and geometric information. Unlike prior methods that rely on the MLLM to implicitly infer spatial structure from auxiliary features, Cog3DMap provides the MLLM decoder with an explicit and compact 3D map, enabling more direct and interpretable spatial reasoning. Experiments on VSTI-Bench [ 12 ] and VSI-Bench [ 42 ] demonstrate that our Cog3DMap establishes new state-of-the-art results on spatial reasoning benchmarks. Moreover, experiments on RoboFAC [ 25 ] validate the compactness of Cog3DMap, achieving competitive or superior performance over previous methods while reducing the number of visual tokens by up to 90.2%. Furthermore, ablation studies confirm the effectiveness of the proposed explicit 3D map representation in improving spatial understanding. In summary, our contributions are as follows: • We propose Cog3DMap , a framework that constructs a 3D memory from multi-view images, facilitating spatial understanding of MLLMs through compact and interpretable 3D structure. • We present an effective strategy to integrate semantic and geometric features for each visual token, thereby enriching it for spatial grounded reasoning. • We demonstrate state-of-the-art performance across various spatial understanding benchmarks, with ablation studies validating the effectiveness and compactness of using explicit 3D for MLLM spatial reasoning. 2 Related Work 2.1 Multimodal Large Language Models (MLLMs) Multimodal Large Language Models (MLLMs) have achieved remarkable success across various general visual reasoning tasks by extending the reasoning capabilities of Large Language Models (LLMs) to encompass visual perception. For instance, LLaVA [ 21 ] introduces visual instruction tuning that aligns visual features with the language embedding space via trainable projection layers. Building upon this paradigm, subsequent architectures have adopted multi-resolution strategies for high-resolution inputs [ 23 ] or utilized unified multimodal encoders [ 20 ] . Among these, the Qwen series [ 36 , 3 , 2 ] has demonstrated exceptional effectiveness across diverse MLLM benchmarks. Despite these advances, existing MLLMs still exhibit suboptimal performance in spatial understanding tasks requiring geometric interpretation of multi-view images or 3D data, primarily due to the scarcity of spatially-annotated datasets and limited visual grounding capabilities. To overcome this limitation, we propose a framework that injects explicit 3D positional information into individual visual tokens, yielding an interpretable and spatially grounded representation that enables effective spatial understanding from multi-view images. 2.2 Visual Geometry Models (VGMs) Predicting 3D structure from multi-view images remains a fundamental challenge in computer vision. Traditionally, this problem has been addressed through per-scene optimization via bundle adjustment [ 31 , 32 ] or neural field optimization [ 27 , 19 ] . Recently, VGGT [ 35 ] has demonstrated the potential of feed-forward transformers in predicting 3D geometries with high fidelity while eliminating costly per-scene optimizations. However, the computational complexity of the transformer restricts scalability to large-scale scenes with numerous viewpoints. CUT3R [ 37 ] overcomes this limitation by adopting a recurrent framework that updates the memory state, which is responsible for storing previous observations. While this approach improves scalability, it suffers from catastrophic forgetting due to its implicit representation. To address this issue, Point3R [ 39 ] introduces an explicit token-based memory, where each token stores a 3D position and associated features. We observe that this recurrent mechanism closely reflects the concept of cognitive maps in cognitive science, progressively constructing a 3D map of the environment as a memory state from sequential observations. 2.3 Spatial Reasoning of MLLMs Early 3D-LLM approaches [ 15 , 40 , 17 ] utilize visual instruction tuning on 3D-text pairs, yet they are constrained by costly 3D data acquisition, resulting in significantly smaller training scales compared to 2D training datasets. Consequently, the recent research paradigm has shifted toward geometry-augmented multi-view LLMs [ 38 , 52 , 12 , 18 ] , which learn from multi-view images rather than heavily processed 3D data. In parallel, some works [ 12 ] further enhance spatial reasoning by introducing VQA samples automatically generated from metadata such as object dimensions, coordinates, and segmentation masks. Without requiring explicit 3D inputs at inference time, these models outperform conventional 3D-LLMs on spatial reasoning benchmarks while retaining the broad visual understanding capabilities of general-purpose MLLMs. A common strategy in this line of work is to leverage VGMs that inject 3D positional information into visual tokens. For instance, VG-LLM [ 52 ] , Spatial-MLLM [ 38 ] , and VLM-3R [ 12 ] fuse multi-view visual tokens with geometric features from VGGT [ 35 ] , while

Read the complete text