SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs
Bo Gu, Zhikang Zhang, Zizhuang Wei, Zhenyuan Chen, Lingyun Li, Zhuoyi Song, [0.5em] Fudan University · 2026
Read SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs by Bo Gu, Zhikang Zhang, Zizhuang Wei, Zhenyuan Chen, Lingyun Li, Zhuoyi Song, [0.5em] Fudan University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09449.txt.
Introduction
Gu and five colleagues build SpaceMind++, a video model building a voxel map from RGB videos. That gives video models a lasting world-centred picture for 3D reasoning. Following the mammalian dual-stream system, where meaning and spatial cues are processed and integrated into an allocentric map, the map reorganises scattered observations into a shared picture. That keeps object permanence and topology across viewpoints.
To use the allocentric picture in a video model without breaking its interface, the team adds Coordinate-Guided Deep Iterative Fusion. It relays map-level spatial knowledge into original 2D traits. The fusion leans on coordinate embeddings and 3D Rotary Positional Encoding, grounding interactions in metric space, like entorhinal binding of sensory traits to metric space. Experiments show SpaceMind++ reaching new best performance on VSI-Bench, with stronger generalisation on SPBench, SITE-Bench, and SPAR-Bench, showing toughness in unseen environments.
Catalog record
| CREATOR | Bo Gu, Zhikang Zhang, Zizhuang Wei, Zhenyuan Chen, Lingyun Li, Zhuoyi Song, [0.5em] Fudan University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09449
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs Bo Gu, Zhikang Zhang, Zizhuang Wei, Zhenyuan Chen, Lingyun Li, Zhuoyi Song, [0.5em] Fudan University Abstract Recent multimodal large language models (MLLMs) have made remarkable progress in visual understanding and language-based reasoning, yet they lack a persistent world-centered representation for spatially consistent reasoning in 3D environments. Inspired by the mammalian dual-stream system, where semantic and spatial cues are processed separately and integrated into an allocentric cognitive map, we propose SpaceMind++ , a video MLLM architecture that explicitly builds a voxelized cognitive map from RGB videos. This map reorganizes fragmented egocentric observations into a shared 3D metric representation, enabling the model to preserve object permanence and spatial topology across changing viewpoints. To make this allocentric representation usable by a pretrained video MLLM without disrupting its native visual-token interface, we introduce Coordinate-Guided Deep Iterative Fusion, a new mechanism that relays map-level spatial knowledge back into the original 2D visual features. This fusion is explicitly guided by coordinate embeddings and 3D Rotary Positional Encoding, which ground semantic interactions in metric 3D space, resembling the entorhinal binding of sensory features to metric space. Extensive experiments show that SpaceMind++ achieves new state-of-the-art performance on VSI-Bench. Furthermore, it demonstrates superior out-of-distribution generalization on SPBench, SITE-Bench, and SPAR-Bench, underscoring its robustness in unseen 3D environments. 1 Introduction Spatial intelligence requires models to reason about the 3D structure of environments beyond isolated visual observations. Despite recent advances that have endowed Multimodal Large Language Models (MLLMs) with strong visual understanding and reasoning capabilities [ 3 , 29 , 32 , 75 ] , a fundamental gap persists between language-level reasoning and 3D spatial organization [ 57 , 70 ] . This disconnect limits the ability of MLLMs to ground their predictions in physical space, leading to unstable spatial reasoning, such as inconsistent distance estimation and object-level hallucinations [ 65 ] . This limitation stems from a lack of spatial constancy: visual observations are egocentric, partial, and transient, with viewpoint-dependent distances, occlusions, and fragmented frame-level evidence of the underlying 3D scene. Recent efforts to build 3D-aware MLLMs attempt to address this issue by incorporating feed-forward geometry encoders alongside general-purpose visual encoders. These methods commonly fuse 2D visual features and 3D geometric cues through MLPs [ 61 ] , cross-attention [ 13 ] , or camera-guided integration [ 72 ] . However, they largely treat geometry as an auxiliary modality aligned with egocentric visual tokens, enhancing spatial cues only locally without forming an explicit and persistent 3D structure. Rather, reliable spatial intelligence requires more than extracting geometric cues from individual observations; it requires a stable world-centered representation that preserves object permanence, metric consistency, and global scene layout across changing viewpoints. Human spatial cognition offers a blueprint for spatially grounded reasoning. Neuroscience studies highlight two key mechanisms for spatial awareness: the dual visual stream [ 18 , 52 ] and the cognitive map [ 22 , 41 , 51 ] . The former separates visual processing into ventral “what” and dorsal “where” pathways, while the latter provides an allocentric representation for reasoning about spatial relationships beyond the current field of view. Although cognitive maps have inspired computational models such as the Tolman-Eichenbaum Machine [ 59 , 60 , 58 ] and visual-language navigation systems [ 20 , 45 ] , their use as an architectural principle for large-scale MLLMs remains under-explored. Inspired by principles of human spatial cognition, we present SpaceMind++ , a video MLLM that introduces an allocentric cognitive map as a first-class architectural component for spatial reasoning. In contrast to prior 3D-aware MLLMs that treat 3D geometry as an auxiliary feature stream, SpaceMind++ projects frame-level 2D semantic features into a voxelized map organized in 3D space. This transformation converts fragmented egocentric observations into a structured world-centered representation, allowing the model to maintain spatial topology across viewpoints. To make this map useful for language reasoning, we further introduce Coordinate-Guided Deep Iterative Fusion (CDIF) , which relays spatial knowledge from the map back to the original visual tokens through iterative map self-attention and map-to-visual cross-attention. The LLM therefore receives geometry-enhanced visual tokens while preserving the pretrained visual-language interface. A key requirement for such fusion is that semantic interactions must be governed by metric 3D relationships rather than by arbitrary sequence order. Inspired by hippocampal–entorhinal spatial coding, which supports metric and allocentric representations of space [ 22 , 44 , 43 ] , we make attention coordinate-aware by injecting explicit 3D coordinate embeddings and applying 3D Continuous Rotary Position Embeddings. This design binds visual semantics to spatial locations during both attention steps, enabling SpaceMind++ to reason over object relations within a 3D metric structure. Experiments on VSIBench and other benchmarks demonstrate that SpaceMind++ achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks. In summary, our main contributions are: • We introduce a brain-inspired design perspective for spatially grounded video MLLMs, drawing motivation from the mammalian dual visual stream and cognitive map to separate semantic recognition, spatial localization, and allocentric scene organization. • We propose SpaceMind++, a novel video MLLM architecture that builds a voxelized allocentric cognitive map from RGB videos, effectively preserving spatial topology and object permanence across diverse viewpoints. • We introduce Coordinate-Guided Deep Iterative Fusion (CDIF), a coordinate-aware mechanism that performs iterative map reasoning and map-to-visual reading with explicit 3D coordinate embeddings and 3D RoPE, while keeping the pretrained MLLM interface intact. • We curate SpaceMind-900K for spatial instruction tuning and validate SpaceMind++ across multiple benchmarks, achieving new state-of-the-art performance on VSI-Bench and strong generalization on SPBench, SITE-Bench, and SPAR-Bench. 2 Related Work 2.1 Spatial Reasoning in Multimodal Large Language Models Multimodal large language models (MLLMs) have achieved strong progress in visual question answering, image captioning, and video understanding [ 1 , 2 , 8 , 31 , 32 , 42 , 75 ] , yet they remain unreliable on spatially grounded tasks such as metric estimation, spatial relationship judgment, and navigation-oriented reasoning [ 49 , 65 , 69 ] . Aiming to solve this problem, existing spatial MLLMs can be broadly grouped into two lines. One line introduces explicit 3D inputs, including point clouds [ 16 , 23 , 36 ] and depth maps [ 5 , 10 , 73 ] , and aligns these structured scene representations with language models. Although effective when high-quality geometry is available, these methods often rely on reconstruction pipelines, limiting their scalability to unconstrained RGB videos. Another line improves RGB-based spatial reasoning by augmenting VLMs with 3D geometry encoders or internal spatial representations, as explored in Spatial-MLLM [ 61 ] , VLM-3R [ 13 ] and SpaceMind [ 72 ] . These approaches demonstrate the value of geometric priors, but typically integrate spatial information through token-level fusion. Such designs enhance local spatial cues but do not explicitly reorganize observations into a persistent world-centered structure. Our work instead treats 3D geometry as the organizing substrate for constructing an allocentric cognitive map, enabling more stable spatial reasoning across changing viewpoints. 2.2 Feed-forward Visual Geometry for 3D Scene Understanding Classical 3D reconstruction pipelines, such as Structure-from-Motion [ 46 ] and Multi-View Stereo [ 47 , 53 , 68 ] , recover camera poses and dense geometry from images. However, their reliance on