SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning

Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan, [0.4em] TAMU UCLA Google UW-Madison [0.3em] https://spatial-stack.github.io/ · 2026

Read SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning by Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan, [0.4em] TAMU UCLA Google UW-Madison [0.3em] https://spatial-stack.github.io/ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27437.txt.

Introduction

Zhang and co-authors improve how vision-language models judge 3D space. Most models fuse shape data only at deep layers. That habit wastes rich cues from early layers. SpatialStack aligns sight, shape, and words at many levels. Local shape detail and broad scene sense both survive. Their model, VLM-SpatialStack, tops several 3D reasoning tests. Extra checks show the layered fusion helps across tasks. The method also transfers well to new scene types.

Catalog record

CREATORJiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan, [0.4em] TAMU UCLA Google UW-Madison [0.3em] https://spatial-stack.github.io/
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57187
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27437

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan, [0.4em] TAMU UCLA Google UW-Madison [0.3em] https://spatial-stack.github.io/ Abstract Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems. This limitation arises from their inability to capture fine-grained 3D geometry and spatial relationships. While recent efforts have introduced multi-view geometry transformers into VLMs, they typically fuse only the deep-layer features from vision and geometry encoders, discarding rich hierarchical signals and creating a fundamental bottleneck for spatial understanding. To overcome this, we propose SpatialStack, a general hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy. Moving beyond conventional late-stage vision-geometry fusion, SpatialStack stacks and synchronizes multi-level geometric features with the language backbone, enabling the model to capture both local geometric precision and global contextual semantics. Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks. Extensive experiments and ablations demonstrate that our multi-level fusion strategy consistently enhances 3D understanding and generalizes robustly across diverse spatial reasoning tasks, establishing SpatialStack as an effective and extensible design paradigm for vision-language-geometry integration in next-generation multimodal physical AI systems. Figure 1 : SpatialStack: Layered Geometry-Language Fusion. Conventional VLMs (a) fuse only a single deep geometry feature with vision tokens, which limits both fine-grained spatial understanding and high-level spatial reasoning. SpatialStack (b) instead stacks multi-level geometry features and injects them hierarchically into successive LLM decoder layers, yielding stronger 3D spatial understanding across benchmarks. 1 Introduction Understanding and reasoning about physical space are fundamental capabilities for any intelligent system that aims to perceive, communicate, and act in the physical world. Motivated by this, recent work on spatial reasoning aims to enable embodied agents to interpret scene layouts, predict interactions, and plan actions in 3D environments, forming a cognitive bridge between perception and action 13 ; 50 ; 60 ; 35 ; 52 ; 53 . Despite remarkable progress in large vision-language models (VLMs), reliable spatial reasoning remains challenging, as these models often fail to effectively encode 3D geometry and spatial relationships and to associate them with language instructions, which are essential for everyday spatial tasks that require both low-level and high-level reasoning. For instance, they struggle to estimate relative distances in static scenes 51 ; 28 and cannot reliably distinguish “left” from “right” when reasoning about motion in dynamic environments 65 . In embodied AI applications such as robotic navigation, manipulation, and spatial assistance under XR, such limitations prevent VLMs from grounding their understanding in the complex and dynamic physical world. Noticing these limitations of conventional VLMs, many recent works still prioritize image-level semantic alignment over the understanding of spatial and geometric structures 42 ; 37 ; 22 . Bridging this gap requires unifying geometric awareness with vision-language reasoning within a single framework, which is a key step toward reliable spatial intelligence. This naturally raises a fundamental question: How can vision–language–geometry be effectively unified in VLMs to enable reliable spatial reasoning? An initial line of work sought to compensate for these weaknesses by integrating explicit geometric inputs (e.g., pre-computed point clouds or depth maps) into VLMs. For instance, early models like 3D-LLM 16 and LEO 17 used external point cloud encoders, while later methods like LLaVA-3D 66 and Video-3D LLM 61 introduced lightweight encoders for RGB-D fusion. However, the reliance on these external, pre-processed inputs significantly limits their applicability. In parallel, rapid advancements in end-to-end multi-view geometry transformers, including DUST3R 48 , CUT3R 47 , and VGGT 46 , have provided a more unified and powerful alternative to map uncalibrated images to 3D point maps. These models can infer rich geometric attributes such as depth, camera pose, and 3D structure directly from multi-view images, thereby bypassing traditional, computationally expensive geometric pipelines (e.g., Structure-from-Motion 41 ). Inspired by this progress, recent multimodal models, such as Spatial-MLLM 50 , VLM-3R 13 , and VG-LLM 60 , have begun integrating these geometry encoders into VLM frameworks, showing initial promise in improving spatial reasoning. Nevertheless, most of these integrations focus only on fusing the final-layer features of geometry transformers with features from vision encoders. This is a critical limitation, as many geometry encoders adopt the DPT architecture 39 , which explicitly extracts multi-level representations from different transformer layers to recover detailed geometric information. At the same time, a generalizable spatial-visual fusion mechanism has to account for hierarchical real-world tasks, ranging from low-level depth estimation and surface reconstruction to high-level relational reasoning and goal-directed planning. By sampling only the last layer, existing models discard the rich hierarchical geometric cues embedded in intermediate layers and overlook how different levels of geometric and semantic features contribute to spatial reasoning. Unsurprisingly, this single-level fusion design can improve performance on specific spatial benchmarks but creates a bottleneck that fundamentally constrains 3D understanding. In this paper, we are motivated by the hierarchical nature of spatial reasoning tasks in 3D environments, and we systematically study how fusion layers across vision encoders, geometry encoders, and large language model (LLM) decoders affect multimodal spatial reasoning. Our analysis first shows that geometry-language fusion in multimodal LLMs follows a hierarchical pattern similar to vision encoding: shallow features enhance fine-grained spatial perception, while deeper features support high-level contextual reasoning. Building on these insights, we introduce SpatialStack, a general hierarchical fusion framework that integrates multi-level geometric features into multimodal LLMs. As shown in Fig. 1 , unlike prior methods that fuse geometry only at deep encoder layers, SpatialStack progressively aligns geometric and language representations throughout the model hierarchy, capturing both detailed local geometry and global semantic context. Extensive experiments on multiple benchmarks demonstrate that our approach significantly improves 3D spatial reasoning, achieving strong performance on tasks requiring both detailed perception and holistic spatial understanding. We summarize our contributions as follows: • We present the first systematic analysis of how fusion layers across vision encoders, geometry encoders, and LLM decoders affect the granularity of spatial reasoning. Our layer-wise study reveals a hierarchical geometry–language correspondence, where shallow layers capture fine spatial details and deeper layers encode global structure and context. • We propose SpatialStack , a general hierarchical fusion framework that progressively aligns multi-level geometric and language features. This design goes beyond conventional final-stage vision-language fusion and supports joint reasoning over local and global spatial context. • While SpatialStack is model-agnostic and can be applied to any base multimodal LLM, we develop VLM-SpatialStack as a concrete realization using the Qwen series. Extensive experiments and ablation studies across multiple benchmarks show that SpatialStack achieves state-of-the-art performance and strong generalization on diverse 3D spatial reasoning tasks. 2 Related Work Figure 2 : Architecture of SpatialStack. A standard VLM backbone is coupled with a multi-view geometry encoder whose layer-wise features are processed by layer-specific projectors and sequentially injected into the LLM decoder, progressively integrating geometric cues. Explanation of the similarity heatmaps on the left is provided in Sec. 3 . This multi-level injection preserves both fine-grained geometric structure and high-level spatial context, supporting more reliable low-level understanding and high-level reasoning. Large Multimodal Models (MLLMs) Early works such as CLIP 38 demonstrated the efficacy of learning joint vision-language representations from web-scale image-text pairs through contrastive pre-training. This

Read the complete text