SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan, [0.4em] TAMU UCLA Google UW-Madison [0.3em] https://spatial-stack.github.io/ Abstract Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems. This limitation arises from their inability to capture fine-grained 3D geometry and spatial relationships. While recent efforts have introduced multi-view geometry transformers into VLMs, they typically fuse only the deep-layer features from vision and geometry encoders, discarding rich hierarchical signals and creating a fundamental bottleneck for spatial understanding. To overcome this, we propose SpatialStack, a general hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy. Moving beyond conventional late-stage vision-geometry fusion, SpatialStack stacks and synchronizes multi-level geometric features with the language backbone, enabling the model to capture both local geometric precision and global contextual semantics. Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks. Extensive experiments and ablations demonstrate that our multi-level fusion strategy consistently enhances 3D understanding and generalizes robustly across diverse spatial reasoning tasks, establishing SpatialStack as an effective and extensible design paradigm for vision-language-geometry integration in next-generation multimodal physical AI systems. Figure 1 : SpatialStack: Layered Geometry-Language Fusion. Conventional VLMs (a) fuse only a single deep geometry feature with vision tokens, which limits both fine-grained spatial understanding and high-level spatial reasoning. SpatialStack (b) instead stacks multi-level geometry features and injects them hierarchically into successive LLM decoder layers, yielding stronger 3D spatial understanding across benchmarks. 1 Introduction Understanding and reasoning about physical space are fundamental capabilities for any intelligent system that aims to perceive, communicate, and act in the physical world. Motivated by this, recent work on spatial reasoning aims to enable embodied agents to interpret scene layouts, predict interactions, and plan actions in 3D environments, forming a cognitive bridge between perception and action 13 ; 50 ; 60 ; 35 ; 52 ; 53 . Despite remarkable progress in large vision-language models (VLMs), reliable spatial reasoning remains challenging, as these models often fail to effectively encode 3D geometry and spatial relationships and to associate them with language instructions, which are essential for everyday spatial tasks that require both low-level and high-level reasoning. For instance, they struggle to estimate relative distances in static scenes 51 ; 28 and cannot reliably distinguish “left” from “right” when reasoning about motion in dynamic environments 65 . In embodied AI applications such as robotic navigation, manipulation, and spatial assistance under XR, such limitations prevent VLMs from grounding their understanding in the complex and dynamic physical world. Noticing these limitations of conventional VLMs, many recent works still prioritize image-level semantic alignment over the understanding of spatial and geometric structures 42 ; 37 ; 22 . Bridging this gap requires unifying geometric awareness with vision-language reasoning within a single framework, which is a key step toward reliable spatial intelligence. This naturally raises a fundamental question: How can vision–language–geometry be effectively unified in VLMs to enable reliable spatial reasoning? An initial line of work sought to compensate for these weaknesses by integrating explicit geometric inputs (e.g., pre-computed point clouds or depth maps) into VLMs. For instance, early models like 3D-LLM 16 and LEO 17 used external point cloud encoders, while later methods like LLaVA-3D 66 and Video-3D LLM 61 introduced lightweight encoders for RGB-D fusion. However, the reliance on these external, pre-processed inputs significantly limits their applicability. In parallel, rapid advancements in end-to-end multi-view geometry transformers, including DUST3R 48 , CUT3R 47 , and VGGT 46 , have provided a more unified and powerful alternative to map uncalibrated images to 3D point maps. These models can infer rich geometric attributes such as depth, camera pose, and 3D structure directly from multi-view images, thereby bypassing traditional, computationally expensive geometric pipelines (e.g., Structure-from-Motion 41 ). Inspired by this progress, recent multimodal models, such as Spatial-MLLM 50 , VLM-3R 13 , and VG-LLM 60 , have begun integrating these geometry encoders into VLM frameworks, showing initial promise in improving spatial reasoning. Nevertheless, most of these integrations focus only on fusing the final-layer features of geometry transformers with features from vision encoders. This is a critical limitation, as many geometry encoders adopt the DPT architecture 39 , which explicitly extracts multi-level representations from different transformer layers to recover detailed geometric information. At the same time, a generalizable spatial-visual fusion mechanism has to account for hierarchical real-world tasks, ranging from low-level depth estimation and surface reconstruction to high-level relational reasoning and goal-directed planning. By sampling only the last layer, existing models discard the rich hierarchical geometric cues embedded in intermediate layers and overlook how different levels of geometric and semantic features contribute to spatial reasoning. Unsurprisingly, this single-level fusion design can improve performance on specific spatial benchmarks but creates a bottleneck that fundamentally constrains 3D understanding. In this paper, we are motivated by the hierarchical nature of spatial reasoning tasks in 3D environments, and we systematically study how fusion layers across vision encoders, geometry encoders, and large language model (LLM) decoders affect multimodal spatial reasoning. Our analysis first shows that geometry-language fusion in multimodal LLMs follows a hierarchical pattern similar to vision encoding: shallow features enhance fine-grained spatial perception, while deeper features support high-level contextual reasoning. Building on these insights, we introduce SpatialStack, a general hierarchical fusion framework that integrates multi-level geometric features into multimodal LLMs. As shown in Fig. 1 , unlike prior methods that fuse geometry only at deep encoder layers, SpatialStack progressively aligns geometric and language representations throughout the model hierarchy, capturing both detailed local geometry and global semantic context. Extensive experiments on multiple benchmarks demonstrate that our approach significantly improves 3D spatial reasoning, achieving strong performance on tasks requiring both detailed perception and holistic spatial understanding. We summarize our contributions as follows: • We present the first systematic analysis of how fusion layers across vision encoders, geometry encoders, and LLM decoders affect the granularity of spatial reasoning. Our layer-wise study reveals a hierarchical geometry–language correspondence, where shallow layers capture fine spatial details and deeper layers encode global structure and context. • We propose SpatialStack , a general hierarchical fusion framework that progressively aligns multi-level geometric and language features. This design goes beyond conventional final-stage vision-language fusion and supports joint reasoning over local and global spatial context. • While SpatialStack is model-agnostic and can be applied to any base multimodal LLM, we develop VLM-SpatialStack as a concrete realization using the Qwen series. Extensive experiments and ablation studies across multiple benchmarks show that SpatialStack achieves state-of-the-art performance and strong generalization on diverse 3D spatial reasoning tasks. 2 Related Work Figure 2 : Architecture of SpatialStack. A standard VLM backbone is coupled with a multi-view geometry encoder whose layer-wise features are processed by layer-specific projectors and sequentially injected into the LLM decoder, progressively integrating geometric cues. Explanation of the similarity heatmaps on the left is provided in Sec. 3 . This multi-level injection preserves both fine-grained geometric structure and high-level spatial context, supporting more reliable low-level understanding and high-level reasoning. Large Multimodal Models (MLLMs) Early works such as CLIP 38 demonstrated the efficacy of learning joint vision-language representations from web-scale image-text pairs through contrastive pre-training. This paradigm was extended by subsequent models like Flamingo 1 , which bridged powerful pre-trained vision and language models, and the BLIP series 26 ; 25 , which introduced bootstrapping methods and lightweight querying transformers to unify understanding and generation. A significant advancement in MLLM development was the advent of visual instruction tuning, effectively employed by models such as InstructBLIP 11 to enhance instruction-following capabilities. Models like Qwen2.5-VL 2 , LLaVA 31 and MiniGPT-4 67 popularized a simple and effective architecture for this tuning, connecting a pre-trained vision encoder to a large language model (LLM) using only a simple projection layer. This simple design has spurred research into more effective fusion strategies, such as exploring different visual encoders 21 or deeper token stacking 36 . This architectural blueprint, while powerful for general-purpose multimodal chat and semantic understanding, established a paradigm of fusing only the final-layer visual features with the language backbone. Consequently, as noted by recent analyses 22 ; 37 ; 42 , these models are trained primarily for semantic alignment and often fail to capture the fine-grained spatial and geometric structures essential for physical reasoning. Spatial Reasoning in Vision-Language Models The limitations of standard MLLMs in spatial reasoning have been well-documented, prompting recent efforts to quantify these deficiencies through benchmarks like VSI Bench 51 , Spar Bench 56 , BLINK 15 , and Cambrian-1 45 . Cambrian-S 53 further demonstrated a lack of “implicit 3D spatial cognition”, while VLM4D 65 was the first to highlight the challenges of spatiotemporal (4D) reasoning in dynamic scenarios, followed by more recent efforts on dynamic 4D understanding and world modeling 18 ; 49 . To address these gaps, one line of work focused on injecting explicit 3D data, such as 3D-LLM 16 which processes point clouds, or more lightweight approaches like LLaVA-3D 66 and Video-3D LLM 61 that endow MLLMs with 3D awareness, with applications in embodied tasks 17 . A different strategy enhanced spatial abilities through novel training paradigms. Spatial-SSRL 35 introduced a self-supervised reinforcement learning framework, and Visual Spatial Tuning (VST) 52 proposed a comprehensive tuning framework with a large-scale dataset (VST-P) and a progressive pipeline. These latter methods enhance spatial intelligence but primarily focus on training objectives and data augmentation rather than the core fusion architecture. Vision-Language-Geometry Fusion The integration of explicit geometric reasoning within MLLMs has been recently catalyzed by the advent of powerful, feed-forward geometry encoders. Models such as DUST3R series 48 ; 23 and CUT3R 47 can infer dense, consistent point maps from unposed multi-view images, while VGGT 46 introduced a unified transformer to predict diverse 3D attributes from video. The availability of these rich geometric features has inspired two parallel fusion paradigms. One line of work focuses on building explicit spatial semantic representations, such as methods that distill 2D image or video foundation model features into 3D or 4D explicit feature field representations 63 ; 44 ; 64 , build queryable 3D world models by fusing pixel-aligned features into 3D maps 20 , or map images directly to semantic radiance fields 12 . A parallel approach, which our work follows, implicitly fuses geometric priors into the latent space of the MLLM. Recent models have shown initial promise in this direction: Spatial-MLLM 50 employs a dual-encoder architecture, VG-LLM 60 fuses features at the patch level, VLM-3R 13 introduces a cross-attention mechanism, and SSR 32 focuses on rationale-guided fusion. However, as identified in our analysis, these integrations typically fuse only the final-layer features from the geometry and vision encoders 50 ; 60 . This single-level fusion design discards the rich, hierarchical geometric cues embedded in intermediate layers, creating a fundamental bottleneck for fine-grained spatial reasoning. Our work, SpatialStack, directly addresses this limitation by introducing a hierarchical fusion framework that progressively aligns multi-level geometry features with the language backbone. 3 How Multi-level Geometry Features Facilitate Spatial Reasoning Qualitative Analysis To validate our motivation, we begin by examining why relying solely on deep-layer geometry features is insufficient. As illustrated in Fig. 2 , we take one input view and unflatten the tokens from different layers of the geometry encoder back into their original H\times W spatial layout. We then select a small patch (red bounding box) as the region of interest (ROI) and compute patch-wise similarity maps between the ROI and all other spatial locations: brighter regions indicate higher similarity, and darker regions indicate lower similarity. We observe a clear trend: shallow layers retain sharp local structures and well-defined geometric boundaries, while deeper layers produce overly homogeneous activations, where many regions appear similar in latent space despite having distinct physical geometry. This mismatch suggests that deep geometry features lose fine-grained spatial cues critical for reasoning about scene layout and spatial relations. These findings motivate our approach: leveraging multi-level geometry features, especially shallow-layer cues, to enrich spatial grounding and improve fine-grained 3D spatial reasoning in VLMs. Figure 3 : Examples of spatial tasks at different levels. The left example ( Low-Level Task ) targets fine-grained geometric perception, such as determining which of two points is closer to the camera. The right example ( High-Level Task ) requires higher-level spatial reasoning, where the model must estimate the distance between two objects by comparing their closest points in 3D space. Quantitative Analysis We further investigate how geometric features from different layers influence spatial reasoning performance. Firstly, we follow the difficulty hierarchy defined in SPAR 56 dataset, which categorizes spatial tasks based on the required complexity of spatial understanding. SPAR divides spatial tasks into three cognitive levels: perception (low) , reasoning (medium) , and imagination (high) . Low-level tasks emphasize fundamental geometric perception, such as single-view depth estimation and distance comparison based on local pixel/feature cues; high-level tasks require aggregating spatial information across multiple viewpoints for global spatial reasoning, such as cross-view object spatial relations and path reasoning (see Fig. 3 ). Based on this criterion, our low-level tasks evaluating fundamental perception include BLINK’s relative depth 15 and specific tasks in SPAR-Bench 56 : depth ( Depth-OC/OO/OC-MV/OO-MV ) and absolute distance ( Dist-OC/OO/OC-MV/OO-MV ) (see 56 for more details). Conversely, all VSI-Bench tasks are categorized as high-level tasks , as they require complex multi-view spatial fusion and 3D relational reasoning. We do not include SPAR’s medium or high tasks in this study, as we aim to establish a clearer two-level comparison that isolates the distinct effects of geometric feature integration on basic perception ability versus complex spatial reasoning ability . Under the task definitions above, we further conduct a quantitative analysis of the performance impact of injecting geometric features from different layers into VLMs. Specifically, following VG-LLM 60 , we extract geometric features from a single layer of the geometry encoder (VGGT 46 ), project them through a projector, and add them to the last-layer features of the vision encoder. We denote this geometry-vision fusion as GVF in the rest of our paper. The fused geometry-vision features are then concatenated with text tokens and fed into the LLM decoder. We experiment with injecting geometric features from the 4th, 11th, 17th, and 23rd layers, and evaluate the performance on the two task levels. As shown in Fig. 4 , the results demonstrate that the choice of injection layer has a significant impact on different levels of spatial tasks: as the injection layer becomes deeper, the performance on low-level tasks declines, while the performance on high-level tasks improves significantly. This phenomenon suggests that geometric features from different layers play distinct roles in spatial understanding: features from shallower layers provide fine-grained local geometric cues beneficial for basic spatial perception, whereas deeper features encode more global structural and semantic relationships, making them more suitable for complex spatial reasoning. Given the complementary strengths of shallow and deep features, a multi-layer fusion strategy should intuitively enhance both perception and reasoning. To test this, Table 1 compares various fusion strategies against Qwen3.5 43 , the base model fine-tuned on our spatial reasoning datasets without any geometric enhancements. Surprisingly, naive multi-layer fusion fails to achieve the best. Instead, it yields a compromised performance, falling behind the 11th-layer single-fusion on low-level tasks and the 23rd-layer on high-level tasks. This sub-optimal outcome reveals that simply adding hierarchical features into the vision pathway causes feature interference rather than synergy. This highlights that merely extracting multi-level cues is insufficient; the true challenge lies in the fusion strategy—a realization that serves as the primary catalyst for SpatialStack. Figure 4 : Effect of Geometry Injection Layers on Spatial Tasks. Deeper layers improve high-level tasks, while low-level tasks peak at layer 11 and decline at deeper layers, suggesting a trade-off between fine-grained perception and higher-level reasoning. Model Low-Level Avg High-Level Avg Overall Qwen3.5 (fine-tuned) 61.37 64.76 63.07 Single-layer (geo enc: 11) 66.11 64.48 65.30 Single-layer (geo enc: 17) 63.89 65.76 64.83 Single-layer (geo enc: 23) 64.33 66.36 65.35 Multi-Layer Fusion 64.69 65.15 64.92 Table 1 : Ablation Results on Geometry Token Fusion Depth. Simply fusing multi-layer geometry features to the visual features yields suboptimal performance, while selecting an appropriate single geometry encoder layer achieves better task-specific trade-offs. 4 Where to fuse Multi-level Geometry Features The observation of feature interference during naive vision-pathway fusion in Section 3 prompts a critical question: where and how should these hierarchical geometric features be integrated to maximally enhance a VLM’s spatial reasoning? Should they be confined to the vision encoder, or directly injected into the language model? 4.1 SpatialStack: Geometry-Language Fusion While most prior works 60 ; 50 ; 13 confine geometric enhancements to the vision encoder, we hypothesize that injecting these features directly into the Large Language Model (LLM) provides a more flexible, high-capacity space for multi-scale spatial reasoning. Inspired by DeepStack’s 36 success in stacking visual tokens within the LLM, we shift geometry integration to the language side and propose SpatialStack: a novel, first-of-its-kind layered geometry–language fusion framework. As shown in Fig. 2 , SpatialStack performs multi-level fusion between a geometry encoder and an LLM decoder. Its key idea is to inject geometric features from multiple layers of the geometry encoder into corresponding layers of the LLM, forming a hierarchy of geometric representations. This progressive stacking introduces geometric cues throughout the decoding process, strengthening spatial grounding and improving reasoning across tasks of varying difficulty. Importantly, SpatialStack is a general framework that can be integrated with any open-source VLM. We instantiate VLM-SpatialStack using the latest Qwen3.5 43 as our primary base model. To ensure a fair comparison with existing baselines 50 ; 60 ; 53 , we also provide an instantiation based on the same base model they use, Qwen2.5-VL 2 . 4.2 VLM-SpatialStack VLM Architecture. Given K input frames \{\mathbf{I}_{k}\in\mathbb{R}^{H\times W\times 3}\}_{k=1}^{K} , each frame is encoded by a shared vision encoder into tokens \mathbf{V}_{k}\in\mathbb{R}^{N\times D_{\text{vis}}} , where p is the patch size and N=\tfrac{H}{p}\!\times\!\tfrac{W}{p} is the number of patch tokens per frame. A spatial merger groups every 2\times 2 neighboring patches (stride factor s=2 ), producing \tilde{\mathbf{V}}_{k}\in\mathbb{R}^{N^{\prime}\times D_{\text{lang}}} with N^{\prime}=\tfrac{N}{s^{2}}=\tfrac{HW}{(ps)^{2}} . Merged tokens from all frames are concatenated along the sequence dimension: \tilde{\mathbf{V}}=[\,\tilde{\mathbf{V}}_{1};\ldots;\tilde{\mathbf{V}}_{K}\,]\in\mathbb{R}^{(KN^{\prime})\times D_{\text{lang}}}. The concatenated visual tokens \tilde{\mathbf{V}} and text tokens \mathbf{T}\in\mathbb{R}^{M\times D_{\text{lang}}} form the multimodal input sequence \mathbf{H}_{0}=[\,\tilde{\mathbf{V}};\mathbf{T}\,] . This sequence is then processed by L stacked transformer layers in the LLM decoder: \mathbf{H}_{L}^{\text{llm}}=f_{L}^{\text{llm}}\!\Big(f_{L-1}^{\text{llm}}\!\big(\cdots f_{1}^{\text{llm}}(\mathbf{H}_{0})\big)\Big), (1) where \mathbf{H}_{L}^{\text{llm}} denotes the final hidden representations produced by the LLM decoder for downstream prediction. Geometry Encoder. We employ the Visual Geometry Grounded Transformer (VGGT) 46 as our geometry encoder. Given the same set of K input images \{\mathbf{I}_{k}\in\mathbb{R}^{H\times W\times 3}\}_{k=1}^{K} , each image is divided into non-overlapping patches of size p\times p , resulting in N=(H/p)\!\times\!(W/p) patch tokens. In addition to the patch tokens, VGGT includes camera and register tokens to encode view-specific and shared geometric context. The initial token sequence for view k is thus \mathbf{Z}_{0}^{(k)}=[\,\mathbf{c}_{k};\,\mathbf{r}_{k};\,\mathbf{p}_{k}\,]\in\mathbb{R}^{(1+R+N)\times D_{\text{geo}}}, (2) where \mathbf{p}_{k} denotes the patch tokens of image \mathbf{I}_{k} . All view-specific sequences are concatenated and jointly processed by L stacked transformer layers: \mathbf{Z}_{L}=f_{L}^{\text{geo}}\!\Big(f_{L-1}^{\text{geo}}\!\big(\cdots f_{1}^{\text{geo}}([\mathbf{Z}_{0}^{(1)};\ldots;\mathbf{Z}_{0}^{(K)}])\big)\Big), (3) where f_{l}^{\text{geo}}(\cdot) denotes the l -th transformer layer in VGGT. While the original VGGT employs Dense Prediction Transformer (DPT) heads 39 for outputs such as depth, point clouds, and camera parameters, we instead extract intermediate hidden states \mathbf{Z}_{l} from selected layers as multi-view geometric features for fusion with the vision–language model. Layered Geometry–Language Fusion. As illustrated in Fig. 2 , we extract multi-level patch features \mathbf{Z}_{l_{i}} from the geometry encoder defined in Eq. 3 . Specifically, we take the patch-token outputs of the l_{i} -th layers ( l_{i}\in\{11,17,23\} , counted from zero) of VGGT after removing camera and register tokens, thereby yielding \mathbf{Z}_{l_{i}}\in\mathbb{R}^{(KN)\times D_{\text{geo}}} that represent geometric information at different, progressively richer abstraction levels. Each feature \mathbf{Z}_{l_{i}} is processed by a layer-specific geometry token merger \mathcal{M}_{\text{geo}}^{(l_{i})} to align its spatial resolution and embedding dimension with that of \mathbf{H} : \mathbf{G}_{l_{i}}=\mathcal{M}_{\text{geo}}^{(l_{i})}(\mathbf{Z}_{l_{i}}),\hskip 10.00002pt\mathbf{G}_{l_{i}}\in\mathbb{R}^{N^{\prime}\times D_{\text{lang}}}. (4) Finally, the geometry features \{\mathbf{G}_{l_{1}},\mathbf{G}_{l_{2}},\mathbf{G}_{l_{3}}\} extracted from VGGT layers \{11,17,23\} are injected into LLM decoder layers \{0,1,2\} as additive residuals: \mathbf{H}^{(j)^{\prime}}=\mathbf{H}^{(j)}+\mathbf{G}_{l_{j}},\hskip 10.00002ptj\in\{0,1,2\}. (5) Optimization. We train the entire model under a single objective, the next-token negative log-likelihood (cross-entropy): \mathcal{L}_{\text{ce}}(\theta)=-\sum_{i=1}^{|o|}\log P_{\theta}\!\big(o^{(i)}\,\big|\,o^{(