Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna · 2026
Read Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA by Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12545.txt.
Introduction
Alam and co-authors ask why vision-language models stay brittle at basic spatial reasoning. Models ace general benchmarks yet stumble on relative position, layout, and counting. The team argues the failure ties to design choices, not just missing data. Prime suspects are CLIP-style encoders and the flattening of each image into a 1D token sequence. Their controlled study inside the LLaVA framework swaps encoders and adds 2D positional encoding. Spatial gaps persist across all variants, though encoder objectives and positional structure shape the behavior.
Catalog record
| CREATOR | Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13115 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12545
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna Abstract Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it. 1 Background and Motivation Modern vision-language models (VLMs) almost universally rely on large pre-trained image encoders such as CLIP and SigLIP ( 10 ; 25 ; 28 ; 22 ; 35 ; 32 ) . These encoders are trained primarily to align global image representations with text and are then integrated into systems such as Flamingo, LLaVA, BLIP-2, KOSMOS, Florence-2, and Molmo ( 2 ; 19 ; 17 ; 15 ; 24 ; 23 ; 34 ; 9 ) . While this paradigm has driven progress on captioning and VQA, it does not explicitly optimize for structured spatial representations. Prior work shows that CLIP-style encoders emphasize semantic alignment while underperforming on fine-grained and spatially grounded tasks ( 31 ; 3 ) . Recent encoders introduce denser or generative objectives ( 22 ; 20 ; 32 ; 11 ) , but their impact on spatial grounding in VLMs remains underexplored. Beyond the encoder, multimodal alignment introduces a second structural bottleneck. Most VLMs flatten images into 1D token sequences before applying 1D rotary positional encoding ( 27 ) , collapsing 2D structure during fusion. Recent analyses argue that this design undermines spatial reasoning even when strong visual features are available ( 36 ; 37 ) . While Qwen2-VL introduces multimodal rotary embeddings that preserve height and width information ( 33 ) , systematic evidence isolating the role of positional structure remains limited. At the evaluation level, spatial reasoning is rarely treated as a first-class capability. Foundational VLMs are primarily reported on general benchmarks, while spatial understanding is often omitted ( 2 ; 19 ; 15 ; 34 ) . Meanwhile, several benchmarks and datasets now explicitly target spatial reasoning, including MMVP, CV-Bench, GQA, VSR, TopViewRS, TallyQA, and CountBenchQA ( 31 ; 30 ; 12 ; 16 ; 14 ; 1 ; 5 ) , as well as training-driven efforts such as RoboSpatial, SpatialVLM, MM-Spatial, and SpatialRGPT ( 26 ; 6 ; 8 ; 7 ) . However, these efforts often conflate data, scale, and architecture, making it difficult to isolate which design factors shape spatial behavior. In this work, we focus on static 2D spatial reasoning and present a controlled diagnostic study within the LLaVA framework. We evaluate frontier VLMs and systematically vary two under-examined design dimensions: the image encoder objective and the positional structure used during multimodal alignment, introducing 2D rotary positional encoding. This setting allows us to isolate how these factors influence spatial grounding and to assess the extent to which they account for observed spatial failures. 2 Experimental Setup 2.1 Methods All experiments are conducted within the LLaVA ( 19 ) framework. We construct controlled LLaVA-1.5 (7B) variants by swapping the image encoder while holding the language backbone and training pipeline fixed. Specifically, we compare CLIP, SigLIP, SigLIP2, and AIMv2, and evaluate each with and without 2D rotary positional encoding (2D-RoPE). Unlike standard 1D RoPE, 2D-RoPE encodes both horizontal and vertical patch indices and is applied to query and key projections during multimodal attention. This preserves explicit 2D structure during image–text fusion and allows us to isolate the effect of positional structure on spatial grounding. Large Language Model f_{\phi} Image \mathbf{X}_{v} Image Encoder (CLIP/SigLIP/SigLIP2/AIMv2) Projection \mathbf{W} \mathbf{X}_{a} : Language Response \mathbf{H}_{v} \mathbf{H}_{q} Language Instruction \mathbf{X}_{q} Figure 1: Our experimental approach with LLaVA Framework 19 that compares the performance of different image encoders and 2D-RoPE variants. 2.2 Training Models are trained using the standard two-stage LLaVA recipe: projection pretraining followed by full instruction tuning, using the original LLaVA datasets. Images are resized to 256\times 256 . Pretraining updates only the projection layer, while instruction tuning updates all parameters. Full training and optimization details are provided in the appendix. 3 Results We evaluate frontier multimodal models and controlled LLaVA-1.5 (7B) variants across spatial reasoning benchmarks. All encoder-swapped models and their 2D-RoPE counterparts are trained on the same 7B LLaVA backbone for controlled comparison. Frontier models between 2B and 8B parameters include LLaVA-NeXT ( 18 ) , LLaVA-OneVision-qwen2-7B-ov-hf ( 13 ) , Qwen2.5-VL-8B ( 4 ) , SmolVLM2-2.2B-Instruct ( 21 ) , Gemma3-4b-it ( 29 ) , PaliGemma2-3b-mix-448 ( 5 ) , and Molmo-7B-D-0924 ( 9 ) . Table 1: Comparison of frontier models and LLaVA variants across spatial understanding benchmarks. Values underlined indicate the best-performing frontier model; values in bold indicate the best-performing LLaVA variant. Models MMVP CV-Bench 2D Overall TallyQA GQA Overall VSR Top- ViewRS Count- BenchQA LLaVA-NeXT 0.667 0.606 0.733 63.786 63.994 0.409 0.515 LLaVA-OneVision 0.767 0.730 0.797 62.140 77.741 0.414 0.823 Qwen2.5-VL 0.770 0.754 0.800 60.391 89.116 0.456 0.891 SmolVLM2 0.687 0.577 0.729 50.574 71.277 0.416 0.692 Gemma3-4b-it 0.708 0.659 0.525 31.277 55.074 0.334 0.713 PaliGemma 0.667 0.624 0.794 62.570 65.139 0.322 0.674 Molmo 0.753 0.728 0.808 55.295 76.432 0.323 0.858 LLaVA v1.5 0.577 0.490 0.707 33.225 55.810 0.384 0.468 LLaVA-2D-RoPE 0.513 0.443 0.654 34.433 57.201 0.283 0.290 LLaVA-SigLIP 0.433 0.412 0.672 25.648 54.910 0.349 0.581 LLaVA-SigLIP-2D-RoPE 0.507 0.425 0.616 38.448 57.692 0.295 0.483 LLaVA-SigLIP2 0.427 0.442 0.684 23.970 52.701 0.371 0.532 LLaVA-SigLIP2-2D-RoPE 0.480 0.415 0.646 34.560 56.465 0.330 0.402 LLaVA-AIMv2 0.513 0.466 0.710 32.541 56.219 0.339 0.739 LLaVA-AIMv2-2D-RoPE 0.560 0.432 0.690 32.342 60.311 0.338 0.719 3.1 Frontier models still fail on spatial reasoning Table 1 shows that Qwen2.5-VL is the strongest frontier model overall, leading on CV-Bench 2D Overall, MMVP, VSR, TopViewRS, and CountBenchQA, while LLaVA-NeXT leads frontier models on GQA Overall and Molmo leads on TallyQA. Despite these gains, spatial performance remains uneven across tasks and models, indicating that spatial grounding is not consistently captured by general-purpose training and scaling. 3.2 Encoder choice dominates spatial performance in controlled LLaVA variants Within the controlled LLaVA setting, we observe that the choice of image encoder strongly shapes spatial behavior. LLaVA-AIMv2 yields the most consistent improvements over the CLIP-based LLaVA baseline, achieving the best LLaVA scores on CV-Bench 2D Overall, TallyQA, and CountBenchQA. LLaVA-AIMv2-2D-RoPE further improves MMVP and achieves the best LLaVA score on VSR. In contrast, SigLIP- and SigLIP2-based variants show more fragmented gains, with LLaVA-SigLIP-2D-RoPE performing best on GQA Overall and LLaVA-SigLIP2 performing best on TopViewRS. Overall, these results suggest that spatial failures in VLMs are strongly