Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
Ilias Stogiannidis, Steven McDonagh, Sotirios A. Tsaftaris · 2025
Read Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models by Ilias Stogiannidis, Steven McDonagh, Sotirios A. Tsaftaris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19707.txt.
Introduction
Ilias Stogiannidis, Steven McDonagh, and Sotirios Tsaftaris benchmark spatial reasoning in vision language models in a 2025 paper. Such models do well at captions and visual questions, but past tests mix spatial skill with object finding and word meaning. The team set out to isolate spatial skill on its own terms.
They break spatial reasoning into parts, such as relations between objects, orientation and navigation, mental rotation, and picturing space. Thirteen top models were tested on made up and real images. Average accuracy across the models sat near random chance. The authors call spatial reasoning a standing weakness of these models and offer their test suite as a base for future work.
Catalog record
| CREATOR | Ilias Stogiannidis, Steven McDonagh, Sotirios A. Tsaftaris |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19707
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models Ilias Stogiannidis, Steven McDonagh, Sotirios A. Tsaftaris Abstract Vision-Language Models (VLMs) have recently emerged as powerful tools, excelling in tasks that integrate visual and textual comprehension, such as image captioning, visual question answering, and image-text retrieval. However, existing benchmarks for VLMs include spatial components, which often fail to isolate spatial reasoning from related tasks such as object detection or semantic comprehension. In this paper, we address these deficiencies with a multi-faceted approach towards understanding spatial reasoning. Informed by the diverse and multi-dimensional nature of human spatial reasoning abilities, we present a detailed analysis that first delineates the core elements of spatial reasoning: spatial relations, orientation and navigation, mental rotation, and spatial visualization, and then assesses the performance of these models in both synthetic and real-world images, bridging controlled and naturalistic contexts. We analyze 13 state-of-the-art Vision-Language Models, uncovering pivotal insights into their spatial reasoning performance. Our results reveal profound shortcomings in current VLMs, with average accuracy across the 13 models approximating random chance, highlighting spatial reasoning as a persistent obstacle. This work not only exposes the pressing need to advance spatial reasoning within VLMs but also establishes a solid platform for future exploration. Code available on GitHub 1 1 1 https://github.com/stogiannidis/srbench and dataset available on HuggingFace . 2 2 2 https://huggingface.co/datasets/stogiannidis/srbench 1 Introduction Vision-Language Models (VLMs) have recently emerged as powerful tools, excelling in tasks that integrate visual and textual comprehension such as image captioning, visual question answering, and image-text retrieval [ 33 , 18 ] . Developed through extensive pretraining on diverse datasets, these models have demonstrated a remarkable ability to interpret complex interactions between visual information and language [ 47 ] . However, despite these advances, a critical capability remains largely unaddressed: spatial reasoning. Figure 1 : Example of VLM Responses to Mental Rotation Tasks: This example underscores a notable constraint of existing VLMs, which have difficulty in precisely understanding potential rotations of the objects shown, exposing a considerable deficiency in their spatial reasoning skills. Spatial reasoning involves comprehending and analyzing the locations, orientations, and relations of objects within a scene—skills that are natural to humans but pose significant challenges for deep learning models [ 51 , 9 , 11 ] . By fostering strong spatial intelligence, deep learning models can improve their ability to make intricate decisions that necessitate an understanding of geometric transformations and spatial context. While current benchmarks for VLMs incorporate tasks related to spatial reasoning, they often prioritize elements such as object detection or semantic interpretation [ 35 ] , leaving the core challenges of spatial cognition underexplored. Additionally, many datasets focus exclusively on specific aspects of spatial reasoning, sometimes augmenting visual data with extra elements such as point clouds, scene graphs, or depth maps to deepen spatial context [ 9 , 12 ] . The absence of a precise definition and structured evaluation approach for spatial reasoning in VLMs has hindered the creation of models capable of attaining human-level spatial reasoning. Humans naturally excel at spatial reasoning, a cognitive faculty that manifests in development and underpins our ability to engage with and navigate intricate environments [ 26 , 42 ] . This competence includes a variety of skills, from calculating distances and orienting things in three dimensions to combining various sensory inputs in a smooth manner. It has been established via rigorous empirical studies in cognitive science and neuroscience that our brain architecture is suited to process spatial relationships. These findings constantly confirm that spatial reasoning is deeply ingrained in our perceptual and motor systems and is not just essential for daily tasks [ 6 , 24 ] . By attempting to understand and reproduce these natural human abilities, we can shed light on the shortcomings of current VLMs and drive the development of computer systems that exhibit closer to human-like spatial awareness. Performant spatial reasoning is crucial beyond lab tests, enabling computational agents in fields such as robotics, autonomous navigation, and augmented reality to adapt to complex real-world environments [ 57 ] . It enhances VLMs for interpreting visual scenes anthropomorphically and drives advancements in technologies requiring precise spatial understanding. The present study aims to fill these gaps by a thorough and systematic analysis of spatial reasoning in the context of VLMs. We provide a thorough analysis that begins with the characterization of the basic components of spatial reasoning, namely spatial relations, orientation and navigation, mental rotation, and spatial visualization. After laying this conceptual foundation, we assess the models’ performance on a wide range of visual stimuli, including both artificially generated images and images from naturalistic environments. By performing a discrete evaluation of each component, we gain a more nuanced understanding of the unique advantages and disadvantages that exist in current VLMs and help us understand their operational capabilities. In summary, our contributions are as follows: • Precise Definition of Spatial Reasoning and its components: We formalize spatial reasoning in VLMs through key components: spatial relations (e.g., object positioning), orientation and navigation (e.g., directionality), mental rotation (e.g., invariant object recognition), and spatial visualization (e.g., transformation prediction). • Extensive Benchmark: We present a novel spatial reasoning benchmark, combining programmatically generated, GenAI-synthesized, and real-world images to bridge controlled evaluation and real-world applicability. • Component-Specific Evaluation: Our benchmark uniquely isolates spatial reasoning components, offering precise insights into VLMs’ strengths and limitations. • Extensive Model Evaluation: We assess 13 advanced VLMs, discovering that their typical spatial reasoning performance hovers around random chance, pointing out ongoing challenges. Figure 2 : Images from our benchmark created algorithmically, drawing inspiration from cognitive tests. Left image depicts paper folding, the middle shows the easy MRT version, and the right displays the hard variant. 2 Spatial Reasoning: Definition and Components Spatial reasoning is a fundamental cognitive ability concerned with understanding and manipulating spatial relationships between objects and oneself within a space [ 32 ] . It encompasses the mental skills involved in visualizing, transforming, and reasoning about spatial information. This ability is not monolithic but rather comprises several distinct yet interrelated components [ 40 ] . In computer science, spatial reasoning is increasingly relevant to fields such as robotics, computer vision, and geographic information systems, where algorithms and systems must effectively process and interpret spatial data [ 52 ] . Understanding these components is crucial for designing effective computational models of spatial intelligence. 2.1 Mental Rotation Mental rotation is a specific and well-studied aspect of spatial visualization, which refers to the ability to mentally rotate two-dimensional (2D) and three-dimensional (3D) objects [ 50 ] . It involves imagining an object as it would appear if rotated in space, and is often measured by tasks requiring individuals to determine if two objects are the same, but presented at different orientations [ 56 ] . Pellegrino et al. 1998 showed that mental rotation is a distinct spatial ability, separate from other spatial skills such as spatial perception. Mental rotation tasks are widely used in cognitive psychology to study spatial processing and have implications for understanding sex-related differences in spatial abilities, as well as neural substrates of spatial cognition [ 63 ] . While closely related to spatial visualization, mental rotation is often considered a more constrained and specific type of spatial transformation, focusing primarily