SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
Michael Ogezi, Freda Shi · 2025
Read SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data by Michael Ogezi, Freda Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20648.txt.
Introduction
Michael Ogezi and Freda Shi build SpaRE, a training set of made up visual questions that boosts space reasoning in vision language models. Such models ace captions and questions yet lag on space links that humans find easy. Their work came out in twenty twenty-five.
Space links are rare in common image text sets, with most in a long thin tail. The team built a vast set from rich image tales in three source sets, with millions of question pairs on space links. Models trained on it gained strongly on space test sets, with a top gain near half on one known test. Scores on general tasks held strong as well.
Catalog record
| CREATOR | Michael Ogezi, Freda Shi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data Michael Ogezi, Freda Shi Abstract Vision-language models (VLMs) work well in tasks ranging from image captioning to visual question answering (VQA), yet they struggle with spatial reasoning, a key skill for understanding our physical world that humans excel at. We find that spatial relations are generally rare in widely used VL datasets, with only a few being well represented while most form a long tail of underrepresented relations. This gap leaves VLMs ill-equipped to handle diverse spatial relationships. To bridge it, we construct a synthetic VQA dataset focused on spatial reasoning generated from hyper-detailed image descriptions in Localized Narratives, DOCCI, and PixMo-Cap. Our dataset consists of 455k samples containing 3.4 million QA pairs. Trained on this dataset, our Spatial-Reasoning Enhanced (SpaRE) VLMs show strong improvements on spatial reasoning benchmarks, achieving up to a 49% performance gain on the What’s Up benchmark, while maintaining strong results on general tasks. Our work narrows the gap between human and VLM spatial reasoning and makes VLMs more capable in real-world tasks such as robotics and navigation. We look to share our code and dataset in due course. 1 Introduction Figure 1: Our synthetic data generation pipeline: Hyper-detailed image descriptions are fed to an LLM that extracts spatial-reasoning question-answer (QA) pairs. Spatial reasoning—the ability to understand and interpret spatial relationships between objects—is a critical component of intelligent systems that interact with the physical world ( Newcombe et al., 2000 ) . Applications such as robotics, autonomous navigation, and extended reality rely heavily on precise spatial understanding to function effectively ( Landsiedel et al., 2017 ; Balakrishnan et al., 2021 ) . Without robust spatial reasoning, these systems risk misinterpreting their environments, leading to failures that could compromise safety and efficiency. Despite impressive advancements in vision-language models (VLMs) for tasks like image captioning, visual question answering (VQA), image-text retrieval, and zero-shot image classification, these models consistently struggle with spatial reasoning ( Kamath et al., 2023 ; Liu et al., 2023a ; Zhang et al., 2024 ) . For instance, VLMs may correctly identify objects in an image but fail to comprehend their spatial arrangement, which is crucial for tasks like scene understanding and navigation. Spatial relations are rare in existing vision-language datasets (see Table 2 ). Common relations (like on, left, and under) dominate, while less frequent ones (like facing, opposite, and surrounding) are severely underrepresented. In fact, the top 17% of relations make up over 90% of all spatial relation examples (see Table 10 in the appendix). This imbalance leaves VLMs poorly equipped to handle the full range of spatial relations. Previous efforts to address this gap have fallen short. Synthetic datasets ( Johnson et al., 2017 ; Agrawal et al., 2023 ) , while providing structured and controlled environments, rely on simplistic geometric shapes and synthetically generated images, which fail to generalize to real-world data. On the other hand, human-curated datasets ( Liu et al., 2023a ; Kamath et al., 2023 ) are limited in both quantity and diversity of spatial relations, leading to continued subpar performance. To bridge this gap, we present a novel approach that leverages the untapped potential of hyper-detailed captions from recent long-form image-captioning datasets. Datasets such as DOCCI ( Onoe et al., 2024 ) , PixMo-Cap ( Deitke et al., 2024 ) , and Localized Narratives ( Pont-Tuset et al., 2020 ) contain rich, detailed descriptions of images, often including explicit descriptions of spatial relationships and object interactions (see Figure 5 in the appendix). Armed with these resources, we use Qwen2.5-3B-Instruct Yang et al. (2024) 1 1 1 https://hf.co/Qwen/Qwen2.5-3B-Instruct to extract synthetic question-answer (QA) pairs focused on spatial relationships. Specifically, we extract spatial information from the hyper-detailed captions and formulate diverse and complex QA pairs that probe various aspects of spatial reasoning. Figure 1 illustrates our synthetic dataset generation method. By maintaining visual realism through the use of real-world images, our approach allows models to learn spatial reasoning skills in contexts they will encounter in practical applications, effectively addressing the domain gap introduced by synthetic visuals. We fine-tune VLMs on our synthesized dataset to produce Spa tial- R easoning E nhanced ( SpaRE ) VLMs. SpaRE VLMs significantly improve performance on spatial reasoning benchmarks, including VSR, What’s Up, 3DSRBench, and RealWorldQA. As shown in Table 3 , we achieve up to a 49% gain on the A split of What’s Up ( Kamath et al., 2023 ) , a benchmark designed to test spatial understanding. Importantly, these enhancements do not come at the expense of general VL performance. SpaRE models maintain their performance on standard benchmarks such as MMMU Yue et al. (2024) , and MMBench ( Liu et al., 2024b ) . This demonstrates that our method enhances spatial reasoning capabilities while preserving overall model effectiveness. By enhancing spatial reasoning in VLMs, our work supports systems that rely on accurate spatial understanding. This includes applications such as self-driving cars handling complex roads, robots operating alongside humans, and assistive technologies aiding visually impaired individuals in navigation. In summary, our contributions are threefold: 1. Quantifying Data Scarcity : We analyze spatial relations in current datasets and find that the top 17% account for about 90% of the samples, revealing a significant gap in representation. 2. Synthetic Spatial Data Generation : We develop a method to generate synthetic spatial reasoning QA pairs from hyper-detailed captions of over one million real-world images using advanced LLMs. 3. Enhancing VLM Spatial Reasoning : Our approach significantly improves VLMs’ spatial reasoning capabilities—by up to 49%—without compromising general VL task performance. The rest of the paper is organized as follows. In Section 2 , we review related work in spatial reasoning and VLMs. Section 3 details our approach to generating synthetic QA pairs and augmenting training data. We present our experiments and results, and a discussion in Section 4 . Finally, we conclude and briefly outline possible future research directions in Section 5 . 2 Background and Related Work 2.1 Spatial Reasoning Abilities in VLMs Spatial reasoning remains a challenge for VLMs. Liu et al. (2023a) introduced VSR, a dataset consisting of image-caption pairs where the binary task is to predict whether the caption accurately describes the spatial relations observed in the image. Their findings demonstrate that models consistently underperform on these tasks. Similarly, Kamath et al. (2023) highlighted that state-of-the-art VLMs struggle with basic spatial relations, performing near random on benchmarks designed to test understanding of concepts like left , right , above , and below . Furthermore, both Gokhale et al. (2023) and Cho et al. (2022) show that text-to-image generation models also struggle with producing images that faithfully represent spatial relations between multiple objects. While Chen et al. (2024a) advanced quantitative spatial reasoning through their data generation pipeline (i.e., predicting approximate distances between objects), qualitative spatial understanding remained unexplored. These findings indicate a significant gap in current models’ abilities to process and reason about spatial information. 2.2 Datasets for Spatial Reasoning Natural Data Popular high-quality, supervised-fine-tuning datasets which we analyze, such as TextCaps ( Sidorov et al., 2020 ) , ShareGPT-4o 2 2 2 https://sharegpt4o.github.io/ , InternVL-SA-1B-Caption 3 3 3 https://hf.co/datasets/OpenGVLab/InternVL-SA-1B-Caption , NewYorkerCaptionContest 4 4 4 https://hf.co/datasets/jmhessel/newyorker_caption_contest , MMInstruct ( Liu et al.,