Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models
Akshar Tumu, Varad Shinde, Parisa Kordjamshidi · 2025
Read Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models by Akshar Tumu, Varad Shinde, Parisa Kordjamshidi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06146.txt.
Introduction
Akshar Tumu, Varad Shinde, and Parisa Kordjamshidi use referring expressions, phrases that single out one object in a scene, to probe spatial understanding in vision-language models. Spatial reasoning is core to human cognition, and current models show strain there. Most tests lean on captioning or question answering, which probe the skill only shallowly.
Their platform stresses three hard cases: ambiguous detection, long expressions stacking several spatial relations, and expressions using negation. Both specialized architectures and large models were tested. All struggled, but their strengths split by the kind of spatial meaning involved, such as topological versus directional relations. The authors map which spatial semantics break which models, and point to the research gaps that follow.
Catalog record
| CREATOR | Akshar Tumu, Varad Shinde, Parisa Kordjamshidi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06146
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models Akshar Tumu, Varad Shinde, Parisa Kordjamshidi Abstract Spatial Reasoning is an important component of human cognition and is an area in which the latest Vision-language models (VLMs) show signs of difficulty. The current analysis works use image captioning tasks and visual question answering. In this work, we propose using the Referring Expression Comprehension task instead as a platform for the evaluation of spatial reasoning by VLMs. This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation (‘not’). In our analysis, we use task-specific architectures as well as large VLMs and highlight their strengths and weaknesses in dealing with these specific situations. While all these models face challenges with the task at hand, the relative behaviors depend on the underlying models and the specific categories of spatial semantics (topological, directional, proximal, etc.). Our results highlight these challenges and behaviors and provide insight into research gaps and future directions. 1 Introduction Vision-language model (VLM) research has boomed in the recent past, owing to the enhanced user interaction and accessibility they provide. Models such as GPT 4o 1 1 1 https://openai.com/index/hello-gpt-4o/ , LLaVA ( 18 ), Google Gemini ( 34 ) have become adept at solving vision-language tasks such as Visual Question Answering (VQA), Image Captioning, and more. However, VLMs still lack human-level ‘Spatial Reasoning’ capabilities ( 17 ; 13 ). Spatial reasoning involves comprehending relations that depict the absolute/relative position or orientation of an object, such as ‘left’, ‘above’, or ‘near’. Inaccurate spatial reasoning by VLMs can lead to serious consequences in embodied AI domains such as autonomous driving and surgical robotics. A focused analysis of VLMs’ spatial reasoning capabilities can help identify potential reasoning issues. Most of the previous works confine their analysis to testing which models work well for spatial relations. We go further to analyze the comparative performance of these models for spatial categories that represent different orientational and positional relations between objects. A novel aspect of our work is the analysis of the effect of varying spatial composition (number of spatial relations) in the expressions on the performance of the models. Previous works focused on spatial analysis with image captioning-related tasks, thus failing to locate the source of error in the presence of visual and linguistic ambiguity. To avoid this, we adopt the Referring Expression Comprehension (REC) task ( 25 ) where the models output bounding boxes around the target entity based on a natural language expression, the analysis of which could reveal the parts of the input that the models fail to comprehend. Comprehension accuracy (or simply, accuracy) is a common metric for this task; it captures how often a model correctly outputs the bounding box around the target entity. We test four popular VLMs - LLaVA ( 18 ), Grounding DINO (GDINO) ( 19 ), DeepSeek-VL2 ( 37 ), and Qwen2.5-VL ( 3 ). We also include ‘MGA-Net’ ( 40 ), a model specifically designed for the REC task. These models offer diversity in the evaluation as they cover different architectural elements, training strategies, and input formats. We further compare these models with an object detector baseline to test if the images are truly complex and require elaborate referring expressions to ground the correct object. (a) The white napkin that is wrapped around the hot dog (b) The white box that is around the mirror (c) The brown table that is to the left of the black cell phone (d) The sandy shore that is near the murky water (e) The baseball player that is to the left of the black helmet and to the right of the home plate (f) The large branch that is to the right of the log that is behind the large bear (g) The black monitor that is to the left of the keyboard or on the desk (h) The blanket that is not green and that is not on the bed (i) The fence that is not black and that is not to the left of the man Figure 1: Figures for qualitative analysis. Bounding box legend - Red: MGA-Net, Blue: GDINO, Yellow: LLaVA, Orange: DeepSeek-VL2, Pink: Qwen2.5-VL, Green: Ground-truth Some of our important findings are as follows: (1) Referring expressions that include spatial relations, in addition to object attributes, result in higher accuracy on the REC task compared to expressions with only attributes. (2) Increasing the spatial complexity (no. of spatial relations) of an expression affects the performance of the VLMs, but models with explicit compositional learning components maintain the performance. (3) Expressions involving dynamic spatial relations yield low average accuracy across models, indicating the difficulty in modelling these relations. (4) The task-specific trained models achieve higher accuracy for expressions with geometric spatial relations (e.g., left of, right of) while the VLMs show relatively better accuracy for expressions having ambiguous relations such as proximity. (5) The models fail to recognize negated spatial relations in referring expressions in multiple instances, though the extent of this failure varies across models. Dataset Object Categories Image Source Avg. Length of Expression Avg. No. of Objects per Image No. of Spatial Relations RefCOCO 80 Real-world 3.6 10.6 59 RefCOCOg 80 Real-world 8.4 8.2 72 CLEVR-Ref+ 3 Synthetic 22.4 6.5 4 CopsRef 508 Real-world 14.4 17.4 51 Table 1: Statistics of Popular Referring Expression Comprehension Datasets. For the last column, the relation types are taken from various resources explained in Section 2 and 21 , in addition to the relations in Table 2 . Category Number Spatial Relations Absolute 56 on the right, on the left, in the middle, in the center, from the right, from the left Adjacency 14 attached, against, on the side, on the back, on the front, on the edge Directional 29 falling off, along, through, across, down, up, hanging from, coming from, around Orientation 0 facing Projective 2361 on top of, beneath, beside, behind, to the left, to the right, under, above, in front of, over, below, underneath Proximity 217 by, close to, near Topological 1054 connected, contain, with, surrounding, surrounded by, inside, between, touching, out of, at, in, on Unallocated 56 next to, enclosing Table 2: Category-wise relation split and number of referring expressions in the CopsRef ( 5 ) test set with 1 spatial relation in each category. While there are no referring expressions with only one relation from the orientation category, these relations co-occur with relations from other categories in some expressions. 2 Related Work Previous works have conducted a broad analysis on the ability of VLMs to perform multimodal perception and reasoning tasks, such as Spatial Reasoning, Multimodal conversation, etc. Many comprehensive real-world benchmarks have been introduced to test multiple VLM capabilities. ( 18 ; 35 ; 20 ; 7 ). Some works ( 33 ; 30 ) focus solely on spatial analysis of VLMs. 36 go a step further to analyze the role of each modality in spatial reasoning. However, these works do not analyze the factors that affect the spatial reasoning ability of the VLMs. Another class of works performs