CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models

Yiqi Zhu, Ziyue Wang, Can Zhang, Peng Li, Yang Liu · 2025

Read CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models by Yiqi Zhu, Ziyue Wang, Can Zhang, Peng Li, Yang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14161.txt.

Introduction

Yiqi Zhu and co-authors build a test for a skill AI vision models quietly lack: understanding a space from a stream of views taken in one spot. When a person turns in place, each new view connects to the last, and the mind rebuilds the whole room. Models that ace single-image questions may still fail at stitching such views together.

Their benchmark, CoSpace, holds 2,918 images and 1,626 questions across seven task types, all built around this continuous view setting. The team ran nineteen leading models, both commercial and open. Most stumbled, including the strongest commercial systems. Open models matched commercial ones in raw accuracy more often than expected, but their answers were far less consistent. The authors argue that this continuous space sense is basic equipment for real-world tasks like robotics and navigation. They offer the benchmark to push model makers to train and test for it.

Catalog record

CREATORYiqi Zhu, Ziyue Wang, Can Zhang, Peng Li, Yang Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models Yiqi Zhu, Ziyue Wang, Can Zhang, Peng Li, Yang Liu Abstract Vision-Language Models (VLMs) have recently witnessed significant progress in visual comprehension. As the permitting length of image context grows, VLMs can now comprehend a broader range of views and spaces. Current benchmarks provide insightful analysis of VLMs in tasks involving complex visual instructions following, multi-image understanding and spatial reasoning. However, they usually focus on spatially irrelevant images or discrete images captured from varied viewpoints. The compositional characteristic of images captured from a static viewpoint remains underestimated. We term this characteristic as Continuous Space Perception . When observing a scene from a static viewpoint while shifting orientations, it produces a series of spatially continuous images, enabling the reconstruction of the entire space. In this paper, we present CoSpace, a multi-image visual understanding benchmark designed to assess the Co ntinuous Space perception ability for VLMs. CoSpace contains 2,918 images and 1,626 question-answer pairs, covering seven types of tasks. We conduct evaluation across 19 proprietary and open-source VLMs. Results reveal that there exist pitfalls on the continuous space perception ability for most of the evaluated models, including proprietary ones. Interestingly, we find that the main discrepancy between open-source and proprietary models lies not in accuracy but in the consistency of responses. We believe that enhancing the ability of continuous space perception is essential for VLMs to perform effectively in real-world tasks and encourage further research to advance this capability. Figure 1 : Illustration of continuous space perception: The left side depicts the construction of an image sequence representing a continuous space. The right side presents an example task related to the continuous space, showing how adjacent images are connected. Note that to answer the question correctly, one must recognize that the cars in images 1 and 2 are the same, as well as those in images 3 and 4. † † *Equal Contribution, †Corresponding Author † † * Project Page: https://thunlp-mt.github.io/CoSpace/ † † * Github Page: https://github.com/THUNLP-MT/CoSpace/ 1 Introduction With the advancements in deep learning and Large Language Models (LLMs), Vision-Language Models (VLMs) have demonstrated exceptional performance across a range of downstream tasks Li et al. 2023 ; Zhang et al. 2024b ; Lin et al. 2023 ; Jia et al. 2024b ; Shiri et al. 2024 ; Chen et al. 2024a . Driven by this, VLMs are now being used in diverse applications, such as planning Chen et al. 2023 ; Yang et al. 2024 , navigation ( Chiang et al. 2024 ; Zhou et al. 2025 ) and embodied tasks ( Li et al. 2024f ; Rajvanshi et al. 2024 ) , and are further applied in real-world scenarios ( Zhang et al. 2024c ; Cao et al. 2024 ) . Real-world scenario, featuring the complicated environment, cast significant challenges on VLMs, especially the ability to capture sophisticated spatial information from surrounding spaces. Benchmarks Evaluation Target Features Num. Img Evaluation Instances Spatial Reasoning Multi-image Perception Static Viewpoint MMRel ( Nie et al. 2024 ) Object-relationship ✗ ✗ 15k 15k Spatial-MM ( Shiri et al. 2024 ) Object-relationship and spatial-CoT ✓ ✗ ✗ 2.3k 400 SpatialEval ( Wang et al. 2024a ) Spatial Reasoning ✓ ✗ ✗ 4.6k 4.6k GSR-Bench ( Rajabi and Kosecka 2024 ) Grounded spatial reasoning ✓ ✗ ✗ 4.9k 4.9k MileBench ( Dingjie et al. 2024 ) Multimodal Long Context ✗ ✓ ✗ 97.9k 6.4k Demon-Core ( Li et al. 2024d ) Demonstrative instruction following ✗ ✓ ✗ 62.8k 18.2k Mementos-Val ( Wang et al. 2024d ) Temporal sequential-image understanding ✗ ✓ ✗ - 699 MIBench ( Liu et al. 2024b ) Fine-grained multi-image understanding ✗ ✓ ✗ 125k 13k BLINK ( Fu et al. 2025 ) General visual perception ✗ 7.3k 3.8k SEED-Bench ( Li et al. 2024a ) Visual comprehension ✗ 1.9k 24k Video-MME ( Fu et al. 2024 ) Visual comprehension ✓ ✗ 0.9k 2.7k MMIU ( Meng et al. 2024 ) Visual comprehension ✓ ✗ 77.7k 11.7k SparklesEval ( Huang et al. 2024 ) Multi-image Chat ✓ ✗ 550 150 CoSpace (Ours) continuous space perception ✓ ✓ ✓ 2,918 1,626 Table 1 : Comparison with selected benchmarks that address multi-image evaluation and/or spatial evaluation. ✓ denotes the main focus of the corresponding feature, indicates that the feature exists but is not the primary focus, while ✗ implies a lack of the feature. Despite previous attempts to advance spatial understanding in VLMs ( Chen et al. 2024a ; Nie et al. 2024 ) , there remain areas requiring further exploration. Various strategies have been explored to empower models with the ability to identify and ground objects based on spatial relationship Chen et al. 2024a ; Jia et al. 2024a . Shiri et al. 2024 and Wang et al. 2024a assess spatial understanding ability for VLMs with the recognition of basic tasks in this domain. However, much of these work center on single-image inputs, without further extending to multi-image scenarios. Works on video understanding ( Zhang et al. 2024b ; Wang et al. 2024c ) present the concept of spatial temporal understanding, but the spatial information of videos involve sequential frames that inherently linking spatial understanding with additional temporal dynamics. Given these attempts in spatial understanding, we pose the question: Are the current efforts adequate for advancing the research of spatial understanding? We claim that a gap still exists between the concept proposed in previous work and more advanced spatial understanding. For humans, exposing to an unfamiliar environment requires determining the position, identifying surrounding objects, and making decisions based on spatial context. Such process typically involves integrating various information about the surrounding space. We refer to it as continuous space perception , which is an essential ability in real-world towards effectively completion of daily tasks. Figure 1 shows an example of continuous space perception: Conceptually, humans perceive continuous visual space as a series of “images”, each representing a segment of the space with overlaps between adjacent views. Mimicking this perception, Figure 1 illustrates the continuous visual space using four images. To answer questions accurately about these images requires a comprehensive understanding of all four images and reconstructing the entire space they represent. For example, to accurately count the number of cars in the space, it is essential to recognize that the cars appearing in the first and second images are the same, as are the cars in the third and fourth images. The continuous space perception ability is equally important for VLMs as for human. Many real-world scenario tasks, such as planning and navigation, require models to effectively integrate the information in the surroundings to take the optimal actions. During this process, the reconstruction of the space heavily relies on continuous space perception. However, as shown in Table 1 , this ability is largely overlooked in existing benchmarks for VLMs. To this end, we present CoSpace , a visual understanding benchmark designed to evaluate the Co ntinuous Space perception ability for VLMs. CoSpace includes four categories and seven tasks, emphasizing the following core abilities: (1) Can VLMs recognize direction and position when presented with spatially continuous images captured from a static viewpoint? (2) Can models identify the correspondence between adjacent images in the same scene? (3)

Read the complete text