Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods

Weichen Liu [email protected], Qiyao Xue [email protected], Haoming Wang [email protected], Xiangyu Yin [email protected], Boyuan Yang [email protected], Wei Gao [email protected] · 2025

Read Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods by Weichen Liu [email protected], Qiyao Xue [email protected], Haoming Wang [email protected], Xiangyu Yin [email protected], Boyuan Yang [email protected], Wei Gao [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15722.txt.

Introduction

Weichen Liu, Qiyao Xue, and four co-authors survey spatial reasoning in multimodal large language models, the ability to perceive and manipulate relations in three-dimensional space. That ability is basic to human intelligence but remains a persistent weakness of current models. Earlier surveys sorted work by input type, such as text, image, or video, which the authors argue misses what drives spatial skill.

Their taxonomy instead organizes tasks by reasoning complexity and links them to cognitive functions. Benchmarks across text-only, vision-language, and embodied settings are mapped onto that scheme, along with scoring methods. Training-based and reasoning-based improvement methods are compared, revealing gaps against human performance and complementary strengths that suggest concrete directions for future research.

Catalog record

CREATORWeichen Liu [email protected], Qiyao Xue [email protected], Haoming Wang [email protected], Xiangyu Yin [email protected], Boyuan Yang [email protected], Wei Gao [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods Weichen Liu [email protected], Qiyao Xue [email protected], Haoming Wang [email protected], Xiangyu Yin [email protected], Boyuan Yang [email protected], Wei Gao [email protected] Abstract Spatial reasoning, which requires ability to perceive and manipulate spatial relationships in the 3D world, is a fundamental aspect of human intelligence, yet remains a persistent challenge for Multimodal large language models (MLLMs). While existing surveys often categorize recent progress based on input modality (e.g., text, image, video, or 3D), we argue that spatial ability is not solely determined by the input format. Instead, our survey introduces a taxonomy that organizes spatial intelligence from cognitive aspect and divides tasks in terms of reasoning complexity, linking them to several cognitive functions. We map existing benchmarks across text-only, vision–language, and embodied settings onto this taxonomy, and review evaluation metrics and methodologies for assessing spatial reasoning ability. This cognitive perspective enables more principled cross-task comparisons and reveals critical gaps between current model capabilities and human-like reasoning. In addition, we analyze methods for improving spatial ability, spanning both training-based and reasoning-based approaches. This dual-perspective analysis clarifies their respective strengths, uncovers complementary mechanisms. By surveying tasks, benchmarks, and recent advances, we aim to provide new researchers with a comprehensive understanding of the field and actionable directions for future research. 1 Introduction The development of Large language model (LLM) represents a significant milestone in artificial intelligence, showcasing unprecedented capabilities in comprehending, reasoning over, and generating human-like natural language. These models, built upon deep learning architectures like the Transformer ( Minaee et al. 2024 ; Vaswani et al. 2017 ) , are pre-trained on vast amounts of linguistic corpora, enabling them to perform a wide array of language-centric tasks, from translation and summarization to complex reasoning ( Han et al. 2024 ; Kalyan 2023 ) . The introduction of models such as GPT-3 has showcased their potential in few-shot learning ( Brown et al. 2020 ) , where they can adapt to new tasks with minimal examples. These advances have established language intelligence as a cornerstone of contemporary artificial intelligence research, demonstrating scalable generalization across a wide range of linguistic tasks. Building on LLM’s linguistic foundation, researchers seek to move towards visual perception and understanding. Vision-Language Models (VLMs) represent an ongoing attempt to bridge perception and language by coupling visual encoders ( Dosovitskiy et al. 2021 ; Liu et al. 2021a ; Radford et al. 2021 ; Liu et al. 2022 ) with pretrained LLM backbones. However, a critical aspect of human-like intelligence that remains a significant challenge for these models is spatial intelligence , which is the ability to perceive, understand, and reason about the spatial relationships between objects, their orientation, and their movement in both real and imaginary spaces. While humans navigate and interact with the three-dimensional world easily, equipping LLMs and VLMs with a comparable level of spatial understanding is an ongoing frontier in AI research. The gap between language intelligence and spatial intelligence manifested explicitly across distinct task categories and implicitly through different human mental representations as shown in Figure 1 . LLMs demonstrate strong proficiency in language intelligence, operating over linguistic data such as text and code. In contrast, spatial intelligence involves reasoning within a grounded, 3D environment, enabling capabilities such as robot navigation and object manipulation. The underlying cause of this disparity lies in a fundamental representational mismatch: whereas the physical world is characterized by continuous geometric structures, LLMs encode and interpret information as discrete, sequential tokens. They learn spatial concepts not as geometric principles, but as statistical co-occurrences bias in vast datasets of text and images. For example, they learn that the words “left of” statistically appear between “cube” and “circle” without a true geometric understanding of the relationship. (a) Different tasks in language and spatial intelligence (b) The Representational Mismatch Figure 1 : The gap between language intelligence and spatial intelligence for MLLMs: (a) Language tasks rely on discrete and sequential token processing, while spatial tasks require grounded reasoning in continuous 3D space. (b) This mismatch reflects the representation-level grounding problem—MLLMs model statistical co-occurrence rather than true geometric relations. This fundamental distinction between linguistic and spatial intelligence is deeply rooted in both cognitive science and neural mechanisms. From a cognitive science perspective, humans reason about space not through linguistic expression alone, but through mental models, which is an internal analog representations preserving geometric and topological relations among objects ( Johnson-Laird 1986 ; Tversky 1993 ) . These mental models allow people to mentally simulate transformations such as rotation, translation, and perspective change ( Byrne & Johnson-Laird 1989 ) . In contrast, language compresses these continuous relationships into discrete categorical tokens (e.g., “left”, “behind”, “on top of”), which describe qualitative spatial relations rather than quantitatively encoding geometry ( Cohn & Renz 2008 ) . The evidence of neural mechanisms further underscores this difference. The hippocampal–entorhinal circuit is known to support spatial cognition through “place cells” and “grid cells”, which encode allocentric maps and metric structure of the environment ( O’Keefe & Dostrovsky 1971 ; Hafting et al. 2005 ; Stensola & Moser 2016 ) . These continuous neural codes form an internal coordinate system enabling path integration, location tracking, and mental navigation through space ( Basu & Nagel 2024 ) . In contrast, the representational substrate of language in the cortex is largely sequential and discrete, optimized for linguistic composition and token prediction, not for metric spatial computation. This mismatch between analog spatial coding and discrete linguistic encoding represents the classic representation-level grounding problem ( Harnad 1990 ) . LLMs are lack of an internal map of space that would allow them to mentally model or adjust these spatial relationships as humans do. Even for VLMs, though grounded in visual perception, typically remain constrained to 2D or projective representations and lack deeper 3D spatial mental modeling. Bridging this gap remains a formidable challenge, our proposed cognitive function taxonomy is specifically designed to reveal and systematically organize these deficiencies within current model capabilities. Figure 2 : Spatial tasks for different application domains Motivation and application: The advancement of spatial intelligence in LLMs and VLMs is crucial for enabling modern MLLMs to evolve towards comprehensive world models capable of understanding and interacting with the real world. While models have shown remarkable progress in language-centric tasks ( Brown et al. 2020 ; Wei et al. 2023 ) , their understanding of spatial relationships remains a significant hurdle ( Zhang et al. 2025b ; Han et al. 2025 ) . This limitation becomes critical when considering the vast range of embodied and spatial perception-grounded applications, driven tasks that inherently require coherent spatial reasoning and continuous interaction with the physical world. As illustrated in 2 , robust spatial intelligence is significant for the successful deployment of MLLMs in a multitude of high-impact domains: • Robotics: For a robot to be truly helpful in unstructured environments, it must be able to reason about the geometric relations between objects and locations ( Landsiedel et al. 2017 ) . This includes tasks like grasping objects, avoiding obstacles, and navigating through cluttered spaces, all of which demand a sophisticated understanding of spatial dynamics. • Autonomous Driving: The safety and reliability of self-driving cars are fundamentally dependent

Read the complete text