SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, Alan Yuille · 2025
Read SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning by Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, Alan Yuille (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20024.txt.
Introduction
Wufei Ma and six co-authors present a vision-language model built for reasoning about three-dimensional space, in a 2025 paper. Top models still struggle with such reasoning, even on questions humans find easy. Past fixes trained models on spatial question data, but the reasoning stayed hidden inside. Errors were hard to trace that way.
Their model, SpatialReasoner, uses clear 3D descriptions shared across its stages. Seeing, computing, and reasoning all work on the same descriptions. That shared form helps the model handle new question types. The team also read its step-by-step traces to find where models go wrong. Tests beat a leading rival by a wide margin on one benchmark. Gains held on novel spatial questions as well. The study links older 3D parsing tools with modern language reasoning.
Catalog record
| CREATOR | Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, Alan Yuille |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, Alan Yuille Abstract Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models. Recent studies explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual question-answering data. However, these methods typically perform spatial reasoning in an implicit manner and often fail on questions that are trivial to humans, even with long chain-of-thought reasoning. In this work, we introduce SpatialReasoner, a novel large vision-language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages–3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability to novel question types. Furthermore, by analyzing the explicit 3D representations in multi-step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMs. Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2.0 by 9.2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning. Project page (code, models and data): spatial-reasoner.github.io 1 Introduction 3D spatial reasoning studies how models perceive, understand, and reason about 3D object properties and spatial relationships. It is not only a fundamental task for vision-language models to achieve human-level intelligence, but also crucial to a range of downstream applications in robotics [ 14 , 18 ] and embodied AI [ 10 ] . Despite the recent advancements of large multi-modal models, such as GPT-4o and Qwen2.5-VL, their 3D spatial reasoning capabilities remain limited and fall far behind human-level performance [ 9 , 32 , 51 ] . Recent 3D-aware large vision-language models (LVLMs) injected 3D knowledge by fine-tuning the model on synthetic 3D-related question-answer pairs [ 9 , 34 ] and achieved improved performance on spatial reasoning benchmarks [ 32 , 45 , 51 ] . On the other hand, large proprietary models such as Gemini 2.0 [ 15 ] have advanced 3D parsing by directly predicting 3D object bounding boxes, enabling the development of powerful generalist robotics models [ 43 ] . We identify two key challenges in 3D spatial reasoning: (1) 3D thinking –the ability to decompose a complex 3D spatial reasoning question into small, manageable steps, and (2) 3D computation –the ability to solve these thinking steps in a consistent and accurate manner. As shown in Figure 1 , prior reasoning methods adopt long chain-of-thought reasoning to tackle the problem but do not have explicit 3D computation. ChatGPT o3 [ 36 ] fails to adopt a systematic approach to solving the problem and instead relies on other visual cues to assist the reasoning ( e.g. , location of the foil package). In contrast, the Gemini 2.0 thinking model [ 15 ] employs an organized strategy to tackle the problem but ultimately fails to arrive at the correct answer due to a lack of reliable 3D computation. In this work, we present SpatialReasoner, a novel large vision-language model (LVLM) built with (1) explicit 3D representations and (2) enhanced and generalizable 3D thinking. Specifically, our SpatialReasoner adopts explicit 3D representations, such as 3D locations and orientations, as an interface that enables coherent and reliable reasoning across multiple stages, i.e. , 3D perception, computation, and reasoning. On the other hand, we would like to learn enhanced 3D thinking capabilities that generalize to novel question types not seen during training. Hence we explore a two-stage training strategy following prior works [ 17 ] . In Stage I, we apply supervised fine-tuning (SFT) to equip the LVLM with explicit 3D representations, enhancing 3D perception and computation capabilities of the model. Then in Stage II, we leverage reinforcement learning (RL) to develop robust and generalizable 3D thinking built on explicit 3D representations. Inspired by previous 3D pseudo-annotation pipelines in [ 9 , 11 , 34 ] , we synthesize basic 3D perception, i.e. , detection and pose estimation, and 3D computation question-answering data interleaved with explicit 3D representations. We further generate standard spatial reasoning question-answer pairs with chain-of-thought reasoning that breaks down complex 3D spatial reasoning questions into multiple steps–3D perception, computation, and reasoning. Experimental results demonstrate that our SpatialReasoner with explicit 3D representations can significantly enhance 3D spatial reasoning abilities of LVLMs and generalize to novel question types. Figure 1 : Comparing 3D spatial reasoning of our SpatialReasoner with previous state-of-the-art models. Our SpatialReasoner builds on explicit 3D representations, performs 3D computation, and reasons about the final answer. Although Gemini 2.0 can also break down complex 3D spatial reasoning questions into small and tractable steps, it lacks reliable 3D computation that leads to the correct answer. Besides enhancing 3D spatial reasoning capabilities of LVLMs, reasoning with explicit 3D representations allows us to interpret the reasoning process and to study the failure modes of LVLMs. We find that the accuracy of 3D perception lags significantly behind that of 3D computation, suggesting that most errors in downstream VQA tasks still stem from failures in 3D perception. Moreover, by predicting key 3D information, such as 3D object locations and orientations, as intermediate results, our SpatialReasoner enables compositional reasoning for 3D spatial tasks [ 46 ] . This not only allows our method to generalize better to novel spatial reasoning questions, but also makes it easily extensible to other tasks that build on our explicit 3D representations. Besides improving spatial reasoning performance on a variety of benchmarks, we experiment on various LVLMs fine-tuned with different combinations of data and training methods to study the key factors toward improved 3D spatial reasoning. Our empirical results lead to the following insights: (1) When developing 3D-aware VLMs, SFT offers a more scalable approach than RL that requires high-quality 3D-aware data, which is often difficult to obtain. Our recipe of RL followed by SFT achieves the best overall performance; (2) 3D-aware LVLMs fine-tuned with RL generalize better than SFT when tested on novel 3D spatial reasoning questions, echoing prior findings [ 12 ] ; (3) Standard LVLMs often exploit 2D reasoning as a shortcut to tackle 3D spatial reasoning problems, whereas our SpatialReasoner avoids the spurious correlations and always reasons with explicit 3D representations, achieving improved and robust performance on challenging real-world 3D spatial reasoning datasets. In summary, our contributions are as follows: (1) We introduce SpatialReasoner, a novel LVLM that solves 3D spatial reasoning problems with a compositional approach based on explicit 3D representations. (2) Reasoning with 3D representations allows us to interpret the reasoning process and to study the failure modes of LVLMs. (3) Our SpatialReasoner effectively improves the 3D spatial reasoning performance on a range of benchmarks and extensive experimental results provide valuable findings on data and training strategy designs for future development of 3D-aware LVLMs. Figure 2 : Overview of our SpatialReasoner design. Our SpatialReasoner adopts explicit 3D representations as an interface to enable coherent and reliable multi-stage spatial reasoning, i.e. , 3D parsing, computation, and reasoning. 2 Related Works 3D