SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models
Wufei Ma, Luoxin Ye, Nessa McWeeney, Celso M de Melo, Jieneng Chen, Alan Yuille · 2025
Read SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models by Wufei Ma, Luoxin Ye, Nessa McWeeney, Celso M de Melo, Jieneng Chen, Alan Yuille (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00788.txt.
Introduction
People read space at a glance — who is left of whom, what will hit what — while the great image-chat models still fumble it. Ma and co-authors teach that glance in 2025. Their model is fed two new diets: drills that pin each thing's place and facing in three axes, and talks that reason over rich spatial ties in real photos.
The facing drills are a first: no past set quizzed models on which way a thing turns. The team also varies the build and the training mix to learn what truly feeds the skill. Thin three-axis data, they hold, is the root lack. Such skill serves robots, drivers, and any agent that must act in a room, not just chat about one.
Catalog record
| CREATOR | Wufei Ma, Luoxin Ye, Nessa McWeeney, Celso M de Melo, Jieneng Chen, Alan Yuille |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00788
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models Wufei Ma, Luoxin Ye, Nessa McWeeney, Celso M de Melo, Jieneng Chen, Alan Yuille Abstract Humans naturally understand 3D spatial relationships, enabling complex reasoning like predicting collisions of vehicles from different directions. Current large multimodal models (LMMs), however, lack of this capability of 3D spatial reasoning. This limitation stems from the scarcity of 3D training data and the bias in current model designs toward 2D data. In this paper, we systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, a large multi-modal model with advanced 3D spatial reasoning abilities. To address data limitations, we develop two types of 3D-informed training datasets: (1) 3D-informed probing data focused on object’s 3D location and orientation, and (2) 3D-informed conversation data for complex spatial relationships. Notably, we are the first to curate VQA data that incorporate 3D orientation relationships on real images. Furthermore, we systematically integrate these two types of training data with the architectural and training designs of LMMs, providing a roadmap for optimal design aimed at achieving superior 3D reasoning capabilities. Our SpatialLLM advances machines toward highly capable 3D-informed reasoning, surpassing GPT-4o performance by 8.7%. Our systematic empirical design and the resulting findings offer valuable insights for future research in this direction. Our project page is available here . 1 Introduction When humans observe a scene, they perceive more than isolated objects; they intuitively understand the spatial relationships among these objects Spelke and Kinzler 2007 , enabling complex reasoning and interaction with the environment. For example, recognizing that a car is approaching an obstacle and might collide involves interpreting the 3D object orientations and spatial relationships. This innate ability to comprehend and reason about 3D space is crucial for navigating and interacting with the physical world. Figure 1 : 3D spatial reasoning is crucial for LMMs to ground objects in 3D space and infer their 3D spatial relationships, such as distance, orientation, and spatial interactions. We systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, an LMM with advanced 3D spatial reasoning abilities. Despite significant advancements, current LMMs Achiam et al. 2023 ; Team et al. 2023 ; Anthropic 2024 ; Liu et al. 2024a ; Liu et al. 2024c , exhibit limited capability of 3D spatial reasoning. These models excel at identifying objects and generating descriptive captions but struggle with tasks that require intricate 3D reasoning, such as predicting object interactions or understanding physical events. The primary challenge lies in the scarcity of high-quality 3D spatial relationship data, which prevents models from learning effective 3D spatial reasoning. Collecting a small set of high-quality 3D-aware data to tackle the first challenge is feasible, albeit labor-intensive, using readily available tools. Recent efforts Chen et al. 2024a ; AI 2024 leverage vision foundation models to recognize Zhang et al. 2024 , detect Liu et al. 2023b , segment Kirillov et al. 2023b and estimate the depth Bhat et al. 2023 ; Yang et al. 2024b in images. These approaches form semi-automated pipelines to construct pseudo-2.5D datasets for training LMMs with spatial reasoning capabilities, such as depth ordering Tong et al. 2024 ; Chen et al. 2024a . However, a significant gap remains: previous works Chen et al. 2024a ; AI 2024 ; Cheng et al. 2024 have primarily focused on 3D distance relationships, overlooking the crucial role of 3D object orientation. This is primarily due to the absence of robust 6D pose estimators. Understanding object orientation is essential for complex spatial reasoning tasks, such as determining if two cars are facing the same direction to avoid dangerous collision. Addressing this gap, we aim to incorporate 3D orientation relationships—converted from ImageNet3D Deng et al. 2009 —into our data engine, making us the first to enable complex spatial reasoning involving 3D orientation relationships. Even with 3D-informed data in hand, fully unlocking its potential remains challenging due to the intricate, multi-stage development process required for an LMM. For example, SpatialVLM Chen et al. 2024a utilizes 3D-informed data during the instruction tuning stage, but it remains uncertain whether such data can enhance other stages like vision encoder pretraining or multimodal alignment. This limitation suggests that a more holistic approach is necessary. In this paper, we present a systematic investigation into building LMMs with enhanced 3D spatial reasoning capabilities through a compound design approach, as illustrated in Fig. 1 . Our key contributions are threefold: First, we identify three fundamental types of 3D spatial relationships that LMMs must master: distance relationships, orientation relationships, and complex spatial reasoning that combines both. To evaluate these capabilities, we introduce SpatialVQA, a comprehensive benchmark containing 1,323 questions that target 3D spatial understanding, built upon diverse urban and indoor scenes from Omni3D Brazil et al. 2023 . Second, we propose a novel compound 3D-informed design that introduces improvements across multiple dimensions, leading to our proposed SpatialLLM model. On the data front, we develop two types of 3D-informed training datasets: (1) 3D-informed probing data focused on object-level 3D properties, and (2) 3D-informed conversation data for for complex spatial relationships. Architecturally, we explore mixing different visual encoders and investigate their 3D awareness capabilities. For training, we introduce 3D-informed data at multiple stages: pre-pretraining, multimodal alignment, and instruction tuning. Third, we present the first comprehensive search over the LMM design space for spatial reasoning tasks and propose a roadmap towards developing state-of-the-art models in this domain. The experiment demonstrates that our proposed SpatialLLM achieves state-of-the-art performance on spatial reasoning tasks, surpassing both GPT-4o Achiam et al. 2023 (by 8.7%) and spatial-centric models like SpatialVLM Chen et al. 2024a (by 10.5%). Our systematic analysis reveals that while architectural improvements contribute a 1.3% gain, the strategic incorporation of 3D-informed data across different training stages yields a substantial 13.7% performance improvement. Our findings highlight the importance of a holistic approach to improving LMMs’ 3D spatial reasoning capabilities, demonstrating that both architectural choices and carefully curated open-source 3D-informed training data play crucial roles in achieving superior performance. This work establishes a new benchmark for spatial reasoning in LMMs and provides valuable insights for future research in this direction. 2 Related Works Large multimodal models (LMMs) . Recently, there has been a significant surge in research on multimodal LLMs, exemplified by products like GPT-4 Achiam et al. 2023 and Gemini Team et al. 2023 , as well as open-source models such as LLaVA Liu et al. 2024c ; Liu et al. 2024a ; Li et al. 2024a and Qwen-VL Bai et al. 2023b . These open-source models combine pre-trained vision encoders Radford et al. 2021 ; Jia et al. 2021 ; Oquab et al. 2024 ; Chen et al. 2024b with LLMs Touvron et al. 2023 ; Bai et al. 2023a ; Jiang et al. 2023 , enabling the language models to handle a wide range of tasks that involve images as input. Typically, these methods has the alignment stage that aligns image features with the input embedding space of the language model using connectors Liu et al. 2024a ; Cha et al. 2024 ; Li et al. 2022 , trained with paired images and captions Sharma et al. 2018 . The instruction fine-tuning Liu et al. 2024c stage uses data that contains both