Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Xianzheng Ma Tao Sun Shuai Chen Yash Bhalgat Jindong Gu, Angel X Chang Iro Armeni Iro Laina Songyou Peng Victor Adrian Prisacariu · 2026
Read Do 3D Large Language Models Really Understand 3D Spatial Relationships? by Xianzheng Ma Tao Sun Shuai Chen Yash Bhalgat Jindong Gu, Angel X Chang Iro Armeni Iro Laina Songyou Peng Victor Adrian Prisacariu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23523.txt.
Introduction
Ma and nine co-authors question whether 3D language models truly grasp spatial relations. A text-only model, tuned on plain question pairs, matches 3D models on the SQA3D test. That hints the test rewards text shortcuts, not 3D reasoning. The team builds Real-3DQA, a stricter bench that filters guessable questions. A taxonomy splits reasoning into clear aspects for scoring. On the new bench, current 3D models stumble once easy cues vanish. A reweighted training goal pushes models toward visual clues and lifts scores a lot.
Catalog record
| CREATOR | Xianzheng Ma Tao Sun Shuai Chen Yash Bhalgat Jindong Gu, Angel X Chang Iro Armeni Iro Laina Songyou Peng Victor Adrian Prisacariu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54661 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23523
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do 3D Large Language Models Really Understand 3D Spatial Relationships? Xianzheng Ma Tao Sun Shuai Chen Yash Bhalgat Jindong Gu, Angel X Chang Iro Armeni Iro Laina Songyou Peng Victor Adrian Prisacariu Abstract Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA , a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs’ performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Figure 1: Our Real-3DQA Benchmark and Solution. We demonstrate the answer differences between 3D-LLM (LEO) and its blind-finetuned version on various questions, with correct answers highlighted in green frames and incorrect ones in red frames. We discover that some questions can be answered correctly regardless of whether 3D information is used, which we consider to be 3D-independent questions. By filtering out these 3D-independent questions and introducing a new viewpoint rotation score, the original models’ performance drops significantly on Real-3DQA. Finally, using our proposed 3D-aware Reweighted Finetuning strategy, performance improves again. 1 Introduction Understanding and reasoning about 3D environments is fundamental for embodied AI, AR/VR applications, autonomous driving, etc. Recent work has proposed 3D Large Language Models (3D-LLMs) Hong et al. 2023 ; Chen et al. 2024 ; Guo et al. 2023 ; Huang et al. 2024a , aiming to empower language models with spatial awareness. Early evaluations focused on tasks such as 3D captioning Chen et al. 2020 and object grounding Yang et al. 2024 ; Achlioptas et al. 2020 , while 3D question answering (3D-QA) Azuma et al. 2022 ; Ye et al. 2021 ; Etesam et al. 2022 soon emerged as a more flexible benchmark for assessing diverse aspects of spatial understanding. More recently, 3D Situated QA Ma et al. 2023 ; Zhang et al. 2025c ; Linghu et al. 2025 extended this line of evaluation by adopting an egocentric, first-person perspective, simulating how humans perceive the world and challenging models to reason continuously about objects and relations in context. While these benchmarks are intended to measure progress toward robust embodied intelligence, we question whether the reported improvements truly reflect genuine 3D spatial reasoning. This paper investigates this gap between perceived progress and actual spatial understanding. Figure 2: Our finding : A language model fine-tuned only on text QA pairs without any 3D inputs ( Blind Finetuned ) can match or even surpass state-of-the-art 3D-LLMs ( Original ) on multiple 3D-QA benchmarks. This exposes a critical weakness in current benchmark design and calls into question their ability to assess genuine 3D reasoning despite linguistic shortcuts. We find that much of this apparent progress is illusory. On the most-widely used situated QA benchmarks, i.e. SQA3D Ma et al. 2023 , where accuracy has increased from 30% to over 50% in recent years, 3D-LLMs can achieve competitive scores without actually processing any 3D data. A simply fine-tuned blind model 1 1 1 We refer readers to section 4.1 for details on how the blind finetuned model is obtained. finetuned on text-only question-answer pairs can match or surpass the performance of recent 3D-LLMs, as illustrated in Fig. 2 . This reveals that many SQA3D questions can be solved solely by linguistic priors, rather than genuine spatial reasoning. Importantly, we observe similar vulnerabilities beyond SQA3D, including on ScanQA and the more recent MSR3D benchmark. Meanwhile, our findings echo those of Huang et al. 2025 ; Li et al. 2025 ; Zhang et al. 2024 ; Li et al. 2025 and indicate that even the latest 3D-LLMs are not immune to this gap. What causes existing benchmarks to fall short in accurately evaluating 3D reasoning? One hypothesis is the presence of bias Vo et al. 2025 in 3D-QA datasets. These biases can stem from both manual annotation Azuma et al. 2022 ; Ye et al. 2021 ; Zhang et al. 2025b ; Huang et al. 2025 and automatic or semi-automatic LLM-assisted generation processes Linghu et al. 2025 ; Zhang et al. 2025c , where linguistic or commonsense priors are embedded in the data. For example, “What is the black rectangular object on the wall?” almost always yields “TV” for indoor scenes regardless of spatial context. These questions allow models to succeed via shortcut learning instead of true 3D understanding. While SQA3D Ma et al. 2023 attempt to mitigate this issue by balancing answer distributions, such surface-level fixes cannot eliminate deeper biases, like preferences for salient objects, canonical configurations, or easily guessable answers. These dataset biases are multi-faceted and difficult to eliminate through either manual correction or distribution balancing. To address this, we introduce Real-3DQA, a benchmark that filters out questions solvable by language priors alone, as summarized in Table 1 . We compare each 3D-LLM’s accuracy to its “blind” counterpart—finetuned on text-only QA pairs without any 3D inputs. Questions answered correctly by both models are considered low in 3D dependency and are removed, as the answer can be inferred without spatial understanding. By filtering out such trivial questions, we implicitly mitigate a broad range of biases without requiring detailed manual intervention. This model-based filtering allows our Real-3DQA more robustly evaluate genuine 3D reasoning. Although removing low–3D–dependency questions prevents linguistic shortcuts, it does not guarantee that a model truly understands spatial structure. To further assess genuine 3D understanding, we propose a cross-question consistency test under viewpoint changes. The key idea is that if a model truly grasps the 3D scene, it should answer the same question correctly even when the observer’s orientation changes. Concretely, we rotate the viewpoint while keeping the position and question fixed, and adjust any directional terms in both the situation description and the expected answer. This results in logically equivalent QA pairs across different reference frames, forcing the model to interpret spatial relations consistently. To quantify this, we further introduce the Viewpoint-Rotation Score, which measures whether the model maintains spatial consistency across varied perspectives. While our proposed benchmark and consistency test offer a more rigorous evaluation, they do not by themselves prevent models from learning shortcuts during training. We find that standard supervised fine-tuning still encourages models to learn linguistic shortcuts, limiting the benefit of 3D information. To address this limitation from the training perspective, we introduce a 3D-aware Reweighted Fine-tuning (3DR-FT) strategy. Our key idea is to quantify the 3D dependency of each question by measuring the performance gap between a blind (text-only) model versus a full 3D-LLM. Based on this gap, we downweight questions that can be guessed easily from text alone, and upweight those requiring genuine spatial reasoning. This adaptive reweighting encourages models to incorporate 3D