Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

Zhangquan Chen Manyuan Zhang Xinlei Yu Xufang Luo Mingze Sun Zihao Pan, Yan Feng Peng Pei Xunliang Cai Ruqi Huang1. Tsinghua Shenzhen International Graduate School, Tsinghua University2. Meituan 3. National University of Singapore · 2025

Read Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views by Zhangquan Chen Manyuan Zhang Xinlei Yu Xufang Luo Mingze Sun Zihao Pan, Yan Feng Peng Pei Xunliang Cai Ruqi Huang1. Tsinghua Shenzhen International Graduate School, Tsinghua University2. Meituan 3. National University of Singapore (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18632.txt.

Introduction

Zhangquan Chen and his ten co-authors give vision-language models a way to imagine three-dimensional space while they reason. Models that see only a few flat views struggle with spatial questions, and earlier fixes relied on text maps or two-dimensional clues with limited power. Their framework, 3DThinker, builds an inner three-dimensional picture during reasoning, without receiving any three-dimensional input and without training on labeled three-dimensional data.

Training runs in two stages. First, supervised learning aligns the model's inner three-dimensional representation with the output of a dedicated three-dimensional foundation model. Then the whole reasoning path is refined using only outcome signals, rewarding final answers rather than steps. Across several benchmarks, 3DThinker consistently beat strong baselines, and the authors present it as a route toward unifying three-dimensional representations with general multimodal reasoning.

Catalog record

CREATORZhangquan Chen Manyuan Zhang Xinlei Yu Xufang Luo Mingze Sun Zihao Pan, Yan Feng Peng Pei Xunliang Cai Ruqi Huang1. Tsinghua Shenzhen International Graduate School, Tsinghua University2. Meituan 3. National University of Singapore
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views Zhangquan Chen Manyuan Zhang Xinlei Yu Xufang Luo Mingze Sun Zihao Pan, Yan Feng Peng Pei Xunliang Cai Ruqi Huang1. Tsinghua Shenzhen International Graduate School, Tsinghua University2. Meituan 3. National University of Singapore Abstract Though recent advances in vision–language models (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker , a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code will be available at https://github.com/zhangquanchen/3DThinker . 1 Introduction Spatial understanding is a critical capability for machines to interact with the real 3D world (e.g., embodied AI, autonomous driving) Wang et al. 2024a ; Xia et al. 2018 ; Zeng et al. 2025a ; Yan et al. 2025a . These systems typically rely on ego-centric, multi-view observations, typically provided by multiple cameras simultaneously capturing limited views of their surroundings. These views are not interchangeable or purely visual; they inherently carry spatial semantics tied to the machine’s frame of reference Gholami et al. 2025 . Consequently, imagining the full scene and performing reasoning based on a few limited views presents an essential problem for spatial intelligence Yang et al. 2025c . Although recent VLMs are pretrained on large-scale image–text corpora, their performance on such spatial reasoning tasks remains notably limited Chen et al. 2024b ; Cheng et al. 2024b ; Liao et al. 2024 ; Yan et al. 2025b . The core bottleneck lies in their inability to extract 3D geometry embedded within images and their restricted capacity for spatial imagination . Figure 1 : Illustration of our 3DThinker . Existing methods typically perform reasoning based solely on pure text or 2D visual cues, without fully exploiting the rich spatial and geometric information inherent in images. Other methods attempt to enhance the input by introducing auxiliary modalities (e.g., depth maps or coordinates), yet these often depend on additional annotations or external tools. In contrast, our framework enables VLMs to intrinsically form 3D mental representations during reasoning, thereby improving their spatial understanding. Recent advances have attempted to enhance the spatial reasoning capabilities of VLMs Ouyang et al. 2025 ; Liu et al. 2025a ; Wu et al. 2025d ; Li et al. 2025d ; Gholami et al. 2025 ; Liu et al. 2025b ; Chen et al. 2025d . As illustrated in Fig. 1 , existing methods can be broadly divided into two categories. The first category performs reasoning with pure text Chen et al. 2024a ; Liu et al. 2025b ; Cheng et al. 2024a ; Ouyang et al. 2025 ; Liu et al. 2025a or 2D visual cues Daxberger et al. 2025 ; Chen et al. 2025d ; Xiao et al. 2024 , whose representational capacity for complex spatial layouts is inherently limited. To mitigate this limitation, methods such as MindCube Yin et al. 2025 train models to generate cognitive maps of 3D layouts; however, they rely on bird’s-eye-vie (BEV) annotations to construct these maps. Ego3D Gholami et al. 2025 further employs external models—GroundingDINO Liu et al. 2024 for referring expression comprehension (REC) and DepthAnythingv2 Yang et al. 2024a for depth estimation, to automatically generate cognitive maps Yet, constrained by the performance of these models, such methods often fail on low-resolution or uncurated images. The second category incorporates auxiliary modalities as additional inputs (e.g., point clouds, camera parameters) Chen et al. 2025a ; Li et al. 2025b . However, these settings restrict the model’s applicability in real-world scenarios where only monocular images are available. Moreover, several recent methods invoke external encoder or tool-usage to obtain prior information (e.g., encoded 3D tokens Fan et al. 2025b , depth maps Liu et al. 2025b ; Chen et al. 2025d ; Cai et al. 2024 ). Importantly, these techniques do not constitute an intrinsic capability of the model and introduce additional inference overhead. These challenges motivate the need for a new method that: G1) 3D-imaginable : can directly learn 3D geometry from limited 2D images; G2) Annotation-free : does not rely on densely annotated data; and G3) Intrinsic : requires no external priors or auxiliary models during inference. The most relevant mental model, Mirage Yang et al. 2025c , leverages ground-truth image embeddings for supervised training, facilitating the continuation of a multimodal trajectory without the need for pixel-level image generation. However, the training of Yang et al. 2025c is heavily reliant on ground-truth image supervision and remains constrained to the ”thinking with image” paradigm, which prevents its effectiveness on (G1) and (G2). Nevertheless, it provides a valuable inspiration, prompting us to introduce a new novel framework, 3DThinker, which enables thinking with 3D mentaling . Unlike prior works that depend on external priors or complex training data construction, our method intrinsically integrates 3D representations into the VLMs, enabling unified reasoning and 3D latent generation within the model. For (G1), our framework enables the model to generate geometric representations from images during the reasoning process. Regarding (G2), we directly project the 3D latent to align with a 3D foundation model, thereby circumventing the need for raw 3D data construction. Consequently, our model can inherently ”think with 3D” without relying on any prior or auxiliary geometry encoder, corresponding to (G3). Simultaneously, since our method allows for the recovery of 3D representations(e.g., point clouds) from 3D latents via the projector, it significantly enhances the interpretability of the large reasoning model . Specifically, we first construct a batch of Chain-of-Thought (CoT) data that incorporates 3D special tokens. Our training framework then proceeds in two main stages. In the first stage, we perform supervised learning, where features from the 3D foundation model (e.g., VGGT Wang et al. 2025b ) are distilled into the native reasoning process of the VLM. To enable the model to think with a 3D mentaling while maintaining textual coherence, we employ both 3D latent alignment loss and the cross-entropy loss. In the second stage, we employ reinforcement learning, optimizing the tokens across the entire sampling trajectory based solely on outcome-driven signals, while preserving the alignment of the 3D latent. That is, we refine 3D mentaling within the trajectory using only outcome as the optimization signal. Our contributions can be summarized as follows. • We are the first to introduce the ”think with 3D mentaling” framework, which operates without dependence on densely

Read the complete text