Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
Dongsheng Wang, Dawei Su, Hui Huang* · 2026
Read Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation by Dongsheng Wang, Dawei Su, Hui Huang* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03100.txt.
Introduction
Wang, Su, and Huang answer questions about 3D scenes with no task training. Their route uses 2D vision-language models. Many views are drawn from a 3D point cloud and fed to a trained model, which then answers. The catch is the input budget: only so much can be fed in, so the right views and parts must be picked.
Their method, KeyVT, picks context at two levels. It first rates views by both their content and their place in space, using pixel features and camera data. It then cuts repeat patches across views. Key tokens are chosen under an optimal transport frame, so a small token set still spans all view features. Tests on three standard benchmarks show clear gains over other tuning-free methods. Scores came close to those of methods that do train on the task.
Catalog record
| CREATOR | Dongsheng Wang, Dawei Su, Hui Huang* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation Dongsheng Wang, Dawei Su, Hui Huang* Abstract Recently, zero-shot 3D scene understanding via 2D Vision-Language Models (VLMs) has gained increasing research interest due to their promising spatial reasoning capabilities. Typically, multiple 2D views are sampled from a 3D point cloud and fed into pre-trained VLMs to answer a given question. This paradigm highlights the critical role of input context quality and raises the challenge of retaining as many task-relevant 3D details as possible under a limited input budget. We propose KeyVT , a hierarchical approach for input context collection at both the view and token levels. Specifically, we combine pixel features with camera parameters and assess view importance based on both semantic content and geometric position, resulting in spatially consistent and task-relevant views. Furthermore, we address redundancy among patches across selected views by identifying representative tokens under the optimal transport (OT) framework, where view tokens and key tokens are formulated as two discrete distributions in the embedding space. These key tokens are expected to cover all view features by minimizing the OT distance. We evaluate our framework on three widely used benchmarks, demonstrating significant improvements over existing tuning-free methods and performance comparable to training-based approaches. † † affiliationnotice: Equal contribution 1 Introduction After processing large-scale vision and text pairs, the Vision-Language Models (VLMs, such as GPT-4o ( 22 ) and Qwen3-VL ( 4 ) ) have demonstrated promising capabilities in various multimodal tasks, including image/video question answering, captioning, and reasoning ( 64 ; 70 ; 4 ; 55 ; 49 ; 41 ) . Inspired by these achievements and the success of pure vision solutions in Tesla’s full self-driving system ( 46 ) , recent studies have begun extending 2D VLMs to the 3D physical world by sampling multiple views from 3D point clouds. Once the view sequences are obtained, 2D VLMs can naturally reason with spatial environments, showing significant potential compared to 3D language models ( 15 ; 53 ; 56 ) . The latter typically rely on huge high-quality 3D-text data (such as point clouds paired with detailed descriptions) to align geometric structures with language ( 10 ; 58 ; 16 ) . However, such annotated 3D data remain scarce, limiting their scalability and practical applicability. The quality of the input context, therefore, plays a central role in 3D scene understanding. Taking 3D question answering (3D-QA) as a representative example, ideal inputs should incorporate as many task-relevant 3D details as possible, such as the main objects and their surrounding environments. Early studies directly employed uniform sampling strategies to select input views. While simple, these approaches often introduce substantial noise ( 70 ) . To enhance input quality, various key-view selection methods have been proposed, including semantic similarity-based retrieval ( 33 ; 68 ) , maximizing useful information ( 45 ; 71 ) , temporal clip selection ( 42 ) , and training additional key-view selectors ( 53 ) . Despite their success, most existing methods focus primarily on query–view relevance, which may overlook critical evidence not explicitly mentioned in the query, degrading downstream performance. More importantly, previous works typically feed the selected key views into VLMs with the number of views fixed at a small fraction of the total (such as 8 or 16 views) due to input budget constraints. Intuitively, these key views often share semantic information and exhibit significant redundancy across regions. Removing such redundancies generally does not harm prediction performance and, in fact, frees up input capacity, allowing additional key views to be incorporated without increasing computational overhead. This principle aligns with token compression techniques in video understanding tasks ( 39 ; 54 ; 38 ; 23 ) , which typically filter out temporally redundant tokens or cluster features to extract representative information. While effective, these clustering-based models are sensitive to densely populated features and may fail to capture diverse tokens across key views ( 12 ) . To overcome these limitations and collect as much task-relevant information as possible within a predefined input budget, we propose a hierarchical framework, KeyVT , which first selects spatially consistent, task-relevant Key V iews and then compresses representative T okens across these views in an unsupervised manner. Moving beyond similarity-based view sampling methods, we introduce geometric parameters for each view to model spatial relationships. Our idea is simple: the main objects and their surrounding environment provide primary evidence about the corresponding regions (As shown in Fig. 1 ). Given the camera parameters of each view, we estimate the relative distance from the first view to the current view. This captures spatial relationships within the entire 3D world and enables segmentation into spatially consistent sub-scenes. Given the fixed number of key views, we design an adaptive strategy to identify the optimal number of selected views for each sub-scene according to its importance, e.g. , more relevant sub-scenes are allocated more key views. Figure 1 : Motivation of our proposed KeyVT . KeyVT proposes to find optimal input context via a hierarchical key-view-then-key-token pipeline that preserves most information while significantly reducing the number of input tokens. Moving beyond the view-level selection, we further explore the token redundancy across the key views. We introduce the concept of virtual tokens, whose number is smaller than that of the key-view tokens while still being able to cover 3D details in the feature space. Unlike previous clustering-based models, we propose to learn virtual tokens under the optimal transport (OT) theory ( 48 ) . OT provides a principled measure of the cost required to transport mass from one distribution to another given a predefined cost function, and recent advances have made OT computationally efficient and widely applicable ( 13 ; 30 ; 51 ; 50 ) . Specifically, all key-view tokens are represented as a redundant set that encodes the task-relevant evidence, and the virtual token set aims to summarize key information by transporting features from the redundant set. Leveraging the ability of OT to model geometric structures in the space of probability distributions, our model is able to discover diverse and representative virtual tokens. Importantly, the learned transport plan of OT measures the probability value from virtual tokens to view tokens, which provides a simple and effective way to identify key tokens by ranking the transport plan. In summary, our contributions are three-fold: • We propose KeyVT , a hierarchical framework for selecting key views and key tokens from 3D scenes, enabling 2D VLMs to access rich 3D details for effective 3D scene understanding. • We explore the spatially consistent, task-relevant key-view selection and OT-based key token compression to jointly extract diverse and representative context under a limited input budget. • We demonstrate that KeyVT achieves significant improvements on three widely used 3D-QA benchmarks, with performance comparable to fine-tuned models. 2 Related Works 3D Large Language Model. Inspired by the huge success from LLMs to VLMs ( 22 ; 4 ) , 3D-LLMs aim to use 3D point clouds as inputs and align such spatial features in the language space. Pioneering works such as PointLLM and LL3DA ( 58 ; 9 ) employ the 3D-to-text loss to learn the projection layer from points to LLMs, showing interesting results in understanding object-level points. To mitigate the