Agentic Collaborative Cognition for Zero-Shot 3D Understanding
Wenxin Wang 1, Bo Zhang 1⋆, Feng Chen 2†, Zixuan Wang 1, Wen Li 3, Changsheng Li 4, Yinjie Lei 1‡ · 2026
Read Agentic Collaborative Cognition for Zero-Shot 3D Understanding by Wenxin Wang 1, Bo Zhang 1⋆, Feng Chen 2†, Zixuan Wang 1, Wen Li 3, Changsheng Li 4, Yinjie Lei 1‡ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24649.txt.
Introduction
Wang and colleagues split three-dimensional scene understanding between two cooperating agents. One decides where to look. The other writes down what it sees. Zero shot 3D work has been recast as video keyframe reading for multimodal models. Fixed videos offer finite viewpoints, though, and leave three-dimensional structure implicit. Their 2026 framework attacks both limits with a closed loop between the two agents.
The Perception Agent summarises the scene into a structured cognitive map, documenting object attributes under consistent instance IDs across viewpoints. The Planning Agent reads that map, decides which query relevant viewpoints are missing, and adds critical views. Perception feedback filters mismatched candidates, and steers the next round of viewpoint planning. Iterating until candidates are resolved, the pair assembles split views into one coherent scene model. Wang's team presents collaborative thinking, rather than a bigger single model, as the route past the viewpoint bottleneck.
Catalog record
| CREATOR | Wenxin Wang 1, Bo Zhang 1⋆, Feng Chen 2†, Zixuan Wang 1, Wen Li 3, Changsheng Li 4, Yinjie Lei 1‡ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentic Collaborative Cognition for Zero-Shot 3D Understanding Wenxin Wang 1, Bo Zhang 1⋆, Feng Chen 2†, Zixuan Wang 1, Wen Li 3, Changsheng Li 4, Yinjie Lei 1‡ Abstract Recent advancements have explored agentic zero-shot 3D understanding by reformulating it as video keyframe understanding with Multimodal Large Language Models (MLLMs). However, existing methods face an intrinsic bottleneck due to the finite observation perspectives inherent in videos and the implicit perception of 3D scenes. In this paper, we propose a collaborative multi-agent framework that assigns a Planning Agent to handle high-level viewpoint planning and supplement novel perspectives, and a Perception Agent to explicitly summarize the 3D scene into a structured holistic cognitive map. Specifically, Planning Agent first analyzes this cognitive map to determine query-relevant viewpoints and supplements missing critical perspectives to ensure comprehensive observation. Subsequently, Perception Agent documents object-level attributes from these views by assigning consistent instance identifiers across viewpoints, thereby integrating fragmented observations into the holistic cognitive map. In parallel, it provides feedback to filter out mismatched candidate objects and guide subsequent viewpoint planning. Through this closed-loop iterative process, two agents collaboratively figure out candidates until Perception Agent determines that sufficient information has been captured to complete the task. Extensive experiments demonstrate that our method achieves state-of-the-art performance on 6 benchmarks, with improvements of 11.1% [email protected] on ScanRefer, 14.6 BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D. Project Page : https://zhangbo135.github.io/agentic-collaborative-cognition/ 1 Introduction 3D scene understanding [ 58 , 23 , 45 ] focuses on interpreting spatial layouts, semantic attributes, and inter-object relationships in complex environments, enabling critical applications from robotics [ 44 , 47 ] to augmented reality [ 29 , 4 ] . Previous studies mainly rely on fully supervised paradigms to achieve 3D visual grounding [ 10 , 61 ] , 3D dense captioning [ 11 , 18 ] , 3D question answering [ 5 , 30 ] , etc . Despite their effectiveness, the heavy reliance on large-scale 3D-text paired datasets limits scalability and adaptability to diverse real-world environments [ 52 , 56 ] . Zero-shot 3D understanding [ 57 , 53 , 40 ] has emerged to reduce the requirement for manually annotated datasets [ 13 , 36 ] . As shown in the top-left of Fig. 1 , previous methods [ 24 , 27 ] typically leverage the general world knowledge of Multimodal Large Language Models (MLLMs) [ 43 , 21 ] to interpret keyframes extracted from scene videos (an inherent component of scanned 3D datasets) through object-centric keyframe selection, which is enabled by key proposal selection and multi-view projection. However, relying solely on video keyframe selection inevitably has a performance bottleneck because: (1) Finite observation perspectives. During 3D scene scanning, videos are typically captured from restricted viewpoints close to the scene surfaces to ensure reconstruction quality. However, they provide only finite frames with fixed viewpoints, which often lack query-relevant critical perspectives; (2) Implicit spatial perception. Existing methods [ 24 , 40 ] typically leverage MLLMs to implicitly infer the spatial layout from keyframes. Although capable of capturing inter-object relationships under similar viewpoints [ 61 , 35 , 42 ] , they struggle to maintain unified spatial cognition when facing significant viewpoint variations [ 51 , 48 ] . Figure 1 : Comparison of existing methods with our method. Left: Existing zero-shot methods [ 24 , 27 ] typically rely on video keyframe selection for 3D visual grounding. Our method actively plans viewpoints to obtain comprehensive views and explicitly summarizes the 3D scene into a structured holistic cognitive map from these views for diverse 3D tasks. Right: Our method achieves substantial improvements over existing zero-shot and most fully supervised methods across 6 benchmarks. In this paper, we propose a collaborative multi-agent framework for zero-shot 3D scene understanding. As illustrated in the bottom-left of Fig. 1 , our method consists of two specialized agents: a Planning Agent that performs flexible planning of query-relevant viewpoints and supplements missing critical perspectives, and a Perception Agent that explicitly summarizes the 3D environment into a structured holistic cognitive map. Specifically, Planning Agent first analyzes this cognitive map to retrieve critical candidate objects and determine query-relevant viewpoints to further figure out these objects. For missing critical perspectives, it supplements rendered views to ensure comprehensive observation for Perception Agent. Subsequently, Perception Agent assigns unique instance identifiers to all objects in each view to ensure cross-view consistency, thereby accurately documenting object appearance and physical attributes, and integrating originally fragmented observations into the cognitive map. Additionally, it provides feedback that filters out mismatched objects to narrow the scope of candidates and guides the next round of viewpoint planning. This iterative collaboration allows the two agents to progressively refine 3D scene cognition until Perception Agent determines that sufficient information has been captured. Empirically, as illustrated in the right of Fig. 1 , our method surpasses previous state-of-the-art zero-shot and most fully supervised methods across 6 benchmarks, achieving substantial improvements of 11.1% [email protected] on ScanRefer [ 10 ] , 14.6 BLEU-1 on 3D-assisted dialog [ 17 ] , and 2.1 EM on SQA3D [ 30 ] . In brief, our main contributions can be summarized as follows: • We propose a collaborative multi-agent framework that reformulates zero-shot 3D understanding as an interactive planning-perception process. • We design a Planning Agent to plan query-relevant viewpoints for comprehensive observation, and a Perception Agent to update the holistic cognitive map, enabling the MLLM to directly observe and understand 3D scenes. • Extensive experiments on 6 benchmarks validate the state-of-the-art performance and generality of our method across diverse 3D understanding tasks. 2 Related work Zero-shot 3D Scene Understanding. Zero-shot 3D scene understanding has been explored as a promising alternative to fully supervised paradigms. Early works [ 50 , 16 , 56 ] typically reformat the 3D scene into textual descriptions for LLMs. For example, LLM-Grounder [ 50 ] uses the LLM to decompose queries into semantic constituents and employs a grounding tool to identify the target. CSVG [ 53 ] constructs structured scene graphs to model inter-object spatial relationships. However, these methods rely solely on text and struggle to capture rich visual details and fine-grained cues. Recent studies [ 49 , 27 , 40 ] leverage MLLMs to understand the 3D scene from 2D videos. SeqVLM [ 27 ] develops a proposal-guided projection strategy that performs object-centric keyframe selection to localize the target object. SPAZER [ 24 ] proposes a coarse-to-fine progressive reasoning mechanism to enable 3D-2D joint decision-making. Despite their effectiveness, they remain limited by the finite observation perspectives and fail to explicitly model the entire 3D scene. 3D Scene Documenting. Effective 3D scene documenting is a key approach to enhance understanding capabilities, evolving from geometric maps to rich semantic structures [ 37 , 33 ] . SceneGraphFusion [ 46 ] proposes a real-time method that incrementally constructs a globally consistent representation to document the 3D scene. More recently, Clio [ 31 ] leverages LLMs to construct language-based open-vocabulary 3D scene graphs. MTU3D [ 65 ] uses FastSAM [ 60 ] and DINO [ 34 ] to implicitly construct a dynamic spatial memory bank. Embodied VideoAgent [ 14 ] analyzes egocentric video and embodied sensory inputs to build a persistent scene memory. However, these representations often focus