CoNav CoNav : Collaborative Cross-Modal Reasoning for Embodied Navigation
Haihong Hao, Mingfei Han, Changlin Li, Zhihui Li, Xiaojun Chang · 2025
Read CoNav CoNav : Collaborative Cross-Modal Reasoning for Embodied Navigation by Haihong Hao, Mingfei Han, Changlin Li, Zhihui Li, Xiaojun Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16663.txt.
Introduction
Robots that roam by sight must juggle both flat views and depth maps, yet joint training data for all three streams is scarce. Haihong Hao and co-authors let a depth-wise model coach a sight-wise one instead, in work from twenty twenty-five. The coach shares its guesses as plain text during the walk.
Light tuning on a small mixed set teaches the walker to use those hints. Scores rose on four roaming tests and two space-question tests. At matched success rates, the coached walker often took shorter paths. The design skips full fusion: text hints carry the depth sense across, which keeps data needs low and lets each model play to its strength.
Catalog record
| CREATOR | Haihong Hao, Mingfei Han, Changlin Li, Zhihui Li, Xiaojun Chang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoNav CoNav : Collaborative Cross-Modal Reasoning for Embodied Navigation Haihong Hao, Mingfei Han, Changlin Li, Zhihui Li, Xiaojun Chang Abstract Embodied navigation demands comprehensive scene understanding and precise spatial reasoning. While image–text models excel at interpreting pixel-level color and lighting cues, 3D–text models capture volumetric structure and spatial relationships. However, unified fusion approaches that jointly fuse 2D images, 3D point clouds, and textual instructions face challenges in limited availability of triple-modality data and difficulty resolving conflicting beliefs among modalities. In this work, we introduce CoNav , a collaborative cross-modal reasoning framework where a pretrained 3D–text model explicitly guides a image-text navigation agent by providing structured spatial-semantic knowledge to resolve ambiguities during navigation. Specifically, we introduce Cross-Modal Belief Alignment, which operationalizes this cross-modal guidance by simply sharing textual hypotheses from the 3D–text model to the navigation agent. Through lightweight fine-tuning on a small 2D–3D–text corpus, the navigation agent learns to integrate visual cues with spatial-semantic knowledge derived from 3D-text model, enabling effective reasoning in Embodied navigation. CoNav achieves significant improvements on four standard embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial reasoning benchmarks (ScanQA, SQA3D). Moreover, under close navigation Success Rate, CoNav often generates shorter paths compared to other methods (as measured by SPL), showcasing the potential and challenges of fusing data from different modalities in embodied navigation. Project Page: https://oceanhao.github.io/CoNav/ 1 Introduction Embodied AI aims to enable agents to interact with the physical 3D world. One key challenge here is embodied navigation task [ 3 , 58 , 38 ] . In this task, the navigation agent needs to move through a 3D environment based on user natural-language instructions. In the past, most embodied navigation methods relied on only 2D visual cues [ 15 , 43 , 80 , 64 , 24 ] by sampling from real-world scenes, which limited the agents’ ability to understand spatial-semantic knowledge. More recently, 3D Large Language Models (3D-LLMs) have emerged, enabling agents to understand spatial relationships by integrating 3D cues like multiview [ 31 ] , bird’s-eye view [ 49 ] , voxel [ 48 ] , point cloud [ 28 , 69 , 72 ] etc. into LLMs. Despite this progress, applying 3D-LLMs to practical tasks like embodied navigation remains challenging. Recent fusion-based navigation methods have attempted to fuse 2D images, 3D point clouds, and text instructions into unified models. For example, LEO [ 29 ] employs three separate encoders to process 2D images, 3D point clouds, and text instructions. The output features from these encoders are then treated as tokens and fed into a single LLM, enabling cross-modal reasoning. LEO trained on 1539 K data pairs and fuse multimodal data, tackle tasks such as 3D scene captioning, embodied navigation and manipulation. Similarly, VER [ 48 ] uses two encoders to separately process text instructions and multi-view observations. The resulting features are then fused and passed through a shared multi-layer transformer to perform cross-modal reasoning. These methods integrate data from different modalities at feature-level within a single unified architecture. Fusion-based navigation methods face two drawbacks. First, they require large corpora of 2D–3D–text triples—yet high-quality datasets typically exist as image–text [ 16 , 10 , 20 , 8 ] or 3D–text pairs [ 51 , 19 , 84 , 69 ] , making comprehensive triple-modality collection difficult. Second, monolithic fusion at feature-level as shown in Figure 1 may obscure each modality’s unique characteristics and eliminates cross-verification by different modalities. Moreover, prior research [ 74 , 63 ] has shown that, under a fixed parameter budget, specialized modular processing often yields better performance than a single monolithic model. In fact, the image–text and 3D–text modalities each offer distinct strengths, as illustrated in Figure 1 . Image-text models rely on pixel-level color and illumination changes to infer boundaries and semantics [ 56 , 35 ] , 3D-text models prioritize spatial geometry and volumetric structure [ 12 , 17 ] , with less sensitivity to texture and color. Figure 1 : (a) Image–text navigation agents rely on only visual cues may fail when dealing with tasks involving spatial distance. Although 3D–text models excel in spatial reasoning, they are less sensitive to textures (e.g., map and painting on the wall). Embodied navigation often demands both visual cues and spatial geometry for reasoning. In the scene of Figure, the agent need to use visual cues to locate a world map and then apply spatial distance reasoning to determine which bedroom is nearest to that map. (b) Fusion of image and 3D can be achieved through three methods. Past unified fusion approaches often employed multimodal fusion at the feature level to complete cross-modal reasoning. By enabling text-based information sharing at the model level, CoNav achieves higher-level cross-modal reasoning. To address these challenges, we propose CoNav, a collaborative cross-modal reasoning framework for embodied navigation. In CoNav, the pretrained 3D–text model explicitly guides the image–text navigation agent by sharing textual spatial-semantic knowledge. To resolve conflicting beliefs among modalities, we perform lightweight fine-tuning of CoNav using only a small 2D–3D–text corpus, jointly performing the embodied navigation task. Specifically, (1) Separate Pretraining. We pretrain the image-text navigation agent on navigation image–text corpora and the 3D–text model on 3D–text pairs separately. (2) Cross-Modal Belief Alignment fine-tuning. To allows the 3D–text model guide the navigation agent using text, we design a Communication Interface that enable information sharing. On this basis, we perform lightweight fine-tuning on a small 2D–3D–text corpus. The agent learns to reason effectively by fusing visual cues with spatial-semantic knowledge derived from the provided 3D information, resolving ambiguities during navigation. In summary, the main contributions of this work are: • We introduce CoNav , a novel collaborative framework for embodied navigation that allows 3D–text model guide the navigation agent, by simply sharing textual hypotheses without requiring large-scale triple-modality datasets. • We demonstrate that Cross-Modal Belief Alignment is an efficient way for embodied agents to integrate visual cues with spatial-semantic knowledge. Model-level fusion for 2D images and 3D point clouds can reduce the inappropriate actions due to the ambiguities relying only on pixel-level color and visual cues. • Extensive experiments demonstrate CoNav’s superior performance not only on embodied navigation benchmarks, but also on spatial reasoning tasks. This integration of multiple available modalities and collaborate to complete task is essential for building a practical and general embodied navigation agent. 2 Related work Vision-language navigation (VLN). VLN aims to let the navigation agents follow linguistic navigation instructions and complete various navigation-related tasks [ 4 , 39 , 83 ] . Initial VLN methods mainly focused on encoding vision and text, and their cross-modal connection. PRESS [ 42 ] adopts BERT encoding instructions. NavGPT2 [ 80 ] employs Q-former to link vision and text, with topological maps supporting action planning. Recently, LLMs is introduced to embodied navigation due to their strong language understanding. NavCoT [ 44 ] fine-tunes LlaMA-7B with Chain of Thought to handle navigation tasks on text-only LLMs. NaviLLM [ 79 ] fine-tunes Vicuna-7B [ 18 ] with multi-task learning. RoomTour3D [ 24 ] enhances geometry-awareness by reconstructed 3D scenes. SAME [ 81 ] propose a new State-Adaptive Mixture of Experts model that enables an agent to infer decisions based on dynamic observations. SRDF [ 67 ] generates super large-scale navigational instruction-trajectory pairs, surpassing