Uncertainty-Aware Gaussian Map for Vision-Language Navigation
Jianzhe Gao, Rui Liu, Yuxuan Xu, Tongtong Cao, Yingxue Zhang, Zhanguang Zhang, Sida Peng, Yi Yang, Wenguan Wang · 2026
Read Uncertainty-Aware Gaussian Map for Vision-Language Navigation by Jianzhe Gao, Rui Liu, Yuxuan Xu, Tongtong Cao, Yingxue Zhang, Zhanguang Zhang, Sida Peng, Yi Yang, Wenguan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26503.txt.
Introduction
Jianzhe Gao and eight colleagues make navigation agents weigh perceptual doubt instead of ignoring it. Their agent builds a Semantic Gaussian Map from panoramic views, encoding scene shape and meaning in 3D Gaussian units. Shape doubt comes from small jitters of position and scale. Meaning doubt comes from jitters of meaning traits. Looks doubt comes from Fisher Information on view sensitivity. Folded into one 3D Value Map, these doubts act as affordances and constraints for reliable choices. Tests across several navigation benchmarks show the approach works.
Catalog record
| CREATOR | Jianzhe Gao, Rui Liu, Yuxuan Xu, Tongtong Cao, Yingxue Zhang, Zhanguang Zhang, Sida Peng, Yi Yang, Wenguan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67543 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26503
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertainty-Aware Gaussian Map for Vision-Language Navigation Jianzhe Gao, Rui Liu, Yuxuan Xu, Tongtong Cao, Yingxue Zhang, Zhanguang Zhang, Sida Peng, Yi Yang, Wenguan Wang Abstract Vision-Language Navigation (VLN) requires an agent to navigate 3D environments following natural language instructions. During navigation, existing agents commonly encounter perceptual uncertainty, such as insufficient evidence for reliable grounding or ambiguity in interpreting spatial cues, yet they typically ignore such information when predicting actions. In this work, we explicitly model three forms of perceptual uncertainty ( i.e. , geometric, semantic, and appearance uncertainty) and integrate them into the agent’s observation space to enable informed decision-making. Concretely, our agent first constructs a Semantic Gaussian Map (SGM), composed of differentiable 3D Gaussian primitives initialized from panoramic observations, that encodes both the geometric structure and semantic content of the environment. On top of SGM, geometric uncertainty is estimated through variational perturbations of Gaussian position and scale to assess structural reliability; semantic uncertainty is captured by perturbing Gaussian semantic attributes to reveal ambiguous interpretations; and appearance uncertainty is characterized by Fisher Information, which measures the sensitivity of rendered observations to Gaussian-level variations. These uncertainties are incorporated into SGM, extending it into a unified 3D Value Map, which grounds them as affordances and constraints that support reliable navigation. Comprehensive evaluations across multiple VLN benchmarks show the effectiveness of our agent. 1 Introduction Vision-Language Navigation (VLN) requires embodied agents to navigate diverse 3D environments following natural language instructions 4 , 26 , 3 , 53 , 27 . To achieve robust performance, agents must combine accurate spatial perception with reliable decision-making strategies 18 , 76 , 77 , 17 . Early agents adopted sequence-to-sequence frameworks 4 , 74 , directly mapping language and visual observations into actions. Later works introduced map-based paradigms that explicitly encoded spatial connectivity via topological graphs 17 , 13 , incorporated semantic information for object-level reasoning 33 , 79 , 25 , and leveraged grid-based 2 , 87 or volumetric voxel-based representations 54 to capture 3D structure. For policy learning, agents evolved from pure imitation 63 , 1 to hybrid approaches that combine imitation and reinforcement with tailored rewards 74 , 84 , 32 , 81 , 97 . More recently, several agents have employed world models 40 , 78 , 7 to perform look-ahead planning. Despite these advances, existing agents typically ignore uncertainty in perception when making decisions. Their training recipes discourage expressing uncertainty or recognizing unreliable situations, instead incentivizing them to predict actions regardless of confidence 54 . For instance, as illustrated in Fig. 1 , agents may confuse visually similar doors, especially when the interior cues behind them provide insufficient evidence, leading to unreliable grounding of the correct target. Besides, Fig. 1 illustrates how occlusions mask critical spatial information, introducing ambiguity in assessing path traversability. Previous agents often fail under such conditions, whereas uncertainty offers valuable cues about the reliability of perception and the feasibility of actions 13 , 2 , 54 . Figure 1: Motivation. Previous VLN agents typically ignore perceptual uncertainty when making decisions. As a result, they often confuse visually similar structures ( e.g. , multiple doors) due to limited interior evidence ( ) and struggle when occlusions obscure spatial cues, leaving traversability ambiguous and causing unsafe or suboptimal paths ( ). In contrast, our agent explicitly models and leverages such uncertainty for more reliable navigation. Brighter colors indicate higher uncertainty. In light of the foregoing discussions, this work explicitly models geometric, semantic, and appearance uncertainty in perception and consolidates them into a unified 3D Value Map for reliable navigation. First , our agent constructs a Semantic Gaussian Map (SGM) that represents the environment as a collection of differentiable 3D Gaussian primitives. Each primitive is initialized from sparse pseudo-lidar point clouds obtained from multi-view RGB-D observations and further enriched with semantic properties based on their object instance or stuff membership in the 3D scene. Second , building on SGM, the agent estimates three forms of perceptual uncertainty in navigation. Geometric uncertainty is modeled through variational inference, which approximates the posterior distribution over position and scale perturbations of Gaussians, thereby assessing structural reliability and enabling the pruning of unreliable primitives. In the same manner, semantic uncertainty is estimated by perturbing the semantic attributes of Gaussians, which reveals ambiguous interpretations and allows the agent to down-weight unreliable semantic cues during decision-making. Appearance uncertainty reflects the sensitivity of rendered observations to Gaussian-level perturbations, quantified by Fisher Information as the reconstruction loss surface curvature around each Gaussian. Third , our agent composes a unified 3D Value Map by transforming these uncertainties into affordances and constraints within its perceptual space, thereby guiding informed and more reliable trajectories. Our agent is evaluated on three VLN benchmarks, i.e. , R2R 4 , RxR 43 , and REVERIE 64 (§ 4.2 , § 4.3 ). It improves SR by 2 % and SPL by 1 % on R2R, yields 1.1 % and 1.7 % increases in SR and nDTW on RxR with comparable SDTW, and achieves 2.94 % and 2.57 % higher RGS and RGSPL scores on REVERIE. Extensive ablation studies confirm the contribution of each component (§ 4.4 ). 2 Related Work Vision-Language Navigation (VLN). Early VLN agents adopted sequence-to-sequence frameworks that directly map instructions and multi-view observations to actions 23 , 4 , 74 . Yet such models struggle with long-horizon reasoning and robustness in unseen environments, which has spurred a variety of extensions. As a primary step, subsequent agents introduced explicit memory mechanisms, such as topological graphs 13 , 55 or episodic memory buffers 15 , 14 , 80 , to better retain and recall spatial and semantic cues over extended trajectories. Later works further advanced the agent with transformer-based architectures that jointly encode instruction and observation 31 , 78 . Moreover, extensive efforts are devoted to mitigating data limitations through instruction generation 98 , 41 , 20 , 21 , 88 and synthetic data creation 7 . Several works enable the agent to explore steps forward by anticipating future observations before decision-making 40 , 78 , 62 . To improve policy robustness, the combination of imitation learning 5 and reinforcement learning 85 has been widely adopted in VLN agents 84 . Others aim to reduce computational costs while maintaining VLN performance by designing lightweight cross-modal or selective memorization architectures 96 . Benefiting from the quality and speed of 3D Gaussian Splatting 37 , 51 , 92 , a growing line of work has adopted it as the agent’s scene representation, showing strong performance 45 , 22 , 27 , 52 . In parallel, a few recent studies begin to explore uncertainty-related signals in VLN. For instance, VLN-Copilot 65 estimates decision-level uncertainty from the action distribution to decide when to request external large language model assistance. In contrast, our work focuses on perception-level uncertainty that arises from the agent’s observations and models geometric, semantic, and appearance reliability to support informed decision-making. Uncertainty Estimation in Deep Learning. Uncertainty estimation has long been recognized as a central challenge in deep learning, with a variety of approaches proposed across vision and robotics 60 . A prominent line of work follows the Bayesian paradigm, which characterizes uncertainty through predictive distributions over model parameters, often approximated via variational inference, Laplace approximation,