What Limits Vision-and-Language Navigation ?

Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li, Kun Liu, Junzhe Xu, Zizhao Yuan, Yixiao Feng, Jiaxi Zhang, Wei Lu, Zecui Zeng, Renjing Xu · 2026

Read What Limits Vision-and-Language Navigation ? by Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li, Kun Liu, Junzhe Xu, Zizhao Yuan, Yixiao Feng, Jiaxi Zhang, Wei Lu, Zecui Zeng, Renjing Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13328.txt.

Introduction

Wang and colleagues present StereoNav, a vision language action system for real world robot navigation. Simulated training often fails outdoors, where light shifts, motion blur and vague instructions break perception. Target location priors give stable visual guidance across domains, while stereo vision fuses meaning and depth into one view. It tops prior egocentric RGB scores on two benchmarks while using fewer parameters and less data, and real robot trials confirm steadier performance outdoors.

Catalog record

CREATORYunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li, Kun Liu, Junzhe Xu, Zizhao Yuan, Yixiao Feng, Jiaxi Zhang, Wei Lu, Zecui Zeng, Renjing Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54916
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Limits Vision-and-Language Navigation ? Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li, Kun Liu, Junzhe Xu, Zizhao Yuan, Yixiao Feng, Jiaxi Zhang, Wei Lu, Zecui Zeng, Renjing Xu Abstract Vision-and-Language Navigation (VLN) is a cornerstone of embodied intelligence. However, current agents often suffer from significant performance degradation when transitioning from simulation to real-world deployment, primarily due to perceptual instability (e.g., lighting variations and motion blur) and under-specified instructions. While existing methods attempt to bridge this gap by scaling up model size and training data, we argue that the bottleneck lies in the lack of robust spatial grounding and cross-domain priors. In this paper, we propose StereoNav, a robust Vision-Language-Action framework designed to enhance real-world navigation consistency. To address the inherent gap between synthetic training and physical execution, we introduce Target-Location Priors as a persistent bridge. These priors provide stable visual guidance that remains invariant across domains, effectively grounding the agent even when instructions are vague. Furthermore, to mitigate visual disturbances like motion blur and illumination shifts, StereoNav leverages stereo vision to construct a unified representation of semantics and geometry, enabling precise action prediction through enhanced depth awareness. Extensive experiments on R2R-CE and RxR-CE demonstrate that StereoNav achieves state-of-the-art egocentric RGB performance, with SR and SPL scores of 81.1% and 68.3%, and 67.5% and 52.0%, respectively, while using significantly fewer parameters and less training data than prior scaling-based approaches. More importantly, real-world robotic deployments confirm that StereoNav substantially improves navigation reliability in complex, unstructured environments. Project page: Link . 1 Introduction Vision-and-Language Navigation (VLN) [ 13 ] is a cornerstone of embodied intelligence, requiring agents to ground natural language instructions into sequential egocentric observations [ 56 ] . Ideally, a robust VLN system should serve as a dependable primitive for advanced downstream tasks, such as mobile manipulation [ 48 ] and loco-manipulation [ 9 ] , where success critically depends on reaching a target with precise spatial alignment. However, despite significant progress in simulation, current VLN agents often exhibit brittle behavior and execution inconsistency when deployed in real-world environments. They struggle to maintain performance amidst the complexities of physical reality, such as varying lighting, camera shake, and unstructured layouts. A prevailing consensus in the community attributes these failures to insufficient cross-modal understanding, leading to a paradigm of "scaling": upgrading to larger and stronger Vision-Language Model (VLM) backbones [ 53 , 9 ] and expanding training corpora with auxiliary data [ 41 ] . As illustrated in Fig. 1 , while these strategies have pushed the state-of-the-art, they are reaching a point of diminishing returns. Models built on comparable backbones show saturated performance, and the correlation between training data scale and success remains surprisingly weak. This suggests that the primary bottleneck may not be "understanding" alone, but rather the lack of robust structural priors that can bridge the gap between abstract instructions and noisy physical perceptions. In this paper, we argue that real-world performance degradation stems from two neglected factors: environmental perception instability and instructional under-specification. In physical settings, egocentric observations are frequently corrupted by motion blur and illumination shifts, which undermine action prediction in models trained on pristine synthetic data. Furthermore, natural language instructions are inherently sparse, providing insufficient guidance for long-range navigation. While humans often navigate using a mental coarse map alongside visual cues, current agents are forced to infer implicit spatial goals solely from ambiguous text, a task that remains difficult to scale through data alone. Figure 1: Performance under different backbones and training data scales. (a) Left: The red trend line shows that success rate increases over time with diminishing returns. The inset reports the MLVU-Dev [ 58 ] scores of the adopted backbone VLMs, indicating that stronger backbones improve early performance, but their gains gradually saturate. (b) Right: Arrows indicate the performance gains from increased training data for each method, while the red trend line suggests that training data scale has only a weak overall correlation with navigation performance across methods. We propose StereoNav, a stereo Vision-Language-Action framework designed for robust embodied navigation. Addressing the challenge of under-specified instructions, we introduce Target-Point Priors that are derived from the coarse or precise knowledge typically available in practical scenarios. By rendering these priors as persistent visual cues, StereoNav equips the agent with global information guidance, ensuring the goal remains spatially anchored despite incomplete linguistic descriptions. Seeking to mitigate perceptual instability (e.g., lighting variations and motion blur), our framework moves beyond conventional monocular perception by incorporating stereo observations. This approach enables unified semantic, structural, and geometric modeling, allowing the agent to maintain stable perception and reliable decision-making through joint action and depth prediction. Our contributions are threefold: • We identify perceptual instability (e.g., lighting and motion blur) and instruction under-specification as key factors that hinder the Sim-to-Real transfer of VLN agents. • We propose StereoNav, which integrates target-point-prior-guided navigation to provide persistent global guidance and stereo-based geometric modeling to ensure robust environmental perception. • We demonstrate that StereoNav achieves state-of-the-art performance on competitive benchmarks while exhibiting significantly higher deployment reliability and execution consistency in real-world robotic tasks compared to monocular alternatives. 2 Related Work 2.1 Vision-and-Language Navigation Vision-and-Language Navigation (VLN) studies how an embodied agent executes natural-language instructions within complex, visually grounded environments [ 13 ] . Early research focused on discrete graph-based formulations [ 2 , 6 ] , which have since evolved toward more realistic continuous settings [ 17 , 29 ] . A predominant trend in the field emphasizes enhancing the agent’s cross-modal understanding as the primary path to performance gains. Consequently, recent state-of-the-art methods have leveraged increasingly powerful pre-trained backbones [ 40 , 12 , 25 , 32 ] , massive-scale navigation and auxiliary datasets [ 51 , 41 , 52 , 9 ] , and sophisticated temporal modeling [ 50 , 22 , 15 , 24 ] . While these advancements have significantly pushed the performance frontier, it is increasingly recognized that understanding ability is not the sole determinant of success in VLN. Recent studies have begun to highlight critical external factors that pose significant challenges to even the most capable agents. Specifically, natural-language instructions often contain inherent ambiguities [ 43 , 14 ] , and agents frequently exhibit sensitivity to visual disturbances or environmental shifts [ 55 , 34 ] . Despite their importance, these factors—instruction ambiguity and visual robustness—are often treated as anecdotal observations rather than being systematically analyzed. Their individual and joint impacts on navigation performance remain insufficiently quantified. Motivated by this, we conduct analysis studies to show how these factors constrain current VLN agents, aiming to provide a more nuanced understanding of the remaining bottlenecks in the field. 2.2 Vision-Language-Action Models for VLN Recent advances in large-scale Vision-Language Models (VLMs) [ 4 , 20 , 54 , 21 , 27 ] have inspired a growing body of work that adapts them to Vision-and-Language Navigation. A representative direction is to fine-tune pre-trained VLMs to directly map observations and language instructions to low-level navigation actions in an end-to-end manner, forming Vision-Language-Action models for VLN [ 53 , 52 , 9 , 41 , 49 , 50 ] . Benefiting from the broad visual-semantic priors learned from large-scale internet data, this paradigm provides a promising route for policy learning in complex environments [

Read the complete text