NaVIDA : Vision-Language Navigation with Inverse Dynamics Augmentation

Weiye Zhu Zekai Zhang Xiangchen Wang Hewei Pan, Teng Wang Tiantian Geng Rongtao Xu Feng Zheng · 2026

Read NaVIDA : Vision-Language Navigation with Inverse Dynamics Augmentation by Weiye Zhu Zekai Zhang Xiangchen Wang Hewei Pan, Teng Wang Tiantian Geng Rongtao Xu Feng Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18188.txt.

Introduction

Weiye Zhu and co-authors present NaVIDA, a navigation agent that learns how its own actions change the world. Their 2026 preprint deals with vision-and-language navigation. An agent reads a natural language instruction and acts on it inside a rich visual scene. Most methods learn reactive links from state to action. They do not model how an action causally reshapes the scene, so errors can compound step by step.

NaVIDA adds inverse dynamics augmentation to the training. Inverse dynamics asks which action could have caused a seen change. That signal teaches cause and effect, not just correlation. Agents that grasp it should follow routes with fewer wrong turns.

Catalog record

CREATORWeiye Zhu Zekai Zhang Xiangchen Wang Hewei Pan, Teng Wang Tiantian Geng Rongtao Xu Feng Zheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18188

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NaVIDA : Vision-Language Navigation with Inverse Dynamics Augmentation Weiye Zhu Zekai Zhang Xiangchen Wang Hewei Pan, Teng Wang Tiantian Geng Rongtao Xu Feng Zheng Abstract Vision-and-Language Navigation (VLN) requires agents to interpret natural language instructions and act coherently in visually rich environments. However, most existing methods rely on reactive state–action mappings without explicitly modeling how actions causally transform subsequent visual observations. Lacking such vision–action causality, agents cannot anticipate the visual changes induced by its own actions, leading to unstable behaviors, weak generalization, and cumulative error along trajectory. To address these issues, we introduce NaVIDA ( Nav igation with I nverse D ynamics A ugmentation), a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution. NaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions. To structure this supervision and extend the effective planning range, NaVIDA employs hierarchical probabilistic action chunking (HPAC), which organizes trajectories into multi-step chunks and provides discriminative, longer-range visual-change cues. To further curb error accumulation and stabilize behavior at inference, an entropy-guided mechanism adaptively sets the execution horizon of action chunks. Extensive experiments show that NaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters (3B vs. 8B). Real-world robot evaluations further validate the practical feasibility and effectiveness of our approach. Code and data will be available upon acceptance. 1 Introduction Figure 1 : Comparision of three VLN paradigms. (a) Vanilla VLN relies on state-action mapping, lacking the causality between vision and actions; (b) Forward Dynamics Modeling (FDM) adds an additional image generation task which predicts future observations but is compute-heavy and distractible; (c) NaVIDA uses inverse dynamics supervision to learn the causality between vision and actions with the action predicition task only, making the learning process more efficient and stable. Vision-and-Language Navigation (VLN) 3 ; 19 ; 32 is a challenging multimodal sequential decision-making task, requiring an agent to navigate effectively through a 3D visual environment based solely on natural language instructions. Accordingly, effective VLN requires not only precise instruction comprehension but also long-horizon temporal modeling and robust generalization to visually diverse, unseen scenes. Motivated by advances in video-based Multimodal Large Language Models (MLLMs) 4 ; 23 ; 48 , recent studies 46 ; 45 ; 8 ; 39 ; 34 ; 49 have integrated these models into VLN to exploit their superior visual perception, semantic understanding, and cross-scene generalization. Many existing works 46 ; 45 ; 8 typically formulate VLN as a short-sighted state-to-action reactive strategy. As shown in Fig. 1 (a), they map current language instruction and past visual observations to a few subsequent actions. This simplification ignores the potential causal relationship between visual information changes and corresponding actions in navigation tasks: how current visual observations are transferred to the next visual observation through the influence of actions . The model cannot predict the visual consequences of its actions, thus failing to correct small biases that gradually accumulate into serious navigational instability, such as future viewpoint misalignment caused by slight premature turns. As shown in Fig. 1 (b), recently some visual prediction-based works 34 ; 47 ; 16 combine forward dynamics modeling (FDM), attempting to use future observations as auxiliary signals for prediction, but such image generation tasks usually require additional prediction modules, such as diffusion-based world models 9 ; 6 , which introduces additional computational burdens 31 ; 47 and causes the model to be confined to redundant details 35 ; 47 . To address these challenges, inspired by the inverse dynamic models (IDM), we propose NaVIDA , a VLN framework that integrates inverse-dynamics supervision (IDS) as an auxiliary signal to learn the causal relationship between visual changes and corresponding actions. This generates a language-independent visual motion concept, allowing the model to understand the physical consequences of its predicted actions without predicting dense image features, even if it cannot see the future frames. To provide IDS with richer and more discriminative learning signals, we introduce a hierarchical probabilistic action chunking (HPAC) mechanism that progressively merges low-level actions into multiple levels of action chunks. Specifically, we first group temporally adjacent atomic actions with consistent visual transition patterns into sub-chunks, and then further merge these sub-chunks into higher-level chunks, thus representing more complete and semantically coherent behavior. Such hierarchical mechanism allows the agent to predict over extended temporal horizons, leading to more stable navigation. Moreover, during inference, we propose an entropy-guided execution selection mechanism that adaptively adjusts the execution horizon of predicted action chunks according to their uncertainty, allowing for early replanning and timely trajectory correction. Extensive experiments demonstrate that NaVIDA significantly enhances the causal understanding of visual actions, achieving state-of-the-art (SOTA) performance on the VLN-CE 19 benchmarks with fewer parameters, thus establishing a simple and lightweight paradigm for VLN research. In summary, our contributions are threefold: • We propose NaVIDA , a novel VLN framework that explicitly integrates inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions, enabling stable navigation behaviors. • The NaVIDA adopts a HPAC mechanism to provide the inverse-dynamics supervision with richer and more discriminative learning signals, together with a dynamic entropy-guided execution horizon selection strategy to mitigate error accumulation during inference. • Comprehensive experiments on VLN-CE benchmarks demonstrate that NaVIDA attains SOTA performance with fewer parameters, confirming the efficiency and effectiveness of our design and highlighting the value of inverse dynamic modeling for VLN. 2 Related Work Vision-and-Language Navigation. VLN tasks require an embodied agent to perceive, reason, and act in response to natural-language instructions. Early studies 15 ; 7 ; 22 focused on discrete navigation on connectivity graphs with panoramic RGB–D inputs, achieving strong in-domain results but relying on heavy sensors and pre-scanned viewpoints. Subsequent works 40 ; 19 have moved toward continuous control and low-level action prediction to better reflect real-world deployment, while several efforts 14 ; 36 ; 1 ; 11 ; 25 ; 30 ; 29 have investigated monocular RGB or RGB–D variants that relax reliance on panoramic sensing. Concurrently, the emergence of MLLMs has spurred monocular-only VLN systems that leverage vision–language backbones for stronger cross-scene transfer and lower hardware cost 46 ; 45 ; 8 ; 39 . Despite these advances, most VLN methods remain limited to reactive state-to-action mapping modeling, neglecting the potential causal relationship between changes in visual observation and corresponding actions. Lacking such causal grounding, agents struggle to anticipate visual consequences, which leads to unstable turning or stop behaviors and long-horizon error accumulation. In contrast, our NaVIDA explicitly encodes the action and induced visual change through inverse-dynamics supervision, enabling the policy to couple language grounding with short-horizon causal prediction and thereby stabilizing execution under monocular sensing. Dynamic Models in Navigation. Reactive state-action modeling directly maps historical visual observations and natural language instructions to the action space, lacking causal modeling between changes in visual information and actions. To mitigate this, early works 12 ; 21 ; 41 explored using additional pre-training tasks such as path-mask prediction, image prediction, or image completion. However, these were mostly limited to discrete environmental settings and often relied on extended contextual cues or language instructions. More recent methods 34 ; 47 ; 16 attempted to combine forward dynamics modeling and decompose the decision task, predicting future images while simultaneously predicting actions.

Read the complete text