Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations

Yibo Cui, Liang Xie, Yu Zhao, Jiawei Sun, Erwei Yin · 2025

Read Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations by Yibo Cui, Liang Xie, Yu Zhao, Jiawei Sun, Erwei Yin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08566.txt.

Introduction

Cui, Xie, Zhao, Sun, and Yin generate training data that teaches navigation agents to follow directions step by step, not just as one block. Agents that navigate by language and sight need fine links between phrases and path segments, yet existing datasets mostly match whole routes to whole texts. Their 2025 framework builds those links automatically.

A route is split into segments, landmarks are spotted by a vision model, and draft directions are written and polished by speaker models. A final filter picks the best landmark mentions, and the pieces merge into one full instruction pair. Agents trained on the new dataset improved across four leading systems. Gains came from both the step-level links and the landmark links. The authors stress no hand labeling was needed.

Catalog record

CREATORYibo Cui, Liang Xie, Yu Zhao, Jiawei Sun, Erwei Yin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations Yibo Cui, Liang Xie, Yu Zhao, Jiawei Sun, Erwei Yin Abstract Vision-Language Navigation (VLN) enables intelligent agents to navigate environments by integrating visual perception and natural language instructions, yet faces significant challenges due to the scarcity of fine-grained cross-modal alignment annotations. Existing datasets primarily focus on global instruction-trajectory matching, neglecting sub-instruction-level and entity-level alignments critical for accurate navigation action decision-making. To address this limitation, we propose FCA-NIG, a generative framework that automatically constructs navigation instructions with dual-level fine-grained cross-modal annotations. In this framework, an augmented trajectory is first divided into sub-trajectories, which are then processed through GLIP-based landmark detection, crafted instruction construction, OFA-Speaker based R2R-like instruction generation, and CLIP-powered entity selection, generating sub-instruction-trajectory pairs with entity-landmark annotations. Finally, these sub-pairs are aggregated to form a complete instruction-trajectory pair. The framework generates the FCA-R2R dataset, the first large-scale augmentation dataset featuring precise sub-instruction-sub-trajectory and entity-landmark alignments. Extensive experiments demonstrate that training with FCA-R2R significantly improves the performance of multiple state-of-the-art VLN agents, including SF, EnvDrop, RecBERT, and HAMT. Incorporating sub-instruction-trajectory alignment enhances agents’ state awareness and decision accuracy, while entity-landmark alignment further boosts navigation performance and generalization. These results highlight the effectiveness of FCA-NIG in generating high-quality, scalable training data without manual annotation, advancing fine-grained cross-modal learning in complex navigation tasks. keywords Vision-Language Navigation,Data Augmentation,Fine-Grained Alignment,Instruction Generation † † corresponding: Corresponding author 1 Introduction Vision-Language Navigation (VLN) represents a pivotal task within embodied artificial intelligence, encapsulating the integration of perceptual understanding from visual input and semantic interpretation derived from natural language instructions to enable intelligent agents to navigate physical or virtual environments [ 3 , 52 ] . VLN has attracted burgeoning research interest from communities of computer vision, natural language processing, and robotics due to its broad application prospects [ 19 , 14 , 16 ] . In VLN tasks, cross-modal alignment is one central challenge for decision-making since the agent requires dynamically ground the received instructions to the surrounding visual observations. For instance, given the instruction ”Walk beside the outside doors and behind the chairs across the room. Turn right and walk up the stairs. Stop on the seventh step.”, the agent needs to find ”outside doors”, ”chairs”, ”stairs” etc. (entity level) in nearby environments to help make action decisions and disregard the initial sub-instructions ”Walk beside the outside doors” (sub-instruction level) after moving several steps. However, due to the scarcity of VLN datasets with these fine-grained alignment signals, previous methods have focused on global matching between instructions and complete visual trajectories (instruction level), leaving cross-modal alignment a significant challenge. Figure 1: An illustration of our speaker-follower framework. The follower (blue) navigates through a 3D photorealistic environment, while the speaker (green) generates grounded descriptions of navigation trajectories along with fine-grained alignment annotations. Several prior works have explored fine-grained cross-modal alignment in VLN. Self-Monitoring [ 40 ] located completed or ongoing sub-instructions through visual-textual co-grounding and a progress monitor module during navigation. AuxRN [ 69 ] reported the percentage of completed procedures at each step using an auxiliary reasoning objective for progress estimation. Qi et al. [ 48 ] designed an object-aware module to enable matching between object-centered instructions and their corresponding visual observations. EntityGraph [ 23 ] modeled visual-textual correlations at the entity level via a language and visual entity-relationship graph. ORIST [ 47 ] used object-level features instead of view-level features as input to a sequential BERT model. Moudgil et al. [ 43 ] developed a scene- and object-aware transformer model using attention masking and view aggregation to explicitly represent objects and scenes. However, these works rely on weak supervision from environment information or coarse-grained data due to the absence of fine-grained alignment signals in visual and textual modalities. Subsequent efforts have aimed to address the data scarcity problem. Zhu et al. [ 70 ] and Hong et al. [ 24 ] first segmented long instructions into sub-instructions using template matching, then grounded them to visual sub-trajectories via dynamic programming and human annotation, respectively. Wang et al. [ 64 ] recognized entity phrases in instructions using text parsers and identified matching environmental landmarks through object detection and weak supervision from pose traces. However, these fine-grained cross-modal alignment signals lack precision. To construct high-quality alignments, He et al. [ 22 ] manually annotated the matching relationships between sub-instructions and sub-trajectories. Additionally, Cui et al. [ 12 ] introduced the GEL-R2R dataset, enriched with grounded entity-landmark annotations. While these efforts improved data quality, their datasets are insufficient and difficult to expand due to the high cost of human annotation. To address the challenges outlined above, we introduce a Generative framework for Navigation Instruction annotated with Fine-grained Cross-modal Alignment (FCA-NIG). This framework is designed to generate trajectory descriptions and simultaneously annotate them with detailed cross-modal information. The FCA-NIG framework operates through six key sub-steps: trajectory chunking, landmark detection, crafted instruction construction, R2R-like instruction generation, entity selection, and sub-pair integration. Broadly, the FCA-NIG framework first segments a randomly sampled augmented trajectory into multiple sub-trajectories. These sub-trajectories are then processed sequentially through the intermediate steps to generate corresponding sub-instructions, forming several sub-instruction sub-trajectory pairs annotated with entity-landmark matches. Finally, these sub-pairs are concatenated to construct the complete instruction-trajectory pair. During the landmark detection phase, we employ the open-world object detection model GLIP [ 35 ] to identify salient objects or scenes in the panoramic observation images of the sub-trajectories, which serve as reference landmarks. In the crafted instruction construction phase, we first establish a template instruction library comprising three core components: horizontal turning, vertical movement adjustments, and spatial relationships with target objects. Leveraging these templates, we construct crafted sub-instructions for each sub-trajectory based on the detected landmarks and associated action turning data. In the R2R-like instruction generation phase, we design a speaker model based on the multi-modal pre-trained language generation model OFA [ 63 ] , named OFA-Speaker. This model includes a panorama encoder, a vision encoder, a text encoder, a cross-modal encoder, and a text decoder, and is fine-tuned on the R2R dataset. Specifically, the OFA-Speaker uses a Transformer-based panorama encoder to extract features from the panoramic images of the sub-trajectories. These features are then concatenated with prompt words and template instructions as input to the OFA model to generate navigation sub-instructions similar to those in the R2R dataset. In the entity selection phase, we first parse the generated sub-instructions using the NLTK [ 4 ] to extract the entities contained within them. We then employ the large-scale multi-modal pre-trained model CLIP [ 51 ] to evaluate the similarity between each extracted entity and the detected visual landmarks. This process identifies the most relevant entity, which is then paired with the corresponding visual landmark to form an entity-landmark match pair. Using the FCA-NIG framework, we generate an augmentation dataset with fine-grained cross-modal alignment annotations, named FCA-R2R. We conduct extensive experiments to evaluate the quality of instruction-trajectory pairs in the FCA-R2R dataset and the effects of two types of fine-grained alignment information: sub-instruction sub-trajectory matching and entity-landmark matching. The evaluation involves four classic VLN agent models (SF [ 17 ] , EnvDrop [ 55 ] , RecBERT [ 25 ] , and HAMT [ 7 ] ), showing that incorporating the FCA-R2R dataset significantly improves

Read the complete text