HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hi erarchical Mem ory System

Kailin Lyu, Kangyi Wu, Pengna Li, Xiuyu Hu, Qingyi Si, Cui Miao, Ning Yang, Zihang Wang, Long Xiao, Lianyu Hu, Jingyuan Sun, Ce Hao · 2026

Read HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hi erarchical Mem ory System by Kailin Lyu, Kangyi Wu, Pengna Li, Xiuyu Hu, Qingyi Si, Cui Miao, Ning Yang, Zihang Wang, Long Xiao, Lianyu Hu, Jingyuan Sun, Ce Hao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14807.txt.

Introduction

Lyu and co-authors cure navigation amnesia in open-source vision-language navigation with HiMemVLN. Zero-shot navigators lean on closed models, with cost and leakage risks. Open models with spatiotemporal reasoning still trail far behind. Step-by-step logs expose a memory failure the team names Navigation Amnesia. A hierarchical memory system joins the multimodal model to fix it. Tests in simulation and real settings back the design. Scores near twice the open-source state of the art, they report.

Catalog record

CREATORKailin Lyu, Kangyi Wu, Pengna Li, Xiuyu Hu, Qingyi Si, Cui Miao, Ning Yang, Zihang Wang, Long Xiao, Lianyu Hu, Jingyuan Sun, Ce Hao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31365
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hi erarchical Mem ory System Kailin Lyu, Kangyi Wu, Pengna Li, Xiuyu Hu, Qingyi Si, Cui Miao, Ning Yang, Zihang Wang, Long Xiao, Lianyu Hu, Jingyuan Sun, Ce Hao Abstract LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) task. However, most zero-shot methods primarily rely on closed-source LLMs as navigators, which face challenges related to high token costs and potential data leakage risks. Recent efforts have attempted to address this by using open-source LLMs combined with a spatiotemporal CoT framework, but they still fall far short compared to closed-source models. In this work, we identify a critical issue, Navigation Amnesia, through a detailed analysis of the navigation process. This issue leads to navigation failures and amplifies the gap between open-source and closed-source methods. To address this, we propose HiMemVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception recall and long-term localization, mitigating the amnesia issue and improving the agent’s navigation performance. Extensive experiments in both simulated and real-world environments demonstrate that HiMemVLN achieves nearly twice the performance of the open-source state-of-the-art method. The code is available at https://github.com/lvkailin0118/HiMemVLN . I Introduction Vision-and-Language Navigation (VLN) is a fundamental task in Embodied AI [ 6 ] . It requires the agent to navigate in novel environments according to natural language instructions. Although supervised learning has achieved strong results on VLN benchmark datasets, its dependence on large-scale human annotations substantially limits generalization to unseen environments [ 4 , 14 , 3 ] . With the rapid progress of large language models (LLMs) [ 1 ] , zero-shot navigation driven by commonsense reasoning has become a promising research direction. However, many state-of-the-art approaches rely on closed-source cloud-hosted models [ 10 , 26 ] , such as GPT-4o [ 16 ] . Despite their strong reasoning capability, cloud dependence introduces practical barriers in real deployments. As illustrated in Fig. 1 (a), these barriers include high API costs and unpredictable communication latency. More critically, uploading real-time visual streams that may contain sensitive spatial information to external servers raises privacy and security risks, which hinders deployment in home service and industrial robotics. Consequently, deploying open-source LLMs (e.g., Llama [ 15 ] and Qwen [ 30 ] ) directly on the robot has become an important pathway toward secure and autonomous embodied agents. Fig. 1: (a) Comparison between GPT-based Navigator and open-source LLM-based Navigator. (b) The Phenomenon of ”Navigation Amnesia”. Short term amnesia leads the agent to enter local loops or perform redundant exploration, whereas long term amnesia causes it to forget global instruction constraints, making decisions drift from the intended route. Although prior work has explored using open-source LLM-based Navigator in continuous environments, a substantial performance gap remains relative to closed-source models [ 23 ] . We argue that a primary driver of this gap is insufficient memory awareness of environmental state and task progress, which leads agents to exhibit an “navigation amnesia” phenomenon during multi step. As shown in Fig. 1 (b), from the perspective of short term perception, existing approaches often convert continuous, dynamic visual streams into discrete textual descriptions, which yields incomplete perception and impoverished scene understanding [ 23 , 37 , 21 ] . In continuous environments, adjacent viewpoints or visually similar scenes frequently produce near duplicate descriptions that fail to preserve discriminative visual cues. As a result, the agent struggles to reliably identify the current state, and may enter local loops or trigger redundant exploration. From the perspective of long horizon planning, open-source models typically have smaller context windows and weaker implicit memory capacity than closed-source models [ 7 , 18 ] , making it difficult to maintain coherent state estimates over extended trajectories. As navigation steps increases and reasoning noise accumulates, the agent is prone to forgetting global constraints in the instruction, causing decisions to drift from the intended route and reducing the ability to self correct. To overcome this problem, we resort to neurobiology [ 22 ] , specifically the Hippocampal Memory Indexing Theory [ 29 ] : Distinct neural circuits coordinate short term and long term memory to support spatial cognition and decision making. Short term memory relies on rapid perceptual processes that encode current visual input, whereas long term memory consolidates and retrieves abstract concepts to maintain a coherent global representation. While this cognitive theory reveals the essence of human cognition, it can be smoothly translated into an architectural principle of VLN: short term memory is visually dominant, enhancing the agent’s immediate perception and state representation of the current scene, while long term memory is semantically dominant, providing stable semantic grounding, generalized knowledge, and contextual information to complete the perception to reasoning to decision pipeline. Based on such inspiration, we propose HiMemVLN , a cognitively grounded framework that hierarchically integrates short term and long term memory into VLN. To address short term memory deficiencies, we introduce a Short Term Localer System . Unlike prior approaches that rely solely on textual histories, we design a Visual Graph Memory module that stores visual features from previously visited viewpoints and aligns the current observation with historical nodes at each navigation step to explicitly localize the agent state. This mechanism enables visual retrospection, allowing the agent to reliably detect revisits and to impose soft constraints that suppress redundant exploration. To mitigate long horizon planning drift, we further propose a Long Term Globaler System . At the beginning of an episode, this system extracts a global navigation schema from the instruction, including the principal direction and the target landmark. During navigation, the Globaler continuously recalls the trajectory history and calibrates current decisions so that local actions remain consistent with the global goal, thereby improving long range reasoning consistency. We perform comprehensive experiments to assess the effectiveness of HiMemVLN in both simulated and real-world environments, validating its performance across diverse settings. In this work, our main contributions are as follows: • We propose HiMemVLN , a hierarchical memory based VLN framework for open source LLMs that improves zero shot navigation reliability under privacy preserving and deployment oriented constraints. • Inspired by cognitive theory, HiMemVLN combines a short term visually dominant Localer for state localization with a long term semantically dominant Globaler for persistent goal grounding, improving long horizon consistency and decision stability. • Across simulated and real environments, HiMemVLN consistently outperforms Open-Nav over diverse scenes, with the best success rate approaching a two fold improvement and a smaller gap to closed-source models. II Related Work II-A Vision-and-Language Navigation Vision-and-Language Navigation (VLN) is a representative task in the field of embodied AI, requiring an agent to follow instructions grounded in visual observations to reach a target location [ 6 ] . Early research primarily adopted a discrete VLN setting based on the MP3D simulator [ 9 ] , in which the agent navigates within predefined graph structures. To bridge the gap between simulated environments and real world applications, Krantz et al [ 17 ] . introduced a benchmark that extends the VLN task from discrete environments to continuous environments (CE), making it more representative of real world navigation scenarios. Early VLN approaches employed LSTM models for cross modal alignment and navigation policy learning [ 2 ,

Read the complete text