EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
Zebin Yang, Sunjian Zheng, Tong Xie, Tianshi Xu, Fan Wang, Jie Tang, Shaoshan Liu, Meng Li · 2025
Read EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval by Zebin Yang, Sunjian Zheng, Tong Xie, Tianshi Xu, Fan Wang, Jie Tang, Shaoshan Liu, Meng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18546.txt.
Introduction
Zebin Yang and his eight co-authors move a robot task onto the device itself. The task is object-goal navigation. An agent must find a named item in a new space. Cloud agents read online maps well. Small local models fail far more on the same maps. Long map text also slows local planning. Their system, EfficientNav, attacks both flaws at once.
A memory step prunes repeated detail from the map. It keeps the parts with real scene meaning. A small model can then grasp the scene. A cache scheme saves past computation for reuse. It groups stored attention data to cut repeated work. Tests used a standard indoor benchmark. EfficientNav beat cloud-based rivals on success rate. It also answered several times faster. Capable home robots, the authors argue, need no remote servers.
Catalog record
| CREATOR | Zebin Yang, Sunjian Zheng, Tong Xie, Tianshi Xu, Fan Wang, Jie Tang, Shaoshan Liu, Meng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval Zebin Yang, Sunjian Zheng, Tong Xie, Tianshi Xu, Fan Wang, Jie Tang, Shaoshan Liu, Meng Li Abstract Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices. At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices. In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps. To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache. Extensive experimental results demonstrate that EfficientNav achieves 11.1% improvement in success rate on HM3D benchmark over GPT-4-based baselines, and demonstrates 6.7 × \times real-time latency reduction and 4.7 × \times end-to-end latency reduction over GPT-4 planner. Our code will be released soon. 1 Introduction Aiming to navigate an agent to an object specified by its category in an unknown environment, object-goal navigation (ObjNav) presents a crucial yet challenging task for embodied agents an2024etpnav ; sridhar2024nomad ; manashty2010scenario ; kernbach2011development . To enhance the generality of ObjNav, recent studies have proposed leveraging the common-sense reasoning capabilities of large language models (LLMs) to achieve long-term planning in a zero-shot manner long2024instructnav ; anwar2024remembr ; shah2023navigation ; kaichen2024towards ; wang2024videoagent . Specifically, LLMs act as planners and determine a series of navigation sub-goals to guide the agent to the final destination dorbala2022clip . In such systems, navigation goals, maps of explored areas, and current observations are typically provided as context for the LLMs chen2024mapgpt ; zhou2024navgpt ; cao2024cognav . This information can be conceived as memory for LLMs, providing information of explored areas and visited objects to facilitate planning and decision-making. Figure 1: (a) Local devices (e.g., NVIDIA Jetson AGX Orin) exhibit constrained memory capacity (32GB). (b) Only smaller LLMs, such as Llama-3.2-11b, can be used as the planner due to memory limitations. (c) The KV cache of map information accumulates with navigation steps, exceeding the memory capacity. (d) Recomputing the KV cache incurs long real-time latency. Though promising, LLM-based ObjNav systems face significant challenges due to the substantial memory requirements and computational complexity of LLMs. To ensure task success, giant LLMs, such as GPT-4, are typically employed, necessitating offloading these models to the cloud wake2023gpt ; cai2024bridging ; zhou2024navgpt ; yang2024mcubert ; wei2025lightmamba . Such computing systems not only suffer from high communication latency, which degrades real-time performance khatib1986real ; shah2023energy ; suleiman2019navion ; mei2006deployment ; murray2016microarchitecture ; liu2019eslam ; zhong2024propd , but also raise privacy concerns hou2023ciphergpt ; lin2024fastquery ; ren2023cham ; zeng2023copriv ; hao2022iron and incur extremely high computational costs bahrini2023chatgpt ; wu2024unveiling . Hence, a computing paradigm shift from offloading LLMs to the cloud towards accommodating LLMs on local servers or embedded devices, i.e., on-device computing, is required for ObjNav systems karumbunathan2022nvidia ; talpes2022dojo . However, enabling on-device ObjNav presents two primary design challenges, largely constrained by limited memory resources. First, only smaller LLMs can be deployed on local devices, thus causing planning performance drops. For example, as shown in Figure 1 (a) and (b), the NVIDIA Jetson AGX Orin karumbunathan2022nvidia only has 32GB of DRAM, which can only accommodate smaller LLMs such as LLaMA-3.2-11b. Since larger LLMs typically have higher model capacity, enabling them to handle more complex and longer navigation tasks, on-device LLMs struggle to achieve the same level of performance as their cloud-based counterparts cao2024cognav ; long2024instructnav . Second, as context information accumulates during the navigation process, its corresponding KV-cache also grows and can eventually exceed the memory constraint (in Figure 1 (c), we set 30% of the Orin memory as the KV cache memory constraint, as the model weights also need to be stored in memory). And re-computing the KV-cache significantly slows down LLM decoding, introducing substantial compute latency (Figure 1 (d)). In this paper, we propose EfficientNav, an on-device memory-augmented ObjNav system that enables efficient zero-shot in-door navigation. We observe that describing the navigation map can lead to long prompt lengths, and the corresponding KV-cache often cannot be fully stored due to memory constraints. Consequently, we select only a portion of the map description and load its KV cache for the LLM. However, the description selection will change the context order and the prefix of the corresponding KV cache. To enable reusing the retrieved KV cache despite the changing prefix, we propose discrete memory caching. This involves clustering the map information into groups and calculating the KV-cache for each group independently. Then, KV-caches within the same group are subject to cross-attention. To mitigate the impact of ignoring cross-attention between groups, we propose attention-based memory clustering. This technique uses LLM attention mechanisms to cluster related information into the same group. Furthermore, we observe that smaller LLMs may struggle to fully understand complex navigation maps. To address this, we propose semantics-aware memory retrieval to prune redundant map information during the group selection process, thereby improving the performance of smaller LLMs. We summarize our contributions as follows: 1 We propose discrete memory caching to prevent saving the KV cache of the whole map description and meet the memory constraints. 2 We propose attention-based memory clustering to reduce the impact of ignoring cross-attention between groups. 3 We propose semantics-aware memory retrieval to efficiently select the relevant groups and prune redundant map information. 4 We conduct extensive experiments and show that EfficientNav can achieves 11.1% success rate improvements over GPT-4-based methods on HM3D dataset, and show 6.7 \times real-time latency and 4.7 \times end-to-end latency reduction over GPT-4 planner. 2 Related Work LLM-based object goal navigation. According to the data processing pipeline, LLM-based ObjNav can be categorized into two paradigms: end-to-end and modular approaches. The end-to-end method, such as NaVid zhang2024navid , directly converts RGB-D inputs into robot control policies, but incurs significant training costs brohan2023rt ; kim2024openvla . The modular approach, which is more prevalent due to simplicity, involves using LLMs as a high-level planner, and a lower-level controller handled by the robot itself. long2024instructnav ; an2024etpnav ; shah2023navigation ; cai2024bridging ; kaichen2024towards . The LLM planner generates sub-goals to reach the final goal conditioned on the environment and instructions. The controller shah2023vint ; sridhar2024nomad , usually a small neural network, finds a trajectory from the current place to the sub-goal and controls the robot’s motion in a real-time manner. We focus on optimizing the LLM planner, since it is the primary efficiency bottleneck due to its long inference latency and high memory cost yin2024sg ; chen2024mapgpt . Figure 2: Software flow of LLM-based ObjNav. Memory Augmentation for object goal navigation. In zero-shot ObjNav tasks, the robot faces inefficiencies during sub-goal planning due to its limited field of view cai2024bridging .