LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno · 2026

Read LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation by Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27871.txt.

Introduction

Vision-language models give robots open-vocabulary navigation skills. Their size usually confines them to the cloud. That breaks low-latency control on small robots. Nicolas Baumann and seven colleagues present LocalNav in 2026. Their strategy distils frontier-model reasoning into a lightweight local model. The student is a four-billion-parameter vision-language model running on embedded hardware such as a Jetson Orin.

A frontier pipeline sets the reference first. Its scene-graph design reaches a success rate near forty percent on the HM3D OVON benchmark. Fine-tuning the small model on about five hundred reasoning traces transfers much of that skill. The student reaches a success rate near thirty-five percent. A reinforcement stage then compresses output length. Token overhead falls by about seventy-two percent, and latency by about the same. With quantisation added, total inference latency drops by about eighty-three percent. Performance is largely preserved.

Catalog record

CREATORNicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno Abstract Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav . Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B-parameter local VLM for edge execution on embedded GPU devices (e.g., Jetson Orin ). We first establish a State of the Art (SotA) , Scene Graph (SG) -based pipeline using Claude Sonnet 4.6 , achieving a 39.7% SR on the HM3D OVON benchmark. We then demonstrate that fine-tuning Qwen3.5-4B on just ∼ \sim 500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models. Finally, we introduce E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task. This downstream optimization reduces TG overhead by 72.1% and latency by 71.8%. Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execution on mobile robots. 1 1 footnotetext: Denotes equal contribution. Figure 1: Schematic overview of the proposed LocalNav system. SFT enables ObjectNav capabilities by distilling from large cloud frontier models, achieving a 26%-point increase in SR on a Qwen3.5-4B model. E-RLVR grounds the agent in its task and regularizes TG length, decreasing generation latency by 71.8%, and thus enabling on-board deployment on resource-constrained hardware. Keywords: Object Goal Navigation, Vision-Language Models, E-RLVR 1 Introduction The emergence of Large Language Models (LLMs) and VLMs with open-vocabulary capabilities [ 1 , 27 , 17 , 23 ] is expanding embodied navigation to open-set instruction following [ 54 , 58 , 48 , 33 ] . While closed-set agents have historically been limited to searching for generic categories defined during training (e.g., ”cabinet” ), VLMs provide the capacity for sophisticated scene understanding, which enables nuanced tasks such as locating ”drawers beneath the microwave” . Consequently, the agent reasons about semantically related objects or employs room-level navigation, utilizing its vision modality for visual verification. Formalized in open vocabulary benchmarks [ 49 , 28 ] , this open-set capability has spurred a paradigm shift in how foundation models are integrated into robotic systems. Following this transition, two distinct research trajectories have emerged: Low-Level Action Prediction: Similar to recent advances in generalist manipulation policies [ 29 , 7 , 6 , 35 , 30 , 46 ] , these methods build Vision Language Actions (VLAs) by conditioning pre-trained foundation models to predict low-level action tokens using Behavior Cloning (BC) . While this grants the model direct control over the robot’s low-level movement, it introduces several challenges. The discrete low-level action space \mathcal{A}_{\text{LL}} used in ObjectNav is typically native to the Habitat simulator, and, as such, can constrain the robot’s agency. Secondly, the approach is heavily dependent on extensive SFT data for BC and inherently constrains the system to the specific action spaces seen during training [ 60 ] . Crucially, as these models are trained end-to-end, fundamental robotic abilities such as obstacle avoidance and spatial memory must be implicitly learned [ 54 ] . Furthermore, because this architecture necessitates continuous inference to maintain control loops, typically requiring 5\text{\,}\mathrm{Hz}10\text{\,}\mathrm{Hz} for robot velocity control [ 42 ] , it imposes a significant computational burden, which presents a severe bottleneck for resource-constrained mobile robots [ 57 ] . High-Level Decision-Making: Rather than predicting low-level actions \mathcal{A}_{\text{LL}} , an emerging line of research utilizes VLM foundation models strictly for abstract semantic reasoning over intermediate spatial representations, such as SGs [ 24 , 47 ] . By decoupling high-level reasoning from low-level control, the VLM can focus on its core strengths: high-level scene comprehension and multimodal reasoning [ 2 , 50 , 5 ] . A dedicated low-level PointGoal navigation policy can then pursue this goal [ 40 ] . This modularity enables the system to rely on dedicated navigation policies to safely traverse environments and avoid obstacles [ 42 , 39 , 48 ] . Furthermore, relying on structured representations, particularly SGs , provides the agent with an explicit memory of the traversed environment. Because the VLM is only invoked at sparse decision points, strict latency budgets are relaxed. However, SotA systems in this category typically rely on massive, cloud-bound VLMs for abstract reasoning [ 47 , 24 , 33 ] . Fully local deployment on mobile robot platforms remains a significant challenge due to the computational constraints of robotic hardware. To summarize, low-level VLA -based systems necessitate continuous inference, while high-level decision-making approaches infer sparsely, but are cloud-bound to large frontier VLMs . Hence, this work adopts a deployment-first strategy, prioritizing the realization of ObjectNav on physical robotic hardware without relying on a cloud connection, as in Figure 1 . By aligning with a modular high-level decision-making paradigm, we enable SotA open-vocabulary ObjectNav while analyzing the efficacy of edge-deployable VLMs and Parameter Efficient Fine-Tuning (PEFT) strategies for embedded Graphics Processing Unit (GPU) -based systems operating under latency, memory, and power budgets. Our core contributions are as follows: (I) Local VLM Optimization: We investigate the extent to which locally deployable VLMs can be optimized for ObjectNav , successfully narrowing the reasoning gap to cloud-based models and achieving a benchmark score of 34.5% with a 4B VLM in a SG -based ObjectNav architecture. (II) Distillation Analysis: We present a comprehensive ablation study of leading cloud-based models and evaluate the efficacy of distilling their ObjectNav reasoning capabilities into a Qwen3.5-4B model, deployable on embedded platforms. (III) Embodied Training Strategies: We explore the impact of E-RLVR on low-parameter models, ablating the effects of holistic embodied Reinforcement Learning (RL) with standard SFT approaches and yielding a training technique that allows the VLM to retain its ObjectNav performance while reducing output TG by 72.8%, resulting in a 71.8% latency reduction. (IV) Performant System Architecture: We design and implement a SotA SG -based navigation architecture, achieving a 39.7% SR on HM3D OVON , built upon the open-source Hydra framework [ 26 ] . The system is modular and allows for drop-in replacements of different VLMs supporting either cloud-based models or VLMs deployed directly on the Jetson Orin AGX for cloud-independent execution. 2 Related Work Object Goal Navigation. Early approaches to ObjectNav generally fall into two categories: end-to-end learning or modular systems. End-to-end methods typically train policies using RL in simulated environments to develop exploration behaviors [ 40 , 45 , 18 ] . To then reduce dependency on training in simulation, modular pipelines construct intermediate representations, such as top-down semantic maps [ 12 , 41 , 38 , 51 ] , explicit exploration frontiers [ 34 ] , or SGs [ 56 , 44 ] , to guide the navigation policy. However, a fundamental limitation across these methods is their reliance on a closed set of object categories

Read the complete text