NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter and Jonas Frey · 2025
Read NaviTrace: Evaluating Embodied Navigation of Vision-Language Models by Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter and Jonas Frey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26909.txt.
Introduction
Tim Windecker, with seven co-authors, introduces NaviTrace, a benchmark for how well vision-language models can guide navigation. Robots need models that can turn an instruction into a path, but testing that in the real world is costly and simulations are often too simple. In NaviTrace, a model receives an instruction and an embodiment type, such as human, legged robot, wheeled robot, or bicycle, and must draw a two-dimensional navigation trace in image space.
Windecker and the team evaluate eight leading models across one thousand scenarios with over three thousand expert traces. Their semantic-aware score blends path warping distance, goal endpoint error, and per-pixel semantic penalties, and it correlates with human preferences. All models lag humans, mainly through poor spatial grounding and goal localisation. The benchmark and leaderboard are public.
Catalog record
| CREATOR | Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter and Jonas Frey |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NaviTrace: Evaluating Embodied Navigation of Vision-Language Models Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter and Jonas Frey Abstract Vision–language models demonstrate unprecedented performance and generalization across a wide range of tasks and scenarios. Integrating these foundation models into robotic navigation systems opens pathways toward building general-purpose robots. Yet, evaluating these models’ navigation capabilities remains constrained by costly real-world trials, overly simplified simulations, and limited benchmarks. We introduce NaviTrace, a high-quality Visual Question Answering benchmark where a model receives an instruction and embodiment type (human, legged robot, wheeled robot, bicycle) and must output a 2D navigation trace in image space. Across 1000 scenarios and more than 3000 expert traces, we systematically evaluate eight state-of-the-art VLMs using a newly introduced semantic-aware trace score. This metric combines Dynamic Time Warping distance, goal endpoint error, and embodiment-conditioned penalties derived from per-pixel semantics and correlates with human preferences. Our evaluation reveals consistent gap to human performance caused by poor spatial grounding and goal localization. NaviTrace establishes a scalable and reproducible benchmark for real-world robotic navigation. The benchmark and leaderboard can be found at https://leggedrobotics.github.io/navitrace_webpage/ . I INTRODUCTION The rise of foundation models with general-purpose capabilities has sparked a push to develop robots that are equally general-purpose—capable of flexible, wide-ranging behavior in real-world environments. Given their significant potential, it is crucial to rigorously assess how these models perform in real-world robotic applications. Such evaluations inform model development, reveal limitations, and establish benchmarks for comparing approaches. However, assessing the navigation capabilities of these models remains challenging. Navigation is critical for numerous robotic applications, including last-mile delivery, industrial inspection, search and rescue, and assistive tools for visually impaired people. Yet only limited evaluation methods currently exist for these capabilities. Applying existing Vision Language Models (VLMs) to the navigation task itself can be straightforward: a text instruction and an image observation can be provided as input, prompting the language model to generate a text description that the robot’s control system can translate into motor commands [ 1 , 2 ] . While various flavors of such systems exist, it remains unclear which VLM system performs best. To evaluate this, there exist three main directions in the literature: The first is to perform real-world closed-loop rollouts on a set of navigation tasks and measure the success rate. However, such experiments are expensive, time-consuming, and inherently do not scale well for evaluating performance across diverse operating environments, while also lacking reproducibility. The second is to run the same closed-loop experiments in simulation. This approach improves reproducibility but still faces significant limitations. The diversity of tasks is constrained by the scenarios created in simulation, which are inherently simplified in terms of dynamics, mostly feature static scenes, and have limited semantics. Important factors, such as varying terrain properties or social norms, that should influence an agent’s navigation behavior, are difficult to encode. Lastly, Visual Question Answering (VQA) benchmarks can overcome some of these limitations by making use of high-quality human annotations, which can incorporate semantics, social preferences, and geometric cues. While several navigation-focused VQA datasets exist, they typically (i) constrain outputs to text answers rather than trace-level plans, and (ii) evaluate only legged or wheeled robot embodiments [ 3 , 4 ] . To fill this gap, we introduce NaviTrace, a VQA benchmark specially designed to evaluate embodiment-specific navigation performance across 1,000 diverse scenarios and four different embodiments. Each task within NaviTrace consists of a single real-world image paired with a high-quality language instruction, enabling efficient data collection while capturing challenging navigation tasks. Following the most intuitive approach to answering navigation questions, we provide solutions per embodiment as 2D paths in image space, which we refer to as traces. This carefully chosen formulation is more expressive than low-level commands such as “Forward” [ 1 ] and can also support longer-horizon planning. It can be seen as an extension of pointing—a common task that is evaluated and optimized in current foundation models [ 5 , 6 ] and widely used to assess the visual grounding of VLMs [ 7 ] . Furthermore, traces have proven beneficial for addressing manipulation tasks [ 8 , 9 , 10 , 11 ] . NaviTrace tests VLMs for instruction following, spatial understanding, and physical understanding of varying embodiments (human, legged robot, wheeled robot, and bicycle), and categorizes scenarios based on the type of navigation challenges. We develop a semantic-aware score to measure how well the predicted navigation trace aligns with human preferences. To achieve this, we combine the Dynamic Time Warping distance to a ground-truth trace, goal endpoint error, and pixelwise embodiment-conditioned penalties derived from a semantic segmentation model. We show that our metric, while inexpensive to compute and annotate, is competitive with more expensive human-derived metrics in aligning with human preferences. Specifically, our main contributions are: 1. NaviTrace: A novel high-quality benchmark for evaluating the ability of VLMs to predict how different embodiments navigate in 1000 diverse and challenging real-world scenarios. 2. Semantic-aware Score : A new metric to measure the accuracy of 2D traces for real-world images. We test the score for alignment with human preferences by showcasing its correlation to human expert judgments. 3. Evaluation of VLMs: Comprehensive assessment of current state-of-the-art VLMs on our benchmark. II RELATED WORK Table I provides an overview of benchmarks evaluating the navigation performance of vision-based agents and VLMs. TABLE I: An overview of vision language navigation and VLM datasets. Task Data Source Annotations Name Type VLM Nav. Description Sim Real Description Automatic Manual Scoring Embodiments R2R [ 12 ] VLN \boldsymbol{\times} \checkmark MP3D [ 13 ] 21,567 language navigation instructions / Success Rate, Navigation Error REVERIE [ 14 ] VLN \boldsymbol{\times} \checkmark MP3D [ 13 ] 21,702 language instructions that require navigating and identifying an object / Success Rate, SPL RxR [ 15 ] VLN \boldsymbol{\times} \checkmark MP3D [ 13 ] 126k multilingual time-aligned language instructions, 126k demonstration paths / Success Rate, SPL, Navigation Error, Normalized Dynamic Time Warping OctoNav-Bench [ 16 ] Embodied Navigation \boldsymbol{\times} \checkmark Habitat (MP3D, HM3D, Gibson, ProcTHOR) 45k+ annotated instructions with trajectories that combine the task types: object goal, point goal, image goal, instance-image goal, and VLN, 10k+ instruction-think-action pairs Success Rate, SPL EgoWalk [ 17 ] VLN \boldsymbol{\times} \checkmark 50\text{\,}\mathrm{h} of egocentric navigation recordings Automatic traversability region and language goals with extracted odometry trajectories MSE, Absolute Displacement Error, and Final Displacement Error CityWalker [ 18 ] Point Goal Nav. \boldsymbol{\times} \checkmark 2000\text{\,}\mathrm{h} of city walking videos from the internet With visual odometry extracted trajectory poses Average Orientation Error SocialNav-SUB [ 3 ] Social Navigation VQA \checkmark \boldsymbol{\times} SCAND [ 19 ] 4968 unique questions, 24840 human responses / Probability of Agreement, Consensus-Weighted Probability of Agreement , Social-LLaVA [ 4 ] Social Navigation VQA \checkmark \boldsymbol{\times} SCAND [ 19 ] 40k questions fully annotated by humans Human judges , NaviTrace (ours) Nav. Traces for VLMs \checkmark \checkmark 1000 diverse real images 1k language instructions, 3k+ traces that describe 2D paths Human-preference aligned nav. metric , , , Vision Language Navigation Benchmarks. Several relevant works focus on the evaluation of vision-language navigation (VLN) tasks, where agents follow natural language instructions using visual input. Well-established benchmarks include Room-to-Room (R2R) [ 12 ] , REVERIE [ 14 ] ,