City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs

Dwip Dalal, Utkarsh Mishra, Narendra Ahuja, Nebojsa Jojic · 2025

Read City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs by Dwip Dalal, Utkarsh Mishra, Narendra Ahuja, Nebojsa Jojic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15933.txt.

Introduction

Dwip Dalal and co authors test whether multimodal language models can navigate real cities from visual input in a 2025 paper. Current benchmarks for embodied agents lean on language tasks or simulators, missing the knowledge heavy reasoning of real navigation. Their new task, sparsely grounded visual navigation, targets that gap.

The CityNav benchmark spans four global cities and over 50 decision points per route, with no annotations or special architecture allowed. Agents must localize from city cues and landmarks, reason spatially, and plan routes on their own. Leading models and reasoning techniques underperform badly. The authors' Verbalization of Path method, which probes the model's cognitive map of landmarks and directions, substantially improves success.

Catalog record

CREATORDwip Dalal, Utkarsh Mishra, Narendra Ahuja, Nebojsa Jojic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs Dwip Dalal, Utkarsh Mishra, Narendra Ahuja, Nebojsa Jojic Abstract Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks. However, current evaluation benchmarks remain predominantly language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. To bridge this critical gap, we introduce the task of Sparsely Grounded Visual Navigation , explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environment. We operationalize this task with CityNav, a comprehensive benchmark encompassing four diverse global cities, specifically constructed to assess raw MLLM-driven agents in city navigation. Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50 + 50+ decision points without additional environmental annotations or specialized architectural modifications. Crucially, agents must autonomously achieve localization through interpreting city-specific cues and recognizing landmarks, perform spatial reasoning, and strategically plan and execute routes to their destinations. Through extensive evaluations, we demonstrate that current state-of-the-art MLLMs and standard reasoning techniques (e.g., Chain-of-Thought, Reflection) significantly underperform in this challenging setting. To address this, we propose Verbalization of Path (VoP) , which explicitly grounds the agent’s internal reasoning by probing an explicit cognitive map (key landmarks and directions toward the destination) from the MLLM, substantially enhancing navigation success. Project Webpage: https://dwipddalal.github.io/AgentNav/ Figure 1 : The figure illustrates our proposed Verbalization of Path (VoP) method, which extracts city-scale cognitive maps from MLLMs for city navigation in the wild. The red bounding boxes on the New York map highlight the streets and locations explicitly referenced by the MLLM during its verbal reasoning. Note that, in New York City, street names typically correspond to entire street segments. 2 2 footnotetext: Work performed during a research internship at Microsoft Research, Redmond. 1 Introduction Pretraining MLLMs on web-scale, interleaved image–text corpora induces broad, transferable world knowledge, enabling robust zero-shot and few-shot generalization across diverse vision–language tasks 4 ; 20 ; 35 ; 13 ; 8 thereby facilitating the development of sophisticated agents 55 ; 19 ; 15 . This emergent behavior has enabled foundation models to exhibit robust general reasoning and instruction-following abilities 7 ; 2 ; 36 ; 37 ; 50 ; 58 . Furthermore, their multimodal successors 34 ; 38 ; 28 ; 51 ; 11 , enhanced by recent scaling methods 10 , integrate high-resolution visual processing, extended context handling, and precise OCR and grounding capabilities. Such advancements facilitate elegant perception-to-action pipelines, enabling researchers to develop embodied agents capable of complex perception, planning, and action tasks 21 ; 22 ; 3 ; 46 ; 67 ; 27 ; 63 . While existing works have demonstrated reasoning capabilities of embodied agents in indoor navigation 67 ; 27 ; 63 , code generation and planning 46 ; 26 , and robotic arm manipulation 22 ; 3 ; 45 , these evaluations predominantly occur within simulated environments and are not knowledge intensive tasks. Therefore do not rigorously assess the agents’ ability to leverage their extensive internal knowledge repositories to execute sequential decisions in dynamic, real-world scenarios. Whereas, outdoor navigation is inherently a knowledge-intensive task demanding extensive cognitive capabilities, such as comprehensive environmental knowledge, sequential decision-making, spatial reasoning, and robust visual grounding using recognizable landmarks 32 . Prior studies on outdoor navigation have typically supplied agents with explicit landmark information embedded within images 9 ; 64 ; 41 , significantly alleviating cognitive load by eliminating the necessity for agents to internally retrieve knowledge for self-localization and planning. In contrast, our work introduces a novel task termed Sparsely Grounded Long-Range Navigation where: Agent must navigate without any landmark annotations or explicit city navigation instructions, relying exclusively on images observed at each intersection. This task requires agents to leverage their intrinsic world knowledge to facilitate spatial understanding, accurate self-positioning, and sequential decision-making to reach the goal. Existing datasets 32 ; 9 exhibit several limitations: 1) they contain relatively short path lengths, 2) they are restricted to one-two cities, 3) many of them are closed source (not available even on request) 4) given their widespread usage over the years, there is a significant likelihood that MLLMs have been exposed to these datasets during training. To address these limitations, we introduce a novel, diverse dataset comprising paths of length greater than 2 Km, and include over 50+ decision points, spanning four distinct cities, thereby testing varied capabilities of MLLMs and significantly enhancing the task complexity. Our dataset is constructed using Google Street View panograph 5 . Alongside the dataset, we provide a robust evaluation platform capable of deploying MLLMs directly onto the Google Street View navigation graph. The platform is explicitly designed to handle practical navigation challenges such as dead ends, missing street connections, and abrupt transitions inherent to Google Street View. We further introduce Verbalization of Path (VoP) , a mechanism designed to explicitly extract and leverage the latent world knowledge internalized by MLLMs. By prompting agents to verbalize navigation paths, as illustrated in Fig. 1 , VoP substantially enhances the performance of MLLM-based agents on long-range navigation tasks. Since navigating unstructured environments is widely regarded as a fundamental hallmark of intelligence 32 , our results highlight the effectiveness of VoP in bridging the gap between static reasoning capabilities and dynamic, real-world sequential decision-making. Our main contributions are 1) We introduce a new task and dataset designed to MLLMs on long-range sequential decision-making that requires leveraging their internal world knowledge. 2) We propose a zero-shot framework Verbalization of Path (VoP) to elicit and utilize the internal world knowledge of MLLMs for effective outdoor navigation. 3) We show that MLLMs can successfully navigate complex urban environments such as New York City, indicating that these models possess extensive structured world knowledge capable of supporting real-world spatial reasoning. 4) We demonstrate that structured reasoning techniques (such as CoT, reflection) effective in static reasoning contexts fail when MLLMs must coax and deploy their internal knowledge for dynamic, sequential decision-making. 2 Related Works Work Bucket World Scope Dist. Horizon Visual obs. Guidance Landmark Map/GPS LLM role EmbodiedBench Embodied-bench Sim Multi-env – Varied RGB Instr – – Raw Embodied Agent Interface Embodied-bench Sim Multi-env – Varied State LTL – – LLM NavGPT Indoor VLN Sim MP3D Room-scale Short Text-desc NL Implicit None LLM NavCoT Indoor VLN Sim MP3D Room-scale Short RGBD NL Implicit None Fine-tune Touchdown Outdoor VLN Real Manhattan \leq 350 m † Short Panorama NL Text None Dataset VELMA Outdoor VLN Real Manhattan \leq 350 m † Short Panorama NL CLIP None LLM Loc4Plan Outdoor VLN Real TD/M2S \leq 350 m † Short Panorama NL Locate None VLM VLN-Video Outdoor VLN Real Multi-city – – Video NL Implicit – Pretrain FLAME Outdoor VLN Real TD/M2S \leq 350 m † Short Panorama NL Implicit None Fine-tune Perceive-Reflect-Plan Outdoor-goal Real City Long-range Long Scene Goal-desc Landmarks None Fine-tune CityNav (ours) Outdoor-goal Real 4 cities \sim 2 km 50+ Intersection-only Goal None None Raw Table 1 : Dataset-level comparison emphasizing how CityNav differs from prior navigation/embodied benchmarks. Dist. indicates typical route-length regime. \dagger denotes works characterized as operating on short outdoor trajectories (typically <350 m), contrasted with CityNav’s long-range setting ( \sim 2

Read the complete text