Can Large Vision Language Models Read Maps like a Human?

Shuo Xing, Shuangyu Xie, Zezhou Sun, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, Zhengzhong Tu · 2025

Read Can Large Vision Language Models Read Maps like a Human? by Shuo Xing, Shuangyu Xie, Zezhou Sun, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, Zhengzhong Tu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14607.txt.

Introduction

Nine researchers ask whether today's big vision-language models can read a map the way a person does. They built MapBench, a set of more than 1,600 path-finding problems drawn from one hundred real, cluttered maps. Given a map image and two named landmarks, a model must produce directions in words. A scene-graph index lets the team convert and score the answers fairly.

Both open and closed leading models struggled, with or without a chain-of-thought method that breaks navigation into steps. Failures exposed weak spatial reasoning and shaky step-by-step decision making rather than simple misreading. The authors release the data and code openly. They present the benchmark as a measuring stick for a skill that assistants will need if they are ever to guide people through real places from a picture of a map.

Catalog record

CREATORShuo Xing, Shuangyu Xie, Zezhou Sun, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, Zhengzhong Tu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Large Vision Language Models Read Maps like a Human? Shuo Xing, Shuangyu Xie, Zezhou Sun, Kaiyuan Chen, Yanjia Huang, Yuping Wang, Jiachen Li, Dezhen Song, Zhengzhong Tu Abstract In this paper, we introduce MapBench —the first dataset specifically designed for human-readable, pixel-based map-based outdoor navigation, curated from complex path finding scenarios. MapBench comprises over 1600 pixel space map path finding problems from 100 diverse maps. In MapBench, LVLMs generate language-based navigation instructions given a map image and a query with beginning and end landmarks. For each map, MapBench provides Map Space Scene Graph (MSSG) as an indexing data structure to convert between natural language and evaluate LVLM-generated results. We demonstrate that MapBench significantly challenges state-of-the-art LVLMs both zero-shot prompting and a Chain-of-Thought (CoT) augmented reasoning framework that decomposes map navigation into sequential cognitive processes. Our evaluation of both open-source and closed-source LVLMs underscores the substantial difficulty posed by MapBench, revealing critical limitations in their spatial reasoning and structured decision-making capabilities. We release all the code and dataset in https://github.com/taco-group/MapBench . \captionof figure MapBench is a dataset of over 1600 map space path-finding problems from 100 diverse map images. MapBench evaluates language-based navigation instructions generated by Large Vision-Language Models (LVLMs) with map images with cluttered and potentially occluded visual symbols. 1 Introduction “Not all those who wander are lost.” – J. R. R. Tolkien Maps have guided travellers since ancient times. From traditional nautical charts to modern digital navigation systems, our capacity to transform two-dimensional pixel representations into actionable navigation instructions remains a distinctly human skill. The recent emergence of Large Vision-Language Models (LVLMs) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] reshapes the boundaries of human and machines in visual perception [ 10 , 11 , 12 ] and language understanding [ 13 , 14 ] across various domains, including biomedical imaging [ 15 , 16 , 17 ] , autonomous vehicles [ 18 , 19 , 20 , 21 , 22 ] , robotics [ 23 , 24 , 25 , 26 ] , and sciences [] In this paper, we explore whether LVLMs can replicate the human cognitive ability of map-space path finding : given a human-readable map image with landmarks and pathways, and a query with beginning and end landmarks, LVLM generates language-based navigation instructions. Effective map-space path finding by LVLMs necessitates three fundamental capabilities: (a) perception that recognizes visual symbols such as colors, texts, areas, and icons in artistic or stylized forms; (b) spatial understanding that contextualizes the symbols to physical environment, addressing orientations, viewpoint, occlusion handling, and scaling; (c) planning that routes between endpoints through landmarks and intersections. While research has advanced each individual component independently [ 27 , 28 , 29 , 30 , 31 ] , the integration of these capabilities must simultaneously interpret visual symbolic representations and extract spatial relationships to enable coherent path finding. Existing LVLMs exhibit limitations in cross-modal misalignment [ 32 , 33 , 34 , 19 ] and understanding complex geometric relationships within visual inputs [ 35 ] . As a result, it is critical to investigate LVLM capabilities specifically as a unified context of map-space path finding, effectively bridging visual perception, spatial reasoning and route planning. We study the map-space path finding task for VLM with Map Space Scene Graph (MSSG), a structured representation that captures visual symbolic-spatial-topological relationship of human-readable maps. Based on MSSG, we introduce MapBench, the first comprehensive dataset specifically designed to evaluate LVLMs on human-readable maps. MapBench encompasses over 1,600 visual path planning queries derived from 100 distinct maps, categorized across 4 visual style elements (landmarks, traversable areas, projection methods, and annotation formats) and 9 scenarios (urban, university, theme park, etc.). Each map includes manually annotated MSSG along with associated queries of beginning and end landmarks. We evaluate multiple state-of-the-art LVLMs on MapBench, including Llama-3.2-11B-Vision-Instruct [ 5 ] , Qwen-2-VL-7B [ 7 ] , GPT-4o [ 36 ] , and GPT-4o mini [ 37 ] under both zero-shot prompting and chain-of-thought reasoning protocols. To further investigate the performance in map-space path finding, we propose a Chain-of-Thought (CoT) augmented reasoning framework that aligns with the proposed MSSG by decomposing map navigation into sequential cognitive processes, including (1) localization of start and destination landmarks; (2) description of surrounding context; (3) path connection through identifying intermediate landmarks to enable LVLMs to perform explicit spatial reasoning. Our results indicate that MapBench poses significant challenges for state-of-the-art LVLMs. Additionally, closed-ended LVLMs outperform their open-sourced LVLMs. The proposed CoT reasoning framework generally delivers better performance compared to zero-shot prompting, though it occasionally introduces redundant information. In summary, our contribution is as follows: • Map Space Scene Graph (MSSG), a visual symbolic-spatial-topological representation of maps, • MSSG-based LVLMs evaluation metrics and conversion algorithms between MSSG and natural language, • MapBench, a human readable map benchmark with 1600 human-annotated map queries • A CoT-augmented reasoning framework that decomposes map navigation into sequential prompting, • Evaluation results of LVLMs on MapBench with both zero-shot and Chain-of-Thought augmented prompting. 2 Related Works Vision Language Model Reasoning and Planning Chain-of-Thought (CoT), whereby the simple inclusion of accurate deduction steps for few-shot examples within the original prompt empowers LLMs to achieve substantial performance improvements in reasoning tasks [ 38 ] , pioneers the revolution reasoning and planning tasks capabilities of large language models and vision language models by prompting based strategies [ 39 , 40 , 41 , 42 , 43 , 44 , 45 ] . Moreover, integrating external tools, knowledge bases, and iterative self-reflection enhances the robustness and adaptability of VLMs, driving them toward more sophisticated, human-like visual intelligence in complex real-world tasks [ 22 , 46 , 47 , 23 , 25 , 48 ] . Visual Language Navigation Visual Language Navigation (VLN) tasks require agents to interpret natural language instructions to navigate through photorealistic or real-world environments. Some previous work has mad significant progress in field. For example, leveraging large language models (LLMs) like NavGPT enhances instruction understanding and reasoning capabilities, enabling zero-shot sequential action predictions by decomposing instructions into sub-goals and incorporating commonsense knowledge [ 49 ] [ 50 ] [ 51 ] . Additionally, frameworks such as DAVIS utilize visual consistency to anticipate discrepancies in unseen environments by combining semi-supervised learning with reinforcement learning and integrating online visual-language mapping techniques, allowing agents to parse natural language instructions into executable plans aligned with real-time visual inputs, thereby facilitating better adaptation during performance [ 52 ] [ 53 ] [ 54 ] [ 55 ] . Despite these advancements, even with the use of frontier-based exploration methods in language-driven zero-shot object navigation [ 56 ] [ 57 ] , existing approaches still heavily rely on detailed visual inputs, including depth and semantic information, and struggle to generalize to unseen environments due to discrepancies between training and testing conditions. Moreover, they seldom utilize high-level, human-readable maps that humans commonly rely on for navigation. VQA reasoning and planning benchmarks Visual Question Answering (VQA) has been a widely studied task in the field of computer vision and natural language processing, with the aim of enabling models to answer questions about images. Prior VQA benchmark extends from CV dataset focusing on evaluating simple

Read the complete text