From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models Conference: ; ;

Lingyao Li, Runlong Yu, Qikai Hu, Bowei Li, Min Deng, Yang Zhou, Xiaowei Jia · 2025

Read From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models Conference: ; ; by Lingyao Li, Runlong Yu, Qikai Hu, Bowei Li, Min Deng, Yang Zhou, Xiaowei Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01608.txt.

Introduction

Lingyao Li and co-authors built in 2025 a benchmark for testing whether language models can locate a photo. Guessing where an image was taken matters for crisis response, forensics, and location intelligence. Their benchmark, IMAGEO-Bench, scores accuracy, distance error, bias, and the reasoning path. It draws on global street scenes, United States landmarks, and a private set of unseen images.

Ten leading models, open and closed, were put through the tests. Closed models generally reasoned better about place. A clear bias appeared: models did well in rich regions like North America and Western Europe and poorly elsewhere. Success hinged on spotting urban, outdoor, street-level scenes with clear landmarks. The authors offer the benchmark as a hard lens on spatial reasoning. Builders of location-aware systems are the intended readers.

Catalog record

CREATORLingyao Li, Runlong Yu, Qikai Hu, Bowei Li, Min Deng, Yang Zhou, Xiaowei Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models Conference: ; ; Lingyao Li, Runlong Yu, Qikai Hu, Bowei Li, Min Deng, Yang Zhou, Xiaowei Jia Abstract . Image geolocalization, the task of identifying the geographic location depicted in an image, is important for applications in crisis response, digital forensics, and location-based intelligence. While recent advances in large language models (LLMs) offer new opportunities for visual reasoning, their ability to perform image geolocalization remains underexplored. In this study, we introduce a benchmark called IMAGEO-Bench that systematically evaluates accuracy, distance error, geospatial bias, and reasoning process. Our benchmark includes three diverse datasets covering global street scenes, points of interest (POIs) in the United States, and a private collection of unseen images. Through experiments on 10 state-of-the-art LLMs, including both open- and closed-source models, we reveal clear performance disparities, with closed-source models generally showing stronger reasoning. Importantly, we uncover geospatial biases as LLMs tend to perform better in high-resource regions (e.g., North America, Western Europe, and California) while exhibiting degraded performance in underrepresented areas. Regression diagnostics demonstrate that successful geolocalization is primarily dependent on recognizing urban settings, outdoor environments, street-level imagery, and identifiable landmarks. Overall, IMAGEO-Bench provides a rigorous lens into the spatial reasoning capabilities of LLMs and offers implications for building geolocation-aware AI systems. Datasets: https://doi.org/10.5281/zenodo.16670471 Code: https://github.com/Lingyao1219/benchmark-reverse-geocoding 1. Introduction Image geolocalization, the process of inferring geographic coordinates or addresses from visuals, presents a complex challenge at the intersection of computer vision and geographic information systems (GIS) ( Lin et al. 2013 ; Workman et al. 2015 ; Jin Kim et al. 2017 ; Zamir and Shah 2014 ) . Image geolocalization requires holistic scene interpretation and reasoning, encompassing both explicit indicators (e.g., landmarks, signage) and contextual cues (e.g., architectural styles, vegetation, spatial layout) to infer a precise location ( Wu and Huang 2022 ; Lin et al. 2013 ) . The accurate identification of location from images has important implications for real-world scenarios, including digital forensics ( Kadha et al. 2025 ) , urban analytics ( Lin et al. 2013 ) and crisis management ( Li et al. 2025a ; Yin et al. 2025 ) . Conventionally, this task has been investigated with computer vision models, using either supervised classification ( Pramanick et al. 2022 ; Seo et al. 2018 ) or retrieval-based methods ( Vivanco Cepeda et al. 2023 ; Zhu et al. 2022 ; Vo and Jacobs 2017 ; Shatwell et al. 2025 ) . These models often leverage metadata embedded in image files ( Samano et al. 2020 ) or rely on extensive GIS databases ( Hays and Efros 2014 ; Brejcha and Čadík 2017 ) to recognize landmark or visual cues. However, early methods often fail to generalize to previously unseen environments and perform inadequately in the absence of distinctive landmarks. Their effectiveness is further constrained by data coverage and resolution, especially when the metadata is missing. Recent advances in large language models (LLMs), such as GPT and Gemini, have transformed the landscape of image understanding and spatial reasoning ( Chen et al. 2024 ; Wu et al. 2024 ; Li et al. 2025b ; Tian et al. 2025 ) . These models can describe intricate details, understand spatial relationships between objects, and infer context from subtle visual cues within an image ( Chen et al. 2024 ; Yang et al. 2025 ) . By leveraging extensive world knowledge and visual-textual alignment, LLMs offer a promising avenue for overcoming the limits of conventional geolocalization, even when confronted with out-of-distribution data ( Zhou et al. 2024 ; Li et al. 2025c ; Xu et al. 2024 ) . Recent research further demonstrates that targeted strategies, such as supervised fine-tuning ( Yi and Shan 2025 ) and retrieval-augmented inference ( Li et al. 2025c ) , can substantially enhance LLMs’ geolocalization capabilities. In parallel with LLM advances, there has been rapid progress in developing benchmarks to evaluate their visual and linguistic capabilities ( Lacoste et al. 2023 ; Hua et al. 2024 ; Fan et al. 2024 ) . A representative example is GEO-Bench, which evaluates LLMs on classification and segmentation tasks in geoscience ( Lacoste et al. 2023 ) . For image geolocalization, benchmarks like Img2Loc ( Zhou et al. 2024 ) and LLMGeo ( Wang et al. 2024 ) have assessed LLMs’ ability to localize images. However, these efforts often rely on established datasets and lag in incorporating the recent model advances. Moreover, key dimensions like LLMs’ spatial reasoning abilities and potential geospatial biases remain underexplored. To address these gaps, we introduce IMAGEO-Bench , a benchmark that systematically evaluates LLMs’ image geolocalization abilities. Our key contributions are listed as follows. • Comprehensive benchmark datasets . We compile three datasets with diverse geographic regions, urban/rural scenes, and location indicators, constructed via stratified sampling and augmented with privately collected images to ensure broad coverage. • Multi-dimensional evaluation framework . We propose an evaluation framework that looks beyond standard accuracy to assess spatial precision and computational cost, providing a nuanced understanding of model performance across various dimensions. • Geographic bias quantification . We quantify geographic biases in LLMs by revealing performance disparities across different regions at both global and local scales. • Reasoning process analysis . We employ statistical regression to analyze the reasoning processes behind LLM geolocalization and to identify the most influential visual features. 2. Related Work 2.1. Image Geolocalization Image geolocalization in computer vision has focused on two main approaches: classification-based and retrieval-based models ( Weyand et al. 2016 ; Seo et al. 2018 ; Muller-Budack et al. 2018 ; Pramanick et al. 2022 ; Clark et al. 2023 ) . Classification methods (e.g., PlaNet ( Weyand et al. 2016 ) ) treat geolocation as a discrete prediction task by assigning an image to a predefined spatial grid. These methods can benefit from global coverage and efficient inference but rely on rigid spatial discretization, which can constrain fine-grained accuracy and interpretability ( Seo et al. 2018 ) . Retrieval-based methods (e.g., IM2GPS ( Hays and Efros 2008 ) ) localize an image by matching it to a geotagged reference set based on visual or semantic similarity. Although enhancements like deep feature learning and probabilistic fusion ( Vo and Jacobs 2017 ) have improved accuracy, these methods remain reliant on dense reference imagery and struggle in sparsely covered areas. Metric learning models like NetVLAD ( Arandjelović et al. 2016 ) , SAFA ( Shi et al. 2020 ) , and VIGOR ( Zhong et al. 2021 ) further boost retrieval performance using cross-view embeddings and attention mechanisms, particularly effective in visually rich urban settings. However, these methods often face several limitations: (i) they rely on large labeled datasets and generalize inadequately to unseen or underrepresented regions; (ii) their predictions are often opaque, lacking interpretability or insight into the reasoning process; and (iii) they struggle to incorporate multimodal semantic cues (e.g., textual signage, architectural styles, or cultural markers) that humans intuitively use for geolocation. These limitations call for an approach that can leverages world knowledge and multimodal cues to generalize more effectively. In this regard, LLMs offer a promising solution due to their reasoning abilities, cross-modal integration, and greater interpretability. 2.2.

Read the complete text