Evaluating Precise Geolocation Inference Capabilities of Vision Language Models

Neel Jay, Hieu Minh “Jord” Nguyen, Trung Dung Hoang, Jacob Haimes · 2025

Read Evaluating Precise Geolocation Inference Capabilities of Vision Language Models by Neel Jay, Hieu Minh “Jord” Nguyen, Trung Dung Hoang, Jacob Haimes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14412.txt.

Introduction

Neel Jay and co-authors measure how precisely today's vision-language models can guess where a photograph was taken, and what that means for privacy. Their 2025 paper builds a benchmark from Google Street View images sampled to match global coverage, then asks foundation models to locate single unseen images. The models were never trained for this task.

Many achieved median errors under three hundred kilometers, often naming the right region from vegetation, signs, and architecture alone. Giving models agent tools, like search and street-level lookup, cut errors by up to roughly a third. The authors frame the result as a warning: any photo posted online now carries a machine-readable location hint, whether or not it has metadata. Defenses, from cropping to background blurring, deserve the same research energy as the capability. The benchmark gives future privacy work a yardstick: how much location leaks from pixels alone.

Catalog record

CREATORNeel Jay, Hieu Minh “Jord” Nguyen, Trung Dung Hoang, Jacob Haimes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Precise Geolocation Inference Capabilities of Vision Language Models Neel Jay, Hieu Minh “Jord” Nguyen, Trung Dung Hoang, Jacob Haimes Abstract The prevalence of Vision-Language Models (VLMs) raises important questions about privacy in an era where visual information is increasingly available. While foundation VLMs demonstrate broad knowledge and learned capabilities, we specifically investigate their ability to infer geographic location from previously unseen image data. This paper introduces a benchmark dataset collected from Google Street View that represents its global distribution of coverage. Foundation models are evaluated on single-image geolocation inference, with many achieving median distance errors of < < 300 km. We further evaluate VLM ‘‘agents” with access to supplemental tools, observing up to a 30.6% decrease in distance error. Our findings establish that modern foundation VLMs can act as powerful image geolocation tools, without being specifically trained for this task. When coupled with increasing accessibility of these models, our findings have greater implications for online privacy. We discuss these risks, as well as future work in this area. 1 1 1 Datasets and code can be found at: https://github.com/njspyx/location-inference (a) Population Distribution Class Population Range Count Small Urban 50,000–200,000 560 Medium Urban 200,000–500,000 539 Metropolitan 500,000–1.5M 323 Large Metropolitan > 1.5M 141 (b) Geographic Distribution Continent Count North America 26 Asia 21 Europe 21 Africa 13 South America 5 Oceania 2 Table 1: Geographical and population distribution of full benchmark dataset. Figure 1: A snapshot of Google Street View images in the full benchmark dataset. Figure 2: Overview of distance error for select benchmarked models, with human benchmark for reference. Human benchmark mean is 5633 km. Full results can be found in Appendix B. Introduction With the proliferation of user-data collecting services on the Internet, it seems that privacy is becoming increasingly scarce ( Jiang et al. 2022 ) . This issue has only been exacerbated by the advent of Large Language Models (LLMs) and multi-modal Vision-Language Models (VLMs). Research has demonstrated AI’s ability to infer user demographics from both text ( Staab et al. 2024 ) and social media images ( Tömekçe et al. 2024 ) . Inspired by the popular geolocation game, GeoGuessr, we investigate whether this capability extends to precise location inference of everyday images. Geolocation inference refers to the ability to determine exact geographical coordinates (latitude and longitude) given an image or series of images ( Haas et al. 2024 ) . While research in this field has advanced substantially over the past decade, it still remains a challenging problem. Previous works built custom datasets and architectures that focus on specific regions (cities or countries) and fail to generalize beyond distribution shifts in testing ( Suresh, Chodosh, and Abello 2018 ; Wu and Huang 2022 ; Berton, Masone, and Caputo 2022 ; Clark et al. 2023 ) . A recent development is the PIGEON architecture ( Haas et al. 2024 ) , which achieved remarkable global geolocation performance. It is notable that the PIGEON authors have not released the model’s training data and weights, citing the ethical risks of public use. Our work demonstrates that superhuman capabilities can be achieved through foundation VLMs and limited scaffolding. While prior studies ( Wang et al. 2024 ; Zhang et al. 2023 ) have demonstrated geolocation in VLMs to some extent, they limit their evaluation to country classification, rather than precise geographical coordinates. In addition, they fail to test the full potential of VLM capability. Compared to custom architectures, foundation models have shown impressive general reasoning and common sense ( Lu et al. 2024 ) , allowing them to use tools and act as autonomous agents. General VLM benchmarks have considered the evaluation of these agents ( Li et al. 2024 ) , but haven’t explored them in the context of geolocation. To address these research gaps, we collect a dataset of images from Google Street View to benchmark image geolocation. An evaluation of single-image location inference is performed on popular VLM families. Finally, we test if enabling model tool-use improves accuracy. The main contributions of this paper are as follows: • Comprehensive dataset for geolocation: Our dataset contains 1602 images taken from Google Street View. The images depict various levels of urbanization and represent a diverse set of countries where Street View data is available. Each image is labeled with exact latitude, longitude, and API parameters. The addition of this metadata means images can be dynamically updated or fetched for agentic purposes. • Base model benchmarks: We present benchmarks of geolocation on single images for popular foundation VLMs. • VLM agent benchmarks: We further test VLM capabilities by giving foundation models access to tools such as Street View or Google Lens. Figure 3: Median distance error for “VLM + Street View” agents over 5 guesses. Larger models typically improve in performance. Methodology Dataset We aim to capture a set of Google Street View images with geographical and urban diversity. Up to 200 unique cities are selected from every continent and population class (see Table 1 ). For each city, we attempt to fetch an image from the Google Street View Static API at a random point within a 10 km radius. Up to 5 attempts were made per city, after which the city is scrapped. The final dataset consists of 1602 images from 1563 unique cities and 88 unique countries, representative of global Street View coverage. Table 1 contains details regarding full dataset distribution. A subset of 319 images is derived by sampling images from each country. Images are labeled with the following: latitude, longitude, city, country, and API parameters used. The field-of-view is fixed to 90 ∘ and pitch (vertical camera rotation relative to a horizontal plane) is fixed to 0 ∘ . The heading (camera rotation relative to true north) is randomized uniformly from 0 ∘ to 360 ∘ . This ensures a diversity of views, as Google Street View images are typically taken on roads/pathways. Since Google Street View consists of a continuous set of billions of images stitched together, the chance of a model already seeing a particular view is near zero. Model Evaluation We benchmark a variety of large- and small-parameter VLMs on the full 1602 image dataset. Popular commercial and open-weight models are chosen. To evaluate image geolocation capabilities, we construct a system prompt allowing models to assume the role of a competitive GeoGuessr player. Models are instructed to provide a Chain-of-Thought (CoT), describing the reasoning process and visual elements of interest in the image. Finally, the models provide a guess, containing predicted country, city, latitude, and longitude. Distance error of predicted coordinates is scored in kilometers (km) using the Haversine formula: d=R\text{ arctan2}(\sqrt{\text{hav }\theta},\sqrt{1-\text{hav }\theta}) where R is the Earth’s mean radius and \theta is the central angle of two points on a sphere. We define: \text{hav }\theta=\sin^{2}\left(\frac{\Delta\phi}{2}\right)+\cos\left(\phi_{1}\right)\cos\left(\phi_{2}\right)\sin^{2}\left(\frac{\Delta\lambda}{2}\right) where \phi are the latitudes of both points and \lambda are the longitudes of both points. Model CoTs are analyzed by tagging them with one of 12 categories. These categories are meant to encapsulate common visual elements that are used for geolocation. Category descriptions, along with CoT results, can be found in Appendix A. VLM Agents Considering the versatile capabilities of foundational models and growing prevalence of AI agents, we evaluate the

Read the complete text