Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models
O. Grainge, S. Waheed, J. Stilgoe, M. Milford, S. Ehsan · 2025
Read Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models by O. Grainge, S. Waheed, J. Stilgoe, M. Milford, S. Ehsan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19967.txt.
Introduction
Grainge and co-authors present a 2025 study grading how well vision-language models can guess where a photo was taken. Pinpointing place from visual clues helps rescue work and teaching, but it also endangers privacy. A model that reads street signs and skylines could stalk people from casual posts. The team tested twenty-five top models on four image sets from varied settings.
Generic street photos fooled the models badly. Social-media-style images, though, were placed correctly about sixty-one percent of the time at the tested precision. The paper probes how the models reason their way to a guess. The authors urge quick safeguards as the skill sharpens with each model generation.
Catalog record
| CREATOR | O. Grainge, S. Waheed, J. Stilgoe, M. Milford, S. Ehsan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models O. Grainge, S. Waheed, J. Stilgoe, M. Milford, S. Ehsan Abstract Geo-localization is the task of identifying the location of an image using visual cues alone. It has beneficial applications, such as improving disaster response, enhancing navigation, and geography education. Recently, Vision-Language Models (VLMs) are increasingly demonstrating capabilities as accurate image geo-locators. This brings significant privacy risks, including those related to stalking and surveillance, considering the widespread uses of AI models and sharing of photos on social media. The precision of these models is likely to improve in the future. Despite these risks, there is little work on systematically evaluating the geolocation precision of Generative VLMs, their limits and potential for unintended inferences. To bridge this gap, we conduct a comprehensive assessment of the geolocation capabilities of 25 state-of-the-art VLMs on four benchmark image datasets captured in diverse environments. Our results offer insight into the internal reasoning of VLMs and highlight their strengths, limitations, and potential societal risks. Our findings indicate that current VLMs perform poorly on generic street-level images yet achieve notably high accuracy (61%) on images resembling social media content, raising significant and urgent privacy concerns. Figure 1: Leaderboard of 22 open-source, open-weight and closed-source vision–language models (VLMs) on the GPTGeoChat dataset ( Mendes et al. 2024 ) . Models are ranked by Recall@1km, which is the percentage of predictions within 1 km of the ground truth. Models with 0% recall or a failure rate above 90% are excluded. Introduction Geo-localization, the task of identifying the geographic location of an image using only its visual content, is an important capability for applications such as disaster response, navigation, and geographic education. Traditional approaches have primarily relied on supervised learning with large geotagged datasets or on large-scale image retrieval methods. Recent advances in Vision-Language Models (VLMs) have revealed that these models possess emergent geo-localization abilities, even without task-specific training. Early work in this area focused on models such as CLIP ( Vivanco, Nayak, and Shah 2023 ; Haas, Alberti, and Skreta 2023 ; Wu and Huang 2022 ) , but more recent efforts have applied generative VLMs to this task ( Zhou et al. 2024 ; Mendes et al. 2024 ; Waheed et al. 2025 ) . These models have shown surprisingly strong performance on geo-localization benchmarks, often matching or exceeding the performance of specialized geo-localization models ( Mendes et al. 2024 ) . The rapid improvement of VLM-based geo-localization offers significant opportunities but also introduces serious risks. The widespread availability and multipurpose nature of VLMs make their use difficult to regulate. Precise geo-localization from publicly shared images can be exploited for stalking, surveillance or discrimination. As the performance and accessibility of VLMs continue to improve, developers and users of VLMs will look for new opportunities to generate value from geolocation. In the absence of strong regulation, risks will grow. We see risk in sociotechnical terms ( Lazar and Nelson 2023 ) . Risk and safety are not just functions of the capabilities of AI models, but of the societal context in which these models sit, and in which they are developed, used and regulated. From this view, privacy is not just about protecting the rights of individuals; it also functions as an important check on the power of those developing and deploying technologies with potential surveillance uses ( Solove 2025 ) . A rigorous evaluation of the capabilities and limitations of these models is therefore essential for responsible deployment and for informing policymakers about potential misuse and the need for safeguards. Despite these concerns, there is limited systematic research on the geo-localization performance of generative VLMs. While prior work has examined data leakage and memorization in large language models (LLMs) ( Carlini et al. 2020 ; Carlini et al. 2022 ) , few studies have investigated how generative VLMs perform in geo-localization across diverse, real-world contexts ( Waheed et al. 2025 ) . Moreover, details of important factors such as differences between model families, the effect of model size, and failure cases have largely been overlooked. To bridge this gap, we present a comprehensive black-box evaluation of 25 state-of-the-art generative VLMs, including open-source, open-weight and closed-source systems. We evaluate these models on four benchmark datasets that collectively capture a broad range of scenarios, including globally distributed images (im2gps ( Hays and Efros 2008 ) and im2gps3k ( Vo, Jacobs, and Hays 2017 ) ), urban street-level imagery (OSV5 ( Astruc et al. 2024 ) ), and data curated to resemble images commonly posted on social media (GPTGeoChat ( Mendes et al. 2024 ) ). In addition to measuring accuracy, we also account for cases where models fail to respond. Our results provide the first large-scale comparative study of generative VLMs in geo-localization and highlight both their performance potential and the privacy risks that arise from their growing deployment in real-world applications. Our key contributions are: • We benchmark 25 state-of-the-art generative VLMs on four diverse datasets, offering a detailed comparison of their strengths and limitations. • We analyze the broader societal and ethical risks associated with VLM-based geo-localization, offering insights for future research and policy. The remainder of this paper is organized as follows. The Related Work section summarizes prior research. Experimental Setup details the models, datasets, evaluation metrics, and methodology. Results and Analysis presents our experimental results and a comprehensive performance analysis. Discussion and Implications examines the societal impact and broader implications of our findings. Finally, Conclusion summarizes the study and outlines future research directions. Related Work This section covers prior research relevant to our work, covering both technical approaches to geo-localization and emerging concerns around privacy and societal implications. Geo-localization and Vision-Language Models Traditional geo-localization research ( Hays and Efros 2008 ) treats localization as a nearest-neighbor problem over a large-scale geotagged image database. This approach works well for landmark recognition ( Arandjelovic et al. 2016 ; Weyand et al. 2020 ; Kendall, Grimes, and Cipolla 2015 ; Wang, Wang, and Xie 2020 ) but faces scalability challenges and perceptual aliasing at global levels ( Muller-Budack, Pustu-Iren, and Ewerth 2018 ; Regmi and Shah 2019 ) . To overcome these limitations, classification-based methods ( Weyand, Kostrikov, and Philbin 2016 ; Muller-Budack, Pustu-Iren, and Ewerth 2018 ; Kordopatis-Zilos et al. 2021 ; Izbicki, Papalexakis, and Tsotras 2020 ) partition the Earth into discrete geo-cells, enabling broader global reasoning but introduce trade-offs between spatial resolution and class imbalance ( Kordopatis-Zilos et al. 2021 ; Haas et al. 2024 ) . Recent advances in Vision-Language Models (VLMs) have introduced a new paradigm for geo-localization. Models like CLIP ( Radford et al. 2021 ) introduce semantic reasoning and contextual understanding to feature representations ( Wu and Huang 2022 ; Haas et al. 2024 ; Luo et al. 2022 ; Zhou et al. 2024 ) . More recently, generative VLMs such as GPT-4v ( Achiam et al. 2023a ) have demonstrated the ability to infer geographic locations directly from images, sometimes surpassing specialized geo-localization models in accuracy ( Mendes et al. 2024 ; Waheed et al. 2025 ) . Despite these advances, existing evaluations remain limited. Most prior work examines only a handful of models, leaving critical key factors such