Object-Level Explanations for Image Geolocation Models: a GeoGuessr use-case
Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin · 2026
Read Object-Level Explanations for Image Geolocation Models: a GeoGuessr use-case by Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00912.txt.
Introduction
Emilie Durrieu, Christophe Hurter, Philippe Muller, and Victor Boutin develop an object-level way to explain image geolocation models, using GeoGuessr as the test case. Human players rely on concrete cues such as road markings, vegetation, and building details. Standard attribution maps highlight diffuse areas, so it is hard to tell whether models use similar evidence or which specific objects drive a prediction.
Their pipeline extracts salient regions from attribution maps and segments them into object-like elements. Deletion and insertion tests then measure how much each element matters, compared with random regions of similar size. On a three-country benchmark, attribution-guided crops consistently retain more predictive information than random crops, giving a step toward interpretable object-level analysis.
Catalog record
| CREATOR | Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00912
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Object-Level Explanations for Image Geolocation Models: a GeoGuessr use-case Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin Abstract When humans play geolocation games such as GeoGuessr, they rely on concrete visual cues, such as road markings, vegetation, or architectural details, to infer where an image was captured. Whether image geolocation models rely on similar object-level evidence remains difficult to determine, as attribution methods like Grad-CAM typically highlight diffuse regions rather than coherent visual entities, making it difficult to link model predictions to specific objects or perceptible patterns. In this work, we propose an object-centric analysis pipeline to investigate the visual evidence used by geolocation models. Starting from attribution maps, we extract salient regions and segment them into object-like elements. We evaluate their predictive relevance through deletion and insertion tests, comparing attribution-guided crops to randomly selected regions with similar coverage. Experiments on a three-country benchmark show that attribution-guided crops consistently retain more information for the model’s prediction than random crops. These results suggest that attribution maps can be decomposed into interpretable, perceptible elements, providing a step toward object-level analysis of geolocation models. 1 Introduction Image-based geolocation aims to predict where an image was captured using only its visual content. Beyond practical applications such as location verification or geographic indexing, this task has gained public attention through games such as Geoguessr 1 1 1 https://www.geoguessr.com and WorldGuessr 2 2 2 https://www.worldguessr.com . When humans solve these tasks, they typically rely on recognizable objects and structures, such as road signs, vegetation, vehicles, and architectural elements. Modern Convolutional Neural Networks (CNNs) achieve strong performance by framing geolocation as a large-scale classification problem 20 . However, understanding which visual evidence drives these predictions remains difficult, limiting our ability to interpret, trust, and analyze geolocation models at a meaningful, concept-level granularity. Moreover, attribution maps such as those obtained with GradCAM 14 often highlight broad or overlapping regions that do not correspond to discrete, perceptible objects, making it difficult to link model decisions to interpretable visual patterns. To address this, we propose an object-centric analysis pipeline for geolocation interpretability. Attribution maps produced from a trained classifier are first thresholded to identify salient regions. These regions are then segmented into object-like elements using a segmentation model, producing discrete visual units that can be analyzed individually. Finally, we evaluate whether these extracted elements preserve predictive information through deletion and insertion-based faithfulness tests (see Figure 1 ). Our experiments on a three-country benchmark show that attribution-guided object-like elements retain more predictive information than randomly selected regions of similar coverage. These results suggest that attribution maps contain localized evidence that can be meaningfully decomposed into perceptible visual regions, opening new opportunities for object-level interpretability and concept-based explanations in image geolocation models. Figure 1 : Overview of the proposed object-centric pipeline: (a) Saliency maps are extracted from a trained classifier using GradCAM++. (b) Images are segmented into candidate regions. (c) Each segment is scored based on overlap with attribution maps to identify relevant object-like elements. (d) Faithfulness validation is performed with a insertion/deletion test to assess their predictive relevance. 2 Related Work 2.1 Geolocation Models Automatic image geolocation aims to predict the location where an image was taken, using either global descriptors or deep learning features. Early methods such as IM2GPS 7 relied on hand-crafted features and nearest-neighbor retrieval over large image databases. More recent approaches employ deep convolutional neural networks to formulate geolocation as a classification problem: PlaNet 20 divides the world into discrete cells and predicts the most likely location, while IM2GPS Revisited 18 and PIGEON 6 improve accuracy using CNN features and hierarchical modeling. These methods achieve impressive performance on both world-scale and street-level datasets, but they generally focus on prediction accuracy rather than understanding the visual cues that drive their decisions. 2.2 Explainable AI in vision Attribution and visualization methods in XAI aim to reveal which parts of an image influence a model’s prediction. Gradient-based methods such as GradCAM 14 and Integrated Gradients 17 generate class-specific attention maps from convolutional features, while perturbation-based methods like LIME 13 and RISE 11 estimate importance by observing output changes under input modifications. Gradient-based approaches are particularly convenient for our study as they provide smooth and spatially coherent maps that can be efficiently computed and combined with segmentation to isolate object-like regions. Applying XAI to geolocation models is relatively unexplored at the object level. Shi et al. 15 visualize city-level CNN predictions to highlight relevant regions, showing that urban structures and landmarks drive the model’s decisions. Our work builds on this idea but focuses on object-level interpretability, extracting object-like elements from attribution-guided regions to study their relevance. 2.3 Concept-based XAI Concept-based explanations provide a more structured view of model reasoning by linking predictions to human-interpretable concepts. Concept Bottleneck Models 9 train networks to first predict predefined concepts before outputting a final label, ensuring interpretability by design. Automated Concept-based Explanation (ACE) 5 extracts visual concepts from images using segmentation and clustering, then evaluates their relevance to model predictions. In contrast, our approach focuses on the geolocation context and extracts object-like elements directly from attribution maps, without relying on predefined concepts or modifying the model. This enables a post-hoc, object-level analysis of the visual cues driving geolocation predictions. 3 Methodology We investigate whether visual regions highlighted by attribution methods capture predictive information in geolocation models. Our approach consists of three main steps illustrated in Figure 1 : (1) Train a CNN classifier for geolocation. (2) Extract attribution-guided regions from model predictions. (3) Decompose these regions into object-like elements and evaluate their predictive relevance. 3.1 Problem Setup Let f be a Convolutional Neural Network (CNN) trained to predict the country of origin among N possible countries. Given an input image x representing a ground-level geolocation image, the model outputs a logit vector f(x) in \mathbb{R}^{N} , which is then converted into a probability distribution via softmax. Our goal is to identify the visual cues present in x that contribute to the model’s predictions. 3.2 Obtaining visual explanations To localize the most predictive image regions, we apply feature attribution methods to our model f . For a given input image and predicted class, these methods produce an attention map highlighting pixels that strongly influence the output. We threshold the map to retain the most salient pixels by selecting the top- p percentile of pixels based on attribution values, with p chosen empirically by evaluating the resulting coverage and faithfulness on a validation set. This strategy balances keeping highly informative pixels while avoiding overly large or diffuse regions. We refer to the obtained salient maps as attribution-guided regions . 3.3 Extracting object-like elements Within each attribution-guided region, we extract discrete visual elements using a segmentation model. We define an object-like element as a perceptible visual unit, such as a car or a street sign. These elements are not assigned semantic labels, and their shapes depend on the segmentation output. Each candidate segment is assigned a relevance score based on three factors: 1. Overlap with high-saliency regions: the fraction of the segment that overlaps pixels above a threshold in the attribution map. 2. Average importance: the mean attribution value within the segment. 3. Central importance: the attribution value at the segment’s geometric center. These factors are then combined