Towards Interpretable Geo-localization: a Concept-Aware Global Image-GPS Alignment Framework
Furong Jia, Lanxin Liu, Ce Hou, Fan Zhang, Xinyan Liu, Yu Liu · 2025
Read Towards Interpretable Geo-localization: a Concept-Aware Global Image-GPS Alignment Framework by Furong Jia, Lanxin Liu, Ce Hou, Fan Zhang, Xinyan Liu, Yu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01910.txt.
Introduction
Furong Jia leads five co-authors on a geo-localization system that explains its guesses through clear concepts. Geo-localization finds where a photo was taken from cues like climate, landmarks, and building style. Strong prior models align images with locations but cannot say why. Concept methods that explain other models fit poorly with that alignment goal.
Their module projects both image and location codes onto a shared bank of geographic concepts. Examples include tropical climate, mountains, and cathedrals as named concept slots. A concept level loss sharpens alignment inside that shared space. Tests beat a leading prior model on accuracy and lift other geospatial tasks too. The concept view also reveals richer reasons behind each location call.
Catalog record
| CREATOR | Furong Jia, Lanxin Liu, Ce Hou, Fan Zhang, Xinyan Liu, Yu Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01910
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Interpretable Geo-localization: a Concept-Aware Global Image-GPS Alignment Framework Furong Jia, Lanxin Liu, Ce Hou, Fan Zhang, Xinyan Liu, Yu Liu Abstract Worldwide geo-localization involves determining the exact geographic location of images captured globally, typically guided by geographic cues such as climate, landmarks, and architectural styles. Despite advancements in geo-localization models like GeoCLIP, which leverages images and location alignment via contrastive learning for accurate predictions, the interpretability of these models remains insufficiently explored. Current concept-based interpretability methods fail to align effectively with Geo-alignment image-location embedding objectives, resulting in suboptimal interpretability and performance. To address this gap, we propose a novel framework integrating global geo-localization with concept bottlenecks. Our method inserts a Concept-Aware Alignment Module that jointly projects image and location embeddings onto a shared bank of geographic concepts (e.g., tropical climate, mountain, cathedral) and minimizes a concept-level loss, enhancing alignment in a concept-specific subspace and enabling robust interpretability. To our knowledge, this is the first work to introduce interpretability into geo-localization. Extensive experiments demonstrate that our approach surpasses GeoCLIP in geo-localization accuracy and boosts performance across diverse geospatial prediction tasks, revealing richer semantic insights into geographic decision-making processes. Introduction Image geo-localization refers to the task that determine geographic coordinates from visual content alone. With applications ranging from ecological monitoring ( Beery et al. 2022 ; Rußwurm et al. 2020 ) to disaster response ( Sathianarayanan, Hsu, and Chang 2024 ) , it has attracted increasing attention. However, achieving accurate worldwide image geo-localization remains technically challenging since the enormous variability in Earth’s geographical landscapes. Recent CLIP-based geo-localization models (Fig. 1 A) ( Vivanco Cepeda, Nayak, and Shah 2023 ; Klemmer et al. 2025 ; Zhong et al. 2022 ) have shown promising performance in zero-shot geo-localization by aligning image and geographical location (latitude, longitude) embeddings in a shared space. While this approach enables global location retrieval, it still faces the challenge of distinguishing visually similar scenes from geographically distinct areas. This is because successful geo-localization requires more than just learning spatial distributions of visual landscapes, it demands an understanding of broader geographic knowledge and fine-grained concepts (Fig. 1 B) to distinguish between locations across the globe. Relying solely on raw geographical coordinates to learn location embeddings is insufficient for encoding such world knowledge. Figure 1: Concept-aware geo-localization overview. (A) CLIP-based geo-localization model predicts GPS coordinates by directly mapping visual embeddings to a location embedding gallery, following the image-location alignment. (B) Our approach inserts a concept bottleneck between image and GPS, grounding GPS prediction in semantically meaningful cues. Beyond performance, interpretability of image geo-localization is also important not only for reliable geographic decision-making but also for understanding the spatial cognitive abilities of AI ( Roberts et al. 2023 ) . The reasoning behind geo-localization is inherently complex, involving diverse cues like landmarks, architectural style, and environmental features. While prior work either overlooks interpretability or focuses solely on pixel-level attribution methods ( Theiner, Müller-Budack, and Ewerth 2022 ) that quantify region importance through low-level visual features, it lacks the conceptual understanding required for high-level tasks. This underscores the need for concept-based interpretability that grounds predictions in human-understandable semantic concepts. To achieve more precise and interpretable geo-localization, we introduce a framework that integrates precise location recognition with concept-driven alignment. Specifically, we introduce a Geography-Driven Concept Set, comprising regionally significant attributes carefully selected for their clear geospatial meaning, allowing the model to ground its predictions in recognizable geographic features. This enhances the model’s interpretability by linking its outputs to well-defined concepts that are easily understood. Additionally, we propose a Concept-Aware Alignment Module that aligns image and location embeddings within a geography-driven concept subspace defined by CLIP-based embeddings. The concept-aware alignment encourages image and location embeddings to converge toward similar distributions within an interpretable semantic space, enhancing interpretability, while preserving accuracy. Our contributions can be summarized as follows: • We develop a systematic approach to construct Geography-Driven Concept Sets from regionally salient attributes for interpretable geographic grounding. • We propose a Concept-Aware Alignment Module that aligns image and location embeddings within an interpretable, geography-driven concept subspace. • Extensive experiments demonstrate that our framework not only provides intuitive explanations but also improves GeoCLIP’s performance on geo-localization tasks as well as downstream tasks. Related Work Image Geo-localization While recent studies have explored the potential of large language models (LLMs) for geo-localization via natural-language reasoning and direct coordinate prediction ( Xu et al. 2024 ; Jia et al. 2024 ; Zhou et al. 2024 ; Jia et al. 2025 ; Ye et al. 2024 ; Wang et al. 2025 ) , the majority of existing methods still fall into two primary paradigms: classification-based and retrieval-based approaches ( Vivanco Cepeda, Nayak, and Shah 2023 ) . Classification-based approaches discretize the earth’s surface into predefined cells and train models to assign each image to one of these regions ( Weyand, Kostrikov, and Philbin 2016 ; Seo et al. 2018 ; Vo, Jacobs, and Hays 2017 ; Pramanick et al. 2022 ; Clark et al. 2023 ) . While conceptually straightforward, this formulation enforces a fixed spatial granularity that often causes large localization errors near cell boundaries or in underrepresented areas. To improve interpretability under this paradigm, Theiner et al. ( Theiner, Müller-Budack, and Ewerth 2022 ) introduce a semantic partitioning strategy that replaces fixed or arbitrarily defined spatial partitions with more coherent region definitions. Another widely adopted paradigm is retrieval-based geo-localization, where the goal is to match a query image against a gallery—either of reference images or location embeddings—based on similarity ( Workman, Souvenir, and Jacobs 2015 ; Tian, Chen, and Shah 2017 ; Liu and Li 2019 ; Shi et al. 2020 ; Yang, Lu, and Zhu 2021 ; Zhu et al. 2023 ) . Among retrieval-based methods, GeoCLIP was the first work employing geographical location encoding and retrieve location through geo-alignment ( Vivanco Cepeda, Nayak, and Shah 2023 ) , leveraging contrastive paradigm and CLIP’s pretrained image encoder ( Radford et al. 2021 ) . Building on this, we propose enhancing image–location retrieval with interpretable geographic concepts as intermediate anchors for alignment. Concept-based Model Interpretation The concept-based interpretation has received considerable attention for its ability to ground model decisions in high-level, human-interpretable concepts ( Yeh et al. 2020 ) . This approach is particularly important in image classification ( Ghorbani et al. 2019 ) and multimodal learning ( Parekh et al. 2024 ) , where providing transparent rationales for predictions is essential for building user trust. The Concept Bottleneck Model (CBM) ( Koh et al. 2020 ) exemplifies this paradigm. CBMs decompose prediction into two sequential stages: first, inferring intermediate concepts from inputs, and then using these inferred concepts to produce the final classification. By employing unsupervised techniques to discover and utilize latent concepts, label-free CBMs ( Oikarinen et al. 2023 ) expand the concept-based paradigm to settings where labeled concept annotations are unavailable. Post-hoc CBMs ( Yuksekgonul, Wang, and Zou 2022 ) enable interpretability in pre-trained models without changing their original parameters. Recently, language models have demonstrated the ability to define and organize concepts in a way that aligns with human cognition, enhancing the usability of concept-based interpretability frameworks ( Yang et al. 2023 ) . This line of work is further advanced by models that eliminate the