Spatially-Weighted CLIP for Street-View Geo-localization
Ting Han 1,2, Fengjiao Li 2, Chunsong Chen 1, Haoling Huang 3, Yiping Chen 1 ,∗, Meiliu Wu 2 ,∗ · 2026
Read Spatially-Weighted CLIP for Street-View Geo-localization by Ting Han 1,2, Fengjiao Li 2, Chunsong Chen 1, Haoling Huang 3, Yiping Chen 1 ,∗, Meiliu Wu 2 ,∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04357.txt.
Introduction
Han and colleagues present SW-CLIP, a framework for street-view geo-localization that builds spatial reasoning into contrastive learning. Standard CLIP treats every non-matching sample as equally negative, ignoring geography. SW-CLIP instead applies Tobler's First Law, modelling place relations by distance. It encodes places as text, swaps one-hot targets for soft labels from geodesic distance, and adds neighbourhood regularization to preserve local structure. On a multi-city dataset it beats standard CLIP on accuracy, long-tail errors and coherence, arguing for geographic over purely semantic alignment.
Catalog record
| CREATOR | Ting Han 1,2, Fengjiao Li 2, Chunsong Chen 1, Haoling Huang 3, Yiping Chen 1 ,∗, Meiliu Wu 2 ,∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13788 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04357
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spatially-Weighted CLIP for Street-View Geo-localization Ting Han 1,2, Fengjiao Li 2, Chunsong Chen 1, Haoling Huang 3, Yiping Chen 1 ,∗, Meiliu Wu 2 ,∗ Abstract keywords GeoAI, Spatial Autocorrelation, Geo-Localization, CLIP, Street-View, Geographic Alignment. 1 Introduction Street-view geo-localization [ 10 , 11 ] aims to infer the geographic position of a query street-level image by matching it to a gallery of geo-tagged observations. In many real-world deployments, images are accompanied by GPS coordinates [ 9 ] but lack dense semantic text annotations, resulting in inherently weak supervision. Semantic text annotations (e.g., descriptions of streets, landmarks, or building functions) often provide richer localization cues than coordinates alone [ 2 ] . By leveraging these two sources of information, we can view street-view geo-localization as learning retrieval-friendly representations under geographic constraints. Specifically, we learn a shared embedding space in which a query image is closer to its true location and the descriptions within its spatial neighborhood, enabling similarity-based retrieval of the most plausible match from a reference database. Within this formulation, contrastive representation learning [ 1 ] is a natural choice, as it can directly exploit paired cross-modal relationships to learn retrieval-oriented embeddings. However, most contrastive objectives used in vision–language learning, especially CLIP-style InfoNCE [ 4 , 3 ] , implicitly assume a single discrete positive for each pair, treating all other candidates as equally negative. This assumption does not align with geographic space, where nearby locations are inherently more related than distant ones . Some CLIP-based efforts (e.g., GeoCLIP [ 7 ] and AddressCLIP [ 9 ] ) in geospatial science show that CLIP-style retrieval can be adapted to geographic tags, such as GPS coordinates and address hierarchies. However, their contrastive supervision remains fundamentally pairwise: each query has a single positive, while all other samples in the mini-batch are treated as equally negative. As a result, the geographic “false-negative” issue is largely unresolved—locations that are spatially close and should be considered weak positives are instead pushed away as hard negatives during training. A defining property of geographic phenomena is spatial autocorrelation, commonly expressed by Tobler’s First Law of Geography (TFL): near things are more related than distant things [ 6 ] . For street-view images, the same principle suggests that observations captured within a neighborhood (e.g., the same block or adjacent intersections) should not be treated as uniformly negative relative to one another. Instead, they often exhibit continuity in both visual appearance and spatial context. If such nearby locations are penalized as hard negatives during training, the learned embedding space can become sensitive and spatially uncorrelated, leading to spatial bias, which means predictions that land in the correct vicinity but not exactly at the paired coordinate. Spatial-aware CLIP [ 2 , 5 , 8 ] variants highlight the importance of modeling spatial structure, but they do not explicitly instantiate Tobler-consistent, distance-decayed supervision for geo-localization. Therefore, our contribution is not “CLIP for geospatial tasks”, but a general mechanism for geographic alignment: distance-weighted soft supervision coupled with neighborhood-consistency regularization to preserve and strengthen neighborhood structure in the embedding space. To this end, we propose Spatially-Weighted CLIP (SW-CLIP) , a novel spatial autocorrelation-oriented reformulation of CLIP that shifts the objective from semantic alignment to geographic alignment. The key idea is to (1) redefine the text branch as location-as-text, where a location caption encodes a real geographic position using coordinate tokens and administrative tokens, and (2) replace the one-hot supervision in InfoNCE with spatially weighted soft labels derived from geodesic distance. In addition, we explicitly enforce neighborhood consistency within each modality: nearby images should have similar image embeddings, and nearby location texts should have similar text embeddings. These components jointly inject spatial thinking into contrastive learning, producing cross-modal representations that preserve neighborhood structure and better support street-view geo-localization. 2 Method Figure 1 : Motivation and overview of SW-CLIP. Standard CLIP training treats all non-matching samples in a mini-batch as equally negative, which can incorrectly penalize geographically nearby observations that share similar scene context. Guided by Tobler’s First Law of Geography and spatial autocorrelation , SW-CLIP replaces the hard one-hot supervision with a distance-aware spatial soft label: nearby locations receive higher similarity targets while distant locations are down-weighted. This geographic weighting reduces false-negative conflicts in contrastive learning and encourages embeddings to be both retrieval-friendly and spatially coherent for street-view geo-localization. 2.1 Problem Definition We assume a dataset of geo-tagged street-view images {(x_{i},p_{i})}^{N}_{i=1} , where x_{i} is an image and p_{i}=(\phi_{i},\lambda_{i}) denotes the latitude and longitude. Hierarchical identifiers such as street s_{i} and city c_{i} are also available. We learn an image encoder F and a text encoder G that map inputs into shared feature spaces. We compute geodesic distances d_{ij} using the numerically stable Haversine formula: d_{ij}=2Rarcsin(\sqrt{a_{ij}}), (1) a_{ij}=sin^{2}(\frac{\Delta\phi}{2})+cos\phi_{i}cos\phi_{j}sin^{2}(\frac{\Delta\lambda}{2}), (2) where the Earth radius R=6,371,000 m and angles are in radians. Distances provide the basis for spatial weighting and neighborhood definition. 2.2 Location-as-Text Standard CLIP aligns images with natural-language captions that describe semantic content. For geo-localization, we instead define a location caption that explicitly encodes the real geographic position. A simple template is: [Street: s_{i} , City: c_{i} , Country: k_{i} . Lat: \phi_{i} , Lon: \lambda_{i} .]. The critical design choice is that the text branch becomes a learnable encoder of geographic position (Text features are not merely semantic; they are representations of a point in geographic space). Location-as-Text allows the model to retrieve locations using the same mechanism by which CLIP retrieves semantic captions, enabling within-text neighborhood constraints that encourage the location-text embedding space to a continuous geographic manifold. Importantly, the key mechanism by which we inject spatial structure into learning is not the coordinate tokenization itself, but our distance-aware supervision. We compute pairwise geodesic distances d_{i}j using the Haversine formula, derive a distance-decayed spatial soft-label distribution w_{i}j , and optimize a spatially-weighted InfoNCE objective so that nearby locations are treated as graded positives rather than uniformly hard negatives. Together with the neighborhood-consistency regularization described in our framework, these components encourage the embedding space to preserve Tobler-consistent neighborhood relationships. 2.3 Spatially-Weighted CLIP Table 1: Main results of the proposed method in geo-localization task. The best results are in bold . Model Med. GE (m) \downarrow Mean GE (m) \downarrow R@1 \uparrow Geo-Align \uparrow SSI \uparrow City-Align \uparrow CLIP (ViT-B) 276.80 6,723.54 0.421 0.000 0.103 0.259 SW-CLIP (ViT-B) 91.96 449.25 0.910 0.550 0.621 0.737 SW-CLIP (ViT-L) 81.10 428.29 0.994 0.657 0.781 0.922 only L-a-T (ViT-B) 185.40 2980.31 0.565 0.123 0.265 0.308 Given a batch of paired embeddings {(v_{i},t_{i})}^{B}_{i=1} , the standard CLIP image-to-text InfoNCE objective is: L_{CLIP}=-\frac{1}{B}\sum^{B}_{i=1}log\frac{exp(sim(v_{i},t_{i})/\tau)}{\sum^{B}_{j=1}exp(sim(v_{i},t_{j})/\tau)}, (3) where \tau is a temperature. This defines exactly one positive per query and treats all other candidates as negatives. Tobler’s first law suggests that nearby locations are more closely related. We construct a spatial soft label distribution w_{ij} over candidate location texts for each image query i . Firstly, we define a local distance-decay kernel: w^{local}_{ij}=exp(-\frac{d^{2}_{ij}}{2\sigma^{2}})\cdot\mathbf{1}[d_{ij}<d_{cut}], (4) where \sigma and d_{cut} denote the scale and cutoff. Secondly, we incorporate hierarchical geographic relationships w^{prior}_{ij} into normalized final weights w_{ij} . Finally, we modify InfoNCE by weighting the log-softmax term with the spatial soft labels: L_{SW}=-\frac{1}{B}\sum^{B}_{i=1}\sum^{B}_{j=1}w_{ij}log\frac{exp(sim(v_{i},t_{j})/\tau)}{\sum^{B}_{k=1}exp(sim(v_{i},t_{k})/\tau)}, (5) shifting CLIP from semantic alignment to geographic alignment. This strategy is rewarded for producing high similarity not only for the exact