Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response

James Walsh, William Fawcett, Grace Colverd, Raúl Ramos-Pollán, [0.6em] Trillium Technologies · 2026

Read Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response by James Walsh, William Fawcett, Grace Colverd, Raúl Ramos-Pollán, [0.6em] Trillium Technologies (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05405.txt.

Introduction

James Walsh and four co-authors built GeoQuery, a way to search satellite imagery using plain language. Archives of Earth photos are huge, but picture-based search models cannot understand text queries. Training a joint model needs paired data and computing power that are not available at global scale.

GeoQuery avoids that training with a two-stage trick. It writes text descriptions for a subset of one hundred thousand map tiles, tuned so text distances mirror visual distances. Queries search the text first, then the wider visual archive. On disaster queries it placed floods best, and responders used it during a Brisbane cyclone.

Catalog record

CREATORJames Walsh, William Fawcett, Grace Colverd, Raúl Ramos-Pollán, [0.6em] Trillium Technologies
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response James Walsh, William Fawcett, Grace Colverd, Raúl Ramos-Pollán, [0.6em] Trillium Technologies Abstract Semantic search of Earth observation archives remains challenging. Visual foundation models such as CLAY produce rich embeddings of satellite imagery but lack the natural-language grounding needed for intuitive query, and full contrastive training of a remote-sensing CLIP-style model requires paired data and compute that are unavailable at global scale. To allow natural-language querying at global scales, we present GeoQuery, a zero-shot retrieval system that sidesteps data and compute constraints through a two-stage semantic and visual search, leveraging a natural-language embedding of a subset (proxy) of global data. Rather than training a joint encoder, we generate language descriptions for a 100k proxy subset of global Sentinel-2 tiles and optimise the description-generation prompt so that distances in the resulting text-embedding space correlate with distances in the frozen CLAY visual-embedding space. Queries are resolved in two stages, with a text-similarity search over the proxy subset followed by a visual nearest-neighbour search over worldwide CLAY embeddings. On 76 disaster-location queries covering UK floods, US wildfires, and US droughts, GeoQuery achieves 31.6% accuracy within 50 km, with the strongest performance on floods (50% within 50 km) where terrain features are well captured by RGB embeddings. Deployed within a crisis-response system called ECHO , GeoQuery identified vulnerable areas during Cyclone Alfred’s 2025 approach to Brisbane, with downstream flood simulations reproducing historical patterns. Prompt-aligned proxies offer a practical bridge between EO foundation models and operational retrieval when full contrastive training is out of reach. 1 Introduction Earth observation (EO) archives have grown faster than the tooling to query them. Operators searching for disaster-relevant imagery using queries such as “areas vulnerable to flooding” and “recent burn scars” must today fall back on coordinate-based queries or manual inspection, because the strongest visual foundation models for satellite imagery, such as CLAY [ 6 ] , produce rich embeddings without text grounding. The natural fix, training a remote-sensing CLIP-style model end-to-end, requires paired image-text data and contrastive compute that are out of reach at global scale. GeoQuery is a zero-shot retrieval system that achieves natural-language access to global Sentinel-2 imagery without end-to-end contrastive training, by aligning text and image spaces indirectly through prompt-aligned proxies. We sample a 100k subset of global tiles, generate a language description for each via a vision-language model, and optimise the description-generation prompt so that pairwise distances in the resulting text embeddings track pairwise distances in the frozen CLAY visual embeddings. A natural-language query is then resolved in two stages. The first performs a text-similarity search over the 100k proxy descriptions, and the second performs a visual nearest-neighbour search over worldwide CLAY embeddings, using the retrieved proxy tiles as visual anchors. We demonstrate GeoQuery within ECHO , an operational crisis-response system using Agentic Action Graphs (AAGs) to orchestrate complex workflows, where it supported Australia’s National Emergency Management Agency during Cyclone Alfred’s 2025 approach to Brisbane. Whilst crisis response provides our motivating application, the contribution is the prompt-aligned proxy method itself, together with an empirical map of where indirect alignment suffices and where it does not, such as flood-prone terrain (well captured) and ephemeral wildfire scars in RGB (poorly captured). We situate GeoQuery against three lines of prior work, namely vision-language foundation models, EO-specific representation learning, and LLM-based agentic orchestration. The remainder of the paper is organised as follows. Section 2 reviews vision-language and EO foundation models alongside prior work on agentic orchestration. Section 3 describes the construction of the joint embedding space and the two-stage retrieval procedure. Section 4 reports the ablation study and the Cyclone Alfred deployment, and section 5 interprets the failure modes and outlines further work. Appendix A gives the full ablation tables, appendices B and C document the ECHO crisis-response framework and its tool library, and appendix D presents CrisisSim case studies. 2 Background Vision-language models have made image retrieval possible through joint embedding spaces. CLIP [ 23 ] demonstrated cross-modal alignment at scale by training on 400 M image-text pairs, enabling zero-shot transfer to novel domains, and subsequent work has extended this approach with larger noisy corpora [ 14 ] , captioning-based bootstrapping [ 19 ] , and improved contrastive objectives [ 36 ] . Recent EO-specific models build on this foundation but face unique challenges. CLAY [ 6 ] provides multi-spectral, multi-resolution embeddings through masked autoencoding on Sentinel and Landsat imagery, capturing seasonal and atmospheric variations but lacking text grounding. Prithvi-EO-2.0 [ 27 ] , trained on 4.2 million global time series samples from NASA’s Harmonised Landsat and Sentinel-2 (HLS) archive at 30 m resolution, performs land-cover classification well but requires task-specific fine-tuning. SatMAE [ 7 ] employs temporal and spectral masking strategies tailored to satellite imagery’s unique characteristics. A wider family of EO foundation models, including Scale-MAE [ 24 ] , SatlasPretrain [ 1 ] , SpectralGPT [ 13 ] , and the multi-sensor DOFA [ 33 ] , addresses spatial scale, dataset coverage, and spectral richness, with a recent survey provided by [ 31 ] and a unified evaluation suite by GEO-Bench [ 18 ] . GeoRSCLIP [ 38 ] and RemoteCLIP [ 20 ] adapt CLIP architectures for remote sensing using the 5 M-pair RS5M corpus and a corpus assembled from heterogeneous box and mask annotations, whilst SkyScript [ 28 ] and DOFA-CLIP [ 32 ] extend the contrastive paradigm with semantically diverse pairs and multi-sensor inputs, but all rely on paired image-text data that does not exist at a globally consistent scale. LLM-based orchestration has been applied across scientific domains, building on foundational work in self-taught tool use [ 25 ] , reasoning-action interleaving [ 35 ] , and multi-agent conversation frameworks [ 30 ] . ChemCrow [ 4 ] augments large language models with eighteen chemistry tools to plan and execute syntheses, whilst Coscientist [ 2 ] drives end-to-end experimental automation, including the optimisation of palladium-catalysed cross-couplings. ProtAgents [ 10 ] coordinates protein design workflows across multiple specialised models, and GeoGPT [ 37 ] demonstrates applications to geospatial analysis but lacks the human oversight required for high-stakes scenarios. These systems decompose complex tasks into tool invocations, but crisis response demands additional constraints, such as interpretability, human validation gates, and deterministic verification. For this work, we place GeoQuery inside of ECHO , an AAG processor (for more details see section B.1 ). Earth observation presents distinctive challenges not encountered in natural image domains. The integration of optical, synthetic-aperture radar (SAR), and hyperspectral modalities, each governed by distinct physical principles and noise characteristics, creates considerable heterogeneity. Temporal variations arising from seasonal cycles, phenological processes, and atmospheric conditions fundamentally alter scene appearance. Moreover, features manifest differently across spatial scales, from sub-metre commercial imagery to 10 m Sentinel-2 acquisitions, whilst geographic and sensor-specific variations introduce substantial domain shifts. Several benchmarks address specific crisis-relevant tasks at scale, such as flood segmentation in Sen1Floods11 [ 3 ] , global flood mapping in WorldFloods [ 21 ] , and post-disaster building damage assessment in xBD [ 12 ] , but each requires task-specific labels rather than supporting open-ended retrieval. Our approach addresses these challenges through a two-stage retrieval strategy that applies expensive vision-language model inference to a proxy subset whilst

Read the complete text