Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models DOI: XXXXXXX.XXXXXXX CCS: Applied computing Document searching CCS: Information systems Recommender systems CCS: Information systems Document filtering CCS: Information systems Geographic information systems
Yuanyuan Tian, Wenwen Li, Xiao Chen, Michael Brook, Michael Brubaker, Anna Liljedahl, Chitta Baral · 2026
Read Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models DOI: XXXXXXX.XXXXXXX CCS: Applied computing Document searching CCS: Information systems Recommender systems CCS: Information systems Document filtering CCS: Information systems Geographic information systems by Yuanyuan Tian, Wenwen Li, Xiao Chen, Michael Brook, Michael Brubaker, Anna Liljedahl, Chitta Baral (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28369.txt.
Introduction
Yuanyuan Tian, Wenwen Li, Xiao Chen, Michael Brook, Michael Brubaker, Anna Liljedahl, and Chitta Baral search for similar environmental events across space and time. Community observers report odd happenings, a beached whale, a sudden fish kill, and linking related reports helps reveal patterns of environmental change that single reports cannot show.
Tian and colleagues combine foundation models in two strategies. CAMERA fuses text and images into richer event descriptions than text alone gives. ASTRA then re-ranks candidates using scale-aware spatial and temporal relevance beside semantic similarity. Tested on reports from the Local Environmental Observer Network, their vision-enhanced methods beat text-only language-model approaches at ranking similar events. Automatic linking, they argue, serves both curators and the public watching local environments change.
Catalog record
| CREATOR | Yuanyuan Tian, Wenwen Li, Xiao Chen, Michael Brook, Michael Brubaker, Anna Liljedahl, Chitta Baral |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28369
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models DOI: XXXXXXX.XXXXXXX CCS: Applied computing Document searching CCS: Information systems Recommender systems CCS: Information systems Document filtering CCS: Information systems Geographic information systems Yuanyuan Tian, Wenwen Li, Xiao Chen, Michael Brook, Michael Brubaker, Anna Liljedahl, Chitta Baral Abstract . Semantic search and recommendation of similar documents, such as news and reports about unusual environmental events (e.g., a dead whale washed ashore in Alaska) that contain spatial and temporal information, is a critical task in Geographic Information Retrieval (GIR). This work presents a novel framework that leverages AI foundation models, including Large Language Models (LLMs) and Vision-Language Models (VLMs), to enable effective similarity search and ranking for such event documents. To support this goal, we introduce two new strategies: (1) CAMERA (Context-Aware Multimodal Event Retrieval Algorithm), which fuses textual and visual information to generate richer embeddings than those derived from text alone; and (2) ASTRA (Adaptive Spatial and Temporal Re-ranking Algorithm), which improves similarity ranking by incorporating scale-dependent spatiotemporal relevance alongside semantic similarity. Experimental results, using a dataset from the Local Environmental Observer Network, demonstrate that our VLM-enhanced methods outperform unimodal, LLM-based approaches in similarity ranking effectiveness. By automatically linking relevant event reports, the proposed framework helps both data curators and the general public gain deeper insights into environmental change and its localized impacts. These findings highlight the potential of AI foundation models to advance GIR through multifaceted, intelligent analysis that integrates key geographic concepts: space, time, scale, and semantics. 1. INTRODUCTION Environmental monitoring increasingly relies on large volumes of heterogeneous reports describing unusual environmental events, such as wildlife mortality, extreme weather anomalies, ecosystem disruptions, and other indicators of environmental change ( 3 ; 10 ) . These reports are often documented in community-based observation systems, scientific platforms, and news media, providing valuable situational awareness for environmental monitoring, disaster response, and environmental research. An important capability for such systems is the ability to discover and recommend events that are similar to a given query event across space and time ( 39 ) . Identifying related events enables analysts to uncover emerging environmental patterns, track the spread of ecological phenomena, and better understand localized impacts of global environmental change ( 12 ) . In Geographic Information Retrieval (GIR), event similarity is inherently multidimensional. Environmental events are characterized by textual descriptions and their spatial location, temporal occurrence, and often visual evidence such as photographs or maps. Consequently, effective retrieval requires jointly reasoning over semantic content, geographic proximity, and temporal dynamics. Traditional retrieval methods, such as keyword-based approaches (e.g., BM25, TF-IDF), rely primarily on lexical overlap between documents. Although these techniques are efficient and interpretable, they struggle to capture deeper semantic relationships across heterogeneous reports. Recent neural retrieval approaches improve semantic matching by embedding queries and documents into a shared dense vector space using pretrained language models. However, these approaches typically treat documents as purely textual objects and overlook the geographic characteristics that fundamentally shape environmental events ( 9 ) . Beyond semantic understanding, environmental event retrieval presents a unique challenge related to scale . The spatial and temporal extent of environmental phenomena can vary dramatically depending on the event type and context. For example, a localized landslide may affect only a small geographic area over a short time window, whereas a mega wildfire or tsunami may span hundreds of kilometers and persist for months. Consequently, determining whether two events are related requires reasoning about the appropriate spatial and temporal scales at which similarity should be evaluated. Existing retrieval and ranking methods typically rely on fixed spatial or temporal thresholds, which may either exclude relevant events or include irrelevant ones when applied across heterogeneous environmental contexts. Addressing this scale dependency remains a longstanding challenge in GIR and GeoAI ( 24 ; 16 ; 25 ) . Recent advances in foundation models offer new opportunities for addressing these challenges. Large Language Models (LLMs) demonstrate strong capabilities in semantic reasoning, and Vision-Language Models (VLMs) enable multimodal understanding. Recent work has assessed the capabilities of LLMs in understanding complex geometries and topological spatial relations, highlighting both their potential and current limitations in geospatial reasoning ( 19 ) . VLMs have been applied to interpret geospatial imagery through language-guided prompts, supporting environmental monitoring and analysis ( 21 ) . These models have shown promise in various retrieval and recommendation tasks. However, most existing applications of foundation models focus on representation learning or text generation, without explicitly incorporating geographic reasoning. In particular, two key challenges remain underexplored for environmental event retrieval. First, effectively integrating visual information with textual event descriptions remains difficult, especially when visual data needs to be interpreted in terms of geographical or environmental context. Second, existing retrieval systems rarely exploit the reasoning capabilities of LLMs to model the scale-dependent nature of spatiotemporal relevance. To address these challenges, we propose a novel framework for multimodal and multiscale spatial-temporal event retrieval and recommendation. Our framework integrates foundation models into a two-stage architecture designed to bridge semantic understanding and geographic reasoning. In the first stage, we introduce the Context-Aware Multimodal Event Retrieval Algorithm (CAMERA) , which constructs enriched event representations by extracting structured visual cues from event images using a VLM. Rather than relying on black-box multimodal embeddings, CAMERA converts visual observations into concise textual descriptors capturing key aspects such as event category, geographic context, and seasonal characteristics. These cues are then integrated with textual metadata to produce multimodal-informed embeddings that remain compatible with standard dense and sparse retrieval pipelines. By combining dense semantic retrieval with lexical matching through rank fusion, CAMERA generates a high-recall candidate set of potentially related events. In the second stage, we introduce the Adaptive Spatio-Temporal Re-ranking Algorithm (ASTRA) , which refines the candidate list by explicitly modeling the scale-dependent nature of environmental events. ASTRA leverages the reasoning capabilities of a LLM to infer query-specific spatial and temporal bandwidths that represent the effective impact scale of the query event. These inferred scales are used to transform raw geographic and temporal distances into adaptive similarity kernels, enabling the ranking model to adjust its sensitivity to spatial and temporal proximity dynamically. The resulting features are integrated within a supervised Learning-to-Rank framework that jointly considers semantic similarity, categorical alignment, and adaptive spatiotemporal relevance. By replacing fixed thresholds ( 43 ) with query-dependent adaptive scales, ASTRA enables a more flexible and context-aware ranking strategy. We evaluate the proposed framework using a large-scale dataset from the Local Environmental Observer (LEO) Network, which contains thousands of environmental event reports contributed by community observers and domain experts. Experimental results demonstrate that our approach improves retrieval and recommendation performance compared with both traditional information retrieval methods and recent neural re-ranking approaches. In particular, our results highlight the importance of integrating multimodal context and scale-aware geographic reasoning for environmental event discovery. The contributions of this work are summarized as follows: • We propose CAMERA , a multimodal retrieval framework that integrates structured visual cue extraction from VLM with hybrid dense and sparse retrieval for environmental event search. • We introduce ASTRA , a scale-aware re-ranking algorithm that leverages LLM reasoning to infer adaptive spatial and temporal scales for event similarity. • We demonstrate that combining multimodal representations with adaptive spatiotemporal reasoning significantly