SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation

Kai-Hendrik Cohrs 1 Zuzanna Osika ∗2 Maria Gonzalez-Calabuig ∗1 Vishal Nedungadi ∗3 Ruben Cartuyvels 4 Steffen Knoblauch 5 Joppe Massant 6 Shruti Nath †7 Patrick Ebel †4 Vasileios Sitokonstantinou †3 · 2025

Read SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation by Kai-Hendrik Cohrs 1 Zuzanna Osika ∗2 Maria Gonzalez-Calabuig ∗1 Vishal Nedungadi ∗3 Ruben Cartuyvels 4 Steffen Knoblauch 5 Joppe Massant 6 Shruti Nath †7 Patrick Ebel †4 Vasileios Sitokonstantinou †3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10370.txt.

Introduction

Maria Gonzalez-Calabuig and nine co-authors tackle a trust problem in Earth observation, where AI models read satellite images. Large geospatial foundation models often fail quietly in places that were rare in their training data. SHRUG-FM is their framework for spotting likely failures in advance and holding back a prediction instead of issuing it.

The framework combines three signals: unusual inputs, unusual internal patterns, and the model's own uncertainty about its answer. Tests cover burn scar mapping, flood mapping, and landslide detection, three urgent tasks after disasters. SHRUG-FM lowered risk on the samples it kept and beat single-signal baselines, while a simple decision tree kept its hold-back choices easy to explain.

Catalog record

CREATORKai-Hendrik Cohrs 1 Zuzanna Osika ∗2 Maria Gonzalez-Calabuig ∗1 Vishal Nedungadi ∗3 Ruben Cartuyvels 4 Steffen Knoblauch 5 Joppe Massant 6 Shruti Nath †7 Patrick Ebel †4 Vasileios Sitokonstantinou †3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation Kai-Hendrik Cohrs 1 Zuzanna Osika ∗2 Maria Gonzalez-Calabuig ∗1 Vishal Nedungadi ∗3 Ruben Cartuyvels 4 Steffen Knoblauch 5 Joppe Massant 6 Shruti Nath †7 Patrick Ebel †4 Vasileios Sitokonstantinou †3 Abstract Geospatial foundation models for Earth observation often fail to perform reliably in environments underrepresented during pretraining. We introduce SHRUG-FM, a framework for reliability-aware prediction that integrates three complementary signals: out-of-distribution (OOD) detection in the input space, OOD detection in the embedding space and task-specific predictive uncertainty. Applied to burn scar segmentation, SHRUG-FM shows that OOD scores correlate with lower performance in specific environmental conditions, while uncertainty-based flags help discard many poorly performing predictions. Linking these flags to land cover attributes from HydroATLAS shows that failures are not random but concentrated in certain geographies, such as low-elevation zones and large river areas, likely due to underrepresentation in pretraining data. SHRUG-FM provides a pathway toward safer and more interpretable deployment of GFMs in climate-sensitive applications, helping bridge the gap between benchmark performance and real-world reliability. 1 Introduction Following the success of foundation models in natural language processing and computer vision, geospatial foundation models (GFMs) for Earth observation (EO) are gaining traction [ 31 ] . Trained on large-scale datasets gathered by satellites such as Sentinel and Landsat, these models aim to learn transferable representations of the Earth’s surface. Recent examples like Clay [ 2 ] , SSL4EO-S12 [ 29 ] , Prithvi [ 8 ] , EarthPT [ 27 ] and Scale-MAE [ 25 ] capture detailed spatial and spectral patterns across diverse regions and timescales. These general-purpose representations are provided as precomputed embeddings or pre-trained models and can be applied to downstream tasks, out-of-the-box or with minimal finetuning to capture more application-specific features. However, ongoing research questions the reliability of foundation models for EO in real-world scenarios [ 34 , 17 ] . These models can fail under spatial extremes, such as underrepresented geographies (e.g., deserts, polar regions, high latitudes etc.) and temporal extremes, such as atypical seasons, extreme weather events or long-term environmental shifts like droughts, due to gaps in geographic or seasonal coverage of the data used to pre-train the models [ 30 , 34 ] . Most models lack built-in mechanisms for detecting out-of-distribution (OOD) inputs or for quantifying uncertainty, often producing overconfident predictions. Recent benchmarks such as REOBench [ 13 ] , GeoBench [ 11 ] and PANGAEA [ 17 ] evaluate GFMs under conditions that better reflect real-world deployment, including shifts in geography, seasonality and sensor type. The results show that GFMs are regularly outperformed by simple supervised baselines, highlighting the need for systematic reliability assessments of their predictions, including OOD detection and uncertainty estimation. We present an initial version of SHRUG-FM (Systematic Handling of Real-world Uncertainty for Geospatial Foundation Models), a framework for reliability-aware prediction in GFMs for EO (Figure 1 ). It integrates three complementary signals: (1) OOD detection in the input space, (2) OOD detection in the embedding space and (3) task-specific predictive uncertainty. The first two identify samples that deviate from the training distribution, either in raw inputs or embeddings, while the third captures uncertainty in the downstream model’s predictions. Together, these signals provide a comprehensive reliability assessment, supporting deployment decisions and guiding future pretraining data strategies. We demonstrate SHRUG-FM on burn scar segmentation (ExEBench [ 34 ] ) using models trained on the SSL4EO-S12 dataset [ 30 ] , which offers global coverage and multiple pretrained encoders of varying sizes and training strategies, enabling systematic comparisons. We incorporate geospatial context via HydroATLAS [ 14 ] , overlaying predictions and uncertainties onto layers such as elevation or land cover. This enables users to identify unreliable predictions and link them to specific environmental features, revealing interpretable gaps in the foundation model’s pretraining. Such diagnostics are critical for climate-sensitive applications, where overconfident or inaccurate predictions can misguide interventions. While demonstrated on burn scars, SHRUG-FM is designed to extend to other GFMs and tasks and we will evaluate its broader utility in future work. Figure 1: The SHRUG-FM framework. It computes three complementary signals: OOD detection in the input, OOD detection in the embeddings and task-specific predictive uncertainty. These are combined to enable reliability-aware predictions, flagging or abstaining from low-confidence outputs. 2 Data and Methods Our work builds on SSL4EO-S12 [ 30 ] , a large-scale global Sentinel-2 dataset (Appendix 5.1 ), and pretrained models derived from it. These models use different self-supervised learning strategies, contrastive learning (MoCo), self-distillation (DINO) and masked autoencoding (MAE), with a ViT-S/16 backbone. All models are frozen and used as feature extractors for downstream segmentation tasks. We apply SHRUG-FM to burn scar segmentation (ExEBench Burn Scars [ 34 ] ). Wildfires are increasing in frequency and intensity under climate change [ 22 ] and accurate mapping of burn scars is essential for understanding local fire regimes, monitoring ecosystem recovery and supporting risk management [ 19 , 18 ] . The burn scars dataset captures high-impact events that are often underrepresented in a foundation model’s pretraining distribution and present real-world challenges such as cloud cover, class imbalance and high variability in event size and year, making them ideal for studying reliability under distribution shift. For uncertainty estimation and evaluation, we train a decoder on the downstream dataset. For the OOD detection components in the raw input and embedding spaces, we match the spectral bands between SSL4EO-S12 and the downstream data to ensure comparability with the pretraining distribution (Appendix 5.1.2 ). To interpret the flags, we examine their spatial distribution and link them with HydroATLAS attributes [ 14 ] (Appendix Table 1 , Fig. 5 ). This reveals which features are most associated with high uncertainty or distributional mismatch, providing insights into what the GFM may have underrepresented during pretraining. Out-of-Distribution Detection. To detect inputs that deviate from the pretraining distribution, we analyze their position in both the raw input space and the embedding space of the foundation model. We apply k -means clustering on the SSL4EO-S12 dataset in both spaces, selecting the number of clusters via the heuristic elbow method ( k{=}15 ). For each test sample, we compute its Euclidean distance to the nearest cluster centroid. We analyze the densities of these distances (Appendix Fig. 3 ) and observe that downstream samples are generally further away from the pretraining cluster centers. We also compute the Nearest Centroid Distance Deficit (NCDD) [ 23 ] , which compares the distance to the nearest centroid with the summed distances to all others (Appendix 5.2.1 for more details). These two metrics provide complementary signals: the distance measures how far a sample lies from the nearest known group, while NCDD quantifies how confidently it belongs to that group versus being ambiguous between others. Uncertainty Quantification. To effectively measure a model’s uncertainty, we must distinguish between aleatoric (due to inherent data variability) and epistemic (due to a lack of knowledge) uncertainty [ 10 ] . While standard models may provide probability outputs, which can be interpreted as aleatoric uncertainty, these are often overconfident, especially for OOD data [ 20 ] . We improve upon these by training an ensemble of decoders (Appendix 5.3.2 ) on bootstrapped data [ 12 ] as well as a model with dropout uncertainty

Read the complete text