Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts
Maria Conchita Agana Navarro, Geng Li, Theo Wolf, María Pérez-Ortiz · 2026
Read Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts by Maria Conchita Agana Navarro, Geng Li, Theo Wolf, María Pérez-Ortiz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23043.txt.
Introduction
Navarro and co-authors test climate foundation models under no-analog shifts. Future climate states may fall outside all training data. Many models train on data that already include future scenes, hiding true weakness. The team trains three models on historical data only, from 1850 to 2014. Tests cover recent years and cross-scenario forcing shifts. The ClimaX foundation model has the lowest raw error. Yet its error shifts most under strong forcing, with rain errors rising notably. Scenario-aware training is needed, they conclude.
Catalog record
| CREATOR | Maria Conchita Agana Navarro, Geng Li, Theo Wolf, María Pérez-Ortiz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36328 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23043
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts Maria Conchita Agana Navarro, Geng Li, Theo Wolf, María Pérez-Ortiz Abstract The accelerating pace of climate change introduces profound non-stationarities that challenge the ability of Machine Learning (ML)-based climate emulators to generalize beyond their training distributions. While these emulators offer computationally efficient alternatives to traditional Earth System Models (ESMs), their reliability remains a potential bottleneck under “no-analog” future climate states—which we define here as regimes where external forcing drives the system into conditions outside the empirical range of the historical training data. A fundamental challenge in evaluating this reliability is data contamination; because many models are trained on simulations that already encompass future scenarios, true out-of-distribution (OOD) performance is often masked. To address this, we systematically benchmark the OOD robustness of three state-of-the-art architectures—U-Net, ConvLSTM, and the ClimaX foundation model—specifically restricted to a historical-only training regime (1850–2014). We evaluate these models using two complementary strategies: (i) temporal extrapolation to the recent climate (2015–2023) and (ii) cross-scenario forcing shifts across divergent emission pathways (SSP1-2.6 and SSP5-8.5). Our analysis within this controlled experimental setup reveals an accuracy vs. stability trade-off: while the ClimaX foundation model achieves the lowest absolute error, it exhibits higher relative performance changes under distribution shifts, with precipitation errors increasing by up to 8.44% under extreme forcing scenarios. Conversely, simpler CNN-based architectures demonstrate greater relative stability. Furthermore, we identify a temperature-precipitation disparity, where models maintain stable temperature projections while exhibiting larger degradation in precipitation under forcing shifts. These findings suggest that when restricted to historical training dynamics, even high-capacity foundation models are sensitive to external forcing trajectories. Our results underscore the necessity of scenario-aware training and rigorous OOD evaluation protocols that account for data contamination to ensure the robustness of climate emulators under a changing climate. 1 Introduction The accelerating pace of climate change and the increasing severity of extreme weather events present urgent challenges for global adaptation and mitigation planning. The 2022 IPCC Sixth Assessment Report (AR6) highlights that continued warming will amplify climate extremes and heighten the risk of crossing critical system thresholds, such as Amazon rainforest dieback or irreversible ice-sheet loss ( 2 ; 12 ) . As governments and industries design infrastructure, assess financial risks, and plan for diverse future emission trajectories, the need for reliable climate projections has never been more pressing. Global Climate Models (GCMs) remain the cornerstone of climate projection, providing physically based simulations of the coupled atmosphere–ocean–land system ( 12 ) . Yet, their computational cost limits the number of scenarios, ensembles, and high-resolution experiments that can be produced, constraining the exploration of uncertainty across socioeconomic pathways and climate trajectories. Machine Learning (ML) offers an appealing complement to traditional modeling: ML-based climate emulators can approximate GCM outputs orders of magnitude faster, enabling rapid experimentation and scenario exploration ( 4 ) . Recent advances—from convolutional and recurrent architectures to Foundation models such as ClimaX ( 10 ) have demonstrated strong skill, yet recent critiques suggest they may overfit to internal variability, leading to performance trade-offs compared to simpler linear baselines ( 9 ) . Furthermore, the ability of these models to maintain fidelity when transferred across divergent forcing scenarios remains an open scientific issue ( 1 ; 5 ) . These approaches generally address two distinct settings: forcing-to-climate mapping (e.g., ClimateBench ( 13 ) ) and weather-learning systems that attempt to derive climate statistics from short-term dynamics, e.g. NeuralGCM ( 6 ) or ACE ( 14 ) ). The latter represents a significantly more complex task, as it requires the model to learn weather physics and accurately aggregate them into stable climate states. However, these gains mask a core unresolved challenge: the ability of ML models to generalize under non-stationary climate conditions . Unlike many ML applications, the climate system is evolving under strong external forcing, meaning that future states will increasingly diverge from the historical distributions on which most ML models are trained. This raises a fundamental methodological challenge regarding data contamination in Out-of-Distribution (OOD) evaluation. Since many emulators are trained on emulated data from physics-based simulations, the models may inadvertently encounter future forcing patterns during pre-training. If an OOD test set contains scenarios that were part of the broader simulation pool used for model development, the evaluation is effectively in-distribution. In this work, we address this by strictly isolating a historical-only training regime to test true “no-analog” extrapolation. Standard ML techniques assume stationary data distributions, an assumption violated by climate trajectories characterized by nonlinear feedbacks and changing boundary conditions. Importantly, climate-change–driven shifts constitute not only differences in the marginal distribution P(X) , but also in the underlying physical relationships P(Y|X) . In such settings, models that have learned spurious correlations or historically bounded patterns may experience degraded performance when confronted with novel forcing combinations ( 8 ; 7 ) . Despite rapid progress, current evaluation protocols provide limited insight into OOD robustness. Frameworks such as ClimateBench ( 13 ) typically evaluate models under a single emission scenario, leaving open the question of how these models behave when forced to extrapolate across divergent pathways. To address this gap, we introduce a comprehensive OOD evaluation framework tailored to the “zero-shot” extrapolation limits of ML climate emulators trained exclusively on historical simulations (1850–2014) following the climate projection protocol established by Nguyen et al. ( 10 ) . We benchmark three architectures (U-Net, ConvLSTM, ClimaX) across two primary axes of non-stationarity: 1. Temporal Extrapolation: Testing models trained on historical forcing (1850–2014) against the recent, accelerated warming of 2015–2023. 2. Cross-Scenario Forcing Shifts: Evaluating the capacity of historical-only models to predict divergent future pathways (SSP1–2.6 and SSP5–8.5). This work is guided by two core questions: (1) how do current ML climate emulators perform when restricted to historical training dynamics, and (2) what do these shifts reveal about the performance-stability trade-off between absolute predictive skill and relative robustness under distribution change? By systematically analyzing model robustness, we identify an “Accuracy vs. Stability” pattern within this experimental configuration: while foundation models achieve high absolute accuracy, they exhibit larger relative performance changes than simpler convolutional models when forced into novel climate regimes. Our results demonstrate that while temperature extrapolation is relatively stable, cross-scenario shifts in precipitation lead to substantial degradation in the evaluated models. This paper makes three main contributions: • We develop a unified OOD evaluation framework for ML-based climate emulation, integrating temporal extrapolation and cross-scenario forcing shifts to capture key forms of non-stationarity. • We provide a systematic comparison of state-of-the-art architectures, including a foundation model specifically evaluated under a historical-only training regime, across these OOD dimensions. • We demonstrate that cross-scenario forcing shifts induce substantial and model-dependent degradation, underscoring the need for domain generalization methods and careful consideration of data contamination in future emulator development. 2 Methodology This section outlines the methodology for adapting Out-of-Distribution (OOD) evaluation techniques to address the challenges specific to ML climate modeling under non-stationary regimes. Our approach operationalizes the theoretical challenges of causal shifts and domain generalization by building on existing OOD frameworks in ML and tailoring them to climate-specific contexts, evaluating how ML models generalize across shifts in spatio-temporal climate data. 2.1 Modeling OOD Generalization in Non-Stationary Climate Dynamics In traditional OOD settings, a model trained on source