Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts
Fanny Lehmann, Firat Ozdemir, Yun Cheng, Torsten Hoefler, Sebastian Schemm, Benedikt Soja, Siddhartha Mishra · 2026
Read Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts by Fanny Lehmann, Firat Ozdemir, Yun Cheng, Torsten Hoefler, Sebastian Schemm, Benedikt Soja, Siddhartha Mishra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30184.txt.
Introduction
Fanny Lehmann and six colleagues benchmark AI weather models on year-long rollouts, sorting long-horizon failures into blow-up, drift and loss of seasonality. Short-range forecasts excel, but rolled-out models develop instabilities. Stability turns on small scales: unstable models boost high-frequency energy, while stable ones denoise added noise. Stable models generate unique trajectories from the initial state, not mere stochastic parrots. Ablations on Vision Transformer designs confirm the findings across nine leading models.
Catalog record
| CREATOR | Fanny Lehmann, Firat Ozdemir, Yun Cheng, Torsten Hoefler, Sebastian Schemm, Benedikt Soja, Siddhartha Mishra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49087 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30184
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts Fanny Lehmann, Firat Ozdemir, Yun Cheng, Torsten Hoefler, Sebastian Schemm, Benedikt Soja, Siddhartha Mishra Abstract While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined “instabilities” when rolled out over longer horizons. This work addresses the lack of a formal taxonomy by categorizing these failures into three distinct regimes — blow-up, drift, and loss of seasonality — through year-long rollouts of nine state-of-the-art AI weather models. Our analysis reveals that stability hinges on the treatment of small spatio-temporal scales: unstable models amplify high-frequency energy, while stable models act as denoisers when noise is added to their inputs. Far from reducing these models to mere stochastic parrots , our findings highlight that stable models generate unique weather trajectories, conditioned on the initial state. We verify our findings through ablation studies on architectural design choices, conducted using state-of-the-art Vision Transformer (ViT) AI weather model architectures. 1 Introduction AI weather models have surpassed numerical solvers on standard metrics at weather time scales (up to 15 days), but producing realistic predictions over longer horizons remains a fundamental challenge. A common approach for long-term prediction applies the model autoregressively, feeding its own output back as input at each step. This yields rollouts of arbitrary length, but predictions frequently blow up or become unrealistically blurry as the number of autoregressive steps grows, and no formal characterization of these failure modes exists [ 5 , 4 ] . While long-term stability remains elusive for many AI weather models, recent climate-scale emulators demonstrate remarkable autoregressive robustness, an effect that appears architecture-agnostic. Success has been observed across a variety of backbone architectures, including Spherical Fourier Neural Operators (ACE2 [ 29 ] , LUCIE [ 14 ] ), UNet Convolutional Neural Networks (DLESyM [ 11 ] ), and 3D Swin Transformers (ArchesWeather [ 10 ] ). Typically, the main difference between AI weather models and AI climate emulators is the spatio-temporal resolution of the data with which they are trained and operate. While most state-of-the-art weather models operate at a spatial resolution of [formula] and temporal resolution of 6 hours, climate emulators lower the spatial resolution to [formula] , at best (see Table A.1 ) and sometimes increase the lead time to one day or more. This work investigates the influence of spatio-temporal resolution on long-term stability as a potential explanation for the success of climate emulators on decadal timescales. Prior work has proposed several hypotheses to explain long-rollout instability. Some works suggest that models need to account for the spherical structure of the Earth to correctly represent all spatial scales, using spherical harmonics or HEALPix meshes for this purpose [ 5 , 16 ] . Chattopadhyay et al. identify spectral bias as the root cause of instability [ 8 ] . They propose a new framework (FouRKS) for spectral regularization but only demonstrate its benefits on low-resolution data ( [formula] and 24h lead time), leaving it unclear whether these benefits generalize to the high-resolution ( [formula] ) regimes common to state-of-the-art AI weather models. AI Earth system models are increasingly positioned as tools for studying climate change, given their demonstrated ability to reproduce atmospheric dynamics. Beyond the physical constraints required to design meaningful climate emulators, a prerequisite for any climate application is that models sustain long rollouts without unphysical drift, blow-up, or spectral artifacts. This work supports active efforts in the AI for Weather and Climate community to address time scales longer than weather, such as the Weather Quest targeting subseasonal to seasonal timescales [ 22 ] and the AI4MIP experiment focusing on climatic time scales. 2 Related Work Several AI models achieve decadal stability: ACE2 sustains rollouts up to 81 years [ 29 ] and NeuralGCM up to 40 years [ 17 ] . However, this stability is not unconditional: ACE2 required selecting specific checkpoints, and NeuralGCM yielded stable 40-year rollouts for only 22 of 37 tested initial conditions [ 17 ] . The Triton model achieves 1-year stability through a multi-grid architecture that addresses spectral bias [ 31 ] , but its predictions exhibit significant smoothing after several months, degrading physical realism. Challenges related to long-term stability have already been raised but only described visually [ 8 ] or at a low spatial resolution [ 12 ] . McCabe et al. give an in-depth analysis of stability failure modes in FNO and propose several architectural changes related to spectral convolutions that improved stability up to 400 rollout steps [ 23 ] . Outside of the weather domain, several approaches have been proposed to stabilize autoregressive models. Multi-step loss functions improve long-term stability but are memory-prohibitive for large models, motivating backpropagation through only the final rollout step [ 7 , 34 ] . Instability often arises when autoregressive predictions drift out of the training distribution. Robustness can be improved by injecting noise during rollout [ 21 ] , denoising predictions via diffusion models as in Thermalizer [ 26 ] , or adding regularization terms such as a Jacobian penalty to the loss function [ 24 ] . Finally, Armegioiu highlights the influence of unresolved variables on rollout stability for generative models and shows the benefits of keeping a memory of the physical system to stabilize rollouts [ 1 ] . Figure 1: Rollouts initialized on January 1st, 2021 for nine AI weather models. The lines show 2-meter temperature averaged across the Arctic for the reference ERA5 (black) and the model (color). The shaded areas extend from the minimum to the maximum temperature over the Arctic for ERA5 (grey) and the model (color). FourCastNet and GenCast results are shown only until they reach unphysical values. 3 A Taxonomy of Rollout Instabilities We analyze nine state-of-the-art weather models spanning model sizes from 3.5M to 4.7B parameters and four architecture families (see Table A.1 ). For each model, we run a 2-year rollout (2,920 autoregressive steps at 6-hour lead time), initialized on January 1, 2021 — outside the training period of all models except AIFS (due to sea surface temperature forcing availability for some models). The ERA5 reanalysis [ 15 ] serves as reference data since all models are trained on the ERA5 dataset. Figure 1 shows 2-meter temperature time series over the Arctic, a particularly sensitive diagnostic region where spherical geometry compresses grid spacing at high latitudes, often causing instabilities to manifest there before propagating to lower latitudes. Figure 1 reveals markedly different behavior across models: Aurora, SFNO, and DLESyM track the ERA5 seasonal cycle closely throughout, while the minimum temperature predicted by AIFS, FourCastNet, and FuXi diverges rapidly from ERA5. Table 1: Number of days until the model predictions blow-up, defined as an exponential growth of the spatial extremes (see Appendix B.1 for details). “-” denotes variables absent from the models. Model T2m U10m MSLP Z500 T500 Q500 U300 T100 Q100 Aurora > 730 > 730 > 730 > 730 > 730 > 730 > 730 > 730 > 730 SFNO > 730 > 730 > 730 > 730 > 730 > 730 > 730 > 730 > 730 Pangu > 730 > 730 > 730 > 730 > 730 > 730 > 730 >