(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models

Maksim Zhdanov, Ana Lucic, Max Welling, Jan-Willem van de Meent · 2026

Read (Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models by Maksim Zhdanov, Ana Lucic, Max Welling, Jan-Willem van de Meent (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16429.txt.

Introduction

Zhdanov, Lucic, Welling and van de Meent present Mosaic, a weather model that preserves spectral detail lost by many machine-learning forecasts. It tackles damping, aliasing and high-frequency leakage with ensemble members from learned functional perturbations and mesh-aligned block-sparse attention, which captures long links at linear cost. At 1.5 degree resolution with 214 million parameters, it matched or beat models trained on six times finer grids and produced calibrated ensembles with near-perfect spectral alignment. A 24-member, 10-day forecast took under 12 seconds on one H100 chip.

Catalog record

CREATORMaksim Zhdanov, Ana Lucic, Max Welling, Jan-Willem van de Meent
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95056
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16429

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models Maksim Zhdanov, Ana Lucic, Max Welling, Jan-Willem van de Meent Abstract We introduce Mosaic , a probabilistic weather forecasting model that addresses two principal sources of spectral degradation in ML-based weather prediction: (1) deterministic training against ensemble means and (2) compressive encoding creating an information bottleneck. Mosaic generates ensemble members through learned functional perturbations and operates on native-resolution grids via block-sparse attention, a hardware-aligned mechanism that captures long-range dependencies at linear cost by sharing keys and values across spatially adjacent queries. At 1.5° resolution with 214M parameters, Mosaic matches or outperforms models trained on 6 × \times finer data on headline upper-air variables and achieves state-of-the-art results among 1.5° models, producing well-calibrated ensembles whose individual members exhibit near-perfect spectral alignment across all resolved frequencies. A 24-member, 10-day forecast takes under 12 s on a single H100 GPU. 1 Introduction Accurate weather forecasts save lives and enable timely decisions during extreme events. Phenomena such as frontal zones and tropical cyclones cause catastrophic damage and span relatively short distances, requiring models that faithfully resolve fine spatial scales. Numerical weather prediction (NWP) systems achieve this by integrating the equations of fluid dynamics, thermodynamics, and radiative transfer, but at a computational cost that scales cubically with grid resolution. query local attn. sparse attn. Figure 1 : Block-sparse attention for weather forecasting. Spatially close query tokens (red block over Tampa Bay) collectively attend to both local key-value pairs (blue block over Florida) and dynamically selected, spatially distributed ones (green blocks). Sparse attention enables capturing long-range dependencies in high-resolution weather data, critical for extreme events such as hurricane formation (note the eye visible in the inset). ML-based weather prediction models (MLWPs; Bi et al. 2023 ; Lam et al. 2023 ; Bodnar et al. 2025 ) have emerged as efficient surrogates, generating 10-day forecasts in under 60 seconds on a single GPU, achieving a 1000-10000 \times speedup over NWP ( Buizza et al. 2018 ; Bauer et al. 2020 ) . However, these models struggle to reproduce the power spectra of atmospheric variables, systematically suppressing spectral power at fine scales ( Bonavita 2024 ) . This happens for two main reasons. First, most MLWPs are deterministic and are trained to approximate the conditional mean over future states, which is inherently smoother than any individual realization. Probabilistic MLWPs ( Price et al. 2023 ; Lang et al. 2024 ; Alet et al. 2025 ) address this limitation by producing ensemble members rather than a single mean prediction. Each member represents a plausible realization that can exhibit the sharp, fine-scale structures present in nature with spectral properties substantially closer to ground truth than their deterministic counterparts. {groupplot} \nextgroupplot \addplot \addlegendentry {}^{\text{mean}} \addplot \addlegendentry {}^{\text{1st}} \addplot \addlegendentry {}^{\text{mean}} \addplot \addlegendentry {}^{\text{1st}} \addplot \addlegendentry \addplot \addlegendentry \nextgroupplot 0.25^{\circ} \addplot \addlegendentry \addplot \addlegendentry \addplot \addlegendentry \addplot \addlegendentry {}^{\text{1st}} 10^{-1} 10^{0} 10^{1} 10^{2} 0.45 0.5 0.55 0.6 Stormer ArchesWeather-Mx4 NeuralGCM ArchesWeatherGen Mosaic (Ours) GenCast GraphCast Pangu-Weather Inference time (seconds per member per step) nRMSE (10-day avg.) (c) Forecast skill vs. inference speed Peak GPU memory: \sim 3 GB \sim 10 GB \sim 70 GB Figure 2 : Spectral analysis and efficiency of MLWP models. (a, b) Spectral power ratios (model / reference) of 10-meter wind speed from a single 10-day forecast at 1.5^{\circ} (ERA5) (a) and 0.25^{\circ} (HRES-fc0) (b) resolution. Individual ensemble members of probabilistic models ( Mosaic , ArchesGen , GenCast ) demonstrate stronger spectral coherence with the reference, while ensemble means and deterministic models ( Stormer , ArchesWeather-Mx4 , GraphCast , Aurora , Pangu-Weather ) deviate substantially, with divergence most pronounced at fine spatial scales. (c) Forecast skill vs. inference speed on a single H100 GPU; marker size indicates peak GPU memory. nRMSE is computed at 240 h lead time at 1.5^{\circ} resolution over a set of key variables (see Appendix C.8 for details). Second, most MLWPs use compressive encoding in their model architecture: they compress high-resolution weather data to a coarser mesh before processing, with a coarsening factor that typically far exceeds the increase in the number of feature channels, creating an information bottleneck shown to cause irreversible information loss in both vision ( Wang et al. 2025 ) and weather models ( Nguyen et al. 2023 ) . Although compression-based approaches are efficient, they increase the complexity of the learned function: fine-scale spatial variations originally distributed across neighboring mesh nodes become entangled within channel dimensions of coarse nodes, requiring the model to jointly recover geometric structure and spectral content. In this work, we introduce Mosaic : a probabilistic weather forecasting model designed to address both sources of spectral degradation. First, we follow ( Alet et al. 2025 ) and use learned functional perturbations to incorporate uncertainty, producing ensemble forecasts whose individual members preserve realistic spectral variability. Second, we propose block-sparse attention: a hardware-aligned formulation of native sparse attention ( Yuan et al. 2025 ) that exploits the intrinsic locality of physical data by sharing keys and values across spatially adjacent queries, computing block-to-block interactions. Each block dynamically selects which regions of the grid to attend to, capturing arbitrarily long-range dependencies at linear cost. To ensure efficient memory access, we process data on the HEALPix mesh ( Gorski et al. 1999 ) , which offers contiguous storage and therefore allows us to operate on blocks. Together, these design choices make Mosaic directly applicable to high-resolution grids without compression. The main contributions of this work are: • We propose block-sparse attention, a hardware-aligned sparse attention mechanism that captures long-range dependencies at linear cost, enabling weather models to operate on native-resolution grids without compressive encoding . • We introduce Mosaic , a probabilistic weather forecasting model that uses block-sparse attention and learned functional perturbations to address both the architectural and statistical sources of spectral degradation in MLWPs. • We demonstrate that Mosaic at 1.5° resolution matches or outperforms models trained on 6 \times finer data on headline upper-air variables, produces well-calibrated ensembles with near-perfect spectral alignment, and generates a 24-member, 10-day forecast in under 12 s on a single H100 GPU. 2 Related Work 2.1 Effective Resolution in Weather Forecasting Abdalla et al. 2013 define the effective resolution of a weather prediction model as the smallest spatial scale it fully resolves – the wavelength at which the model’s power spectrum starts to deviate from the ground truth. Multiple studies ( Bonavita 2024 ; Husain et al. 2025 ) show that MLWPs fail to reproduce realistic spectra and therefore exhibit low effective resolution, struggling to resolve sharp phenomena such as frontal zones and tropical cyclones that span 50-80 km, despite being trained on data at 28 km spatial resolution. Gupta et al. 2025 find systematic underestimation of spectral power at mesoscales (10-100 km) across multiple MLWPs. Li et al. 2025 demonstrate that Pangu ( Bi et al. 2023 ) underestimates kinetic energy at wavelengths below 1000 km and fails to replicate the characteristic {-\frac{5}{3}} spectral slope of physics-based models. Several strategies address this spectral degradation. Subich et al. 2025 modify the training objective to penalize spectral discrepancy directly, improving GraphCast’s effective resolution from 1,250 to 160 km. Similarly, Bonev et al. 2025 also optimize in the spectral domain, achieving

Read the complete text