A Scale-Adaptive Framework for Joint Spatiotemporal Super-Resolution with Diffusion Models

Max Defez, Filippo Quarenghi, Mathieu Vrac, Stephan Mandt, Tom Beucler · 2026

Read A Scale-Adaptive Framework for Joint Spatiotemporal Super-Resolution with Diffusion Models by Max Defez, Filippo Quarenghi, Mathieu Vrac, Stephan Mandt, Tom Beucler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21903.txt.

Introduction

Defez and four colleagues present a scale-adaptive system for joint super-resolution with diffusion models. Climate work usually sharpens space or time alone, and joint models serve only one pair of factors. Their system reuses one design across scales. It predicts the mean directly, with attention, then models the leftover with diffusion. An optional mass-conservation step keeps totals. Retuning three settings on retraining makes it work across scales. On French precipitation, it spans 1 to 25 in space and 1 to 6 in time.

Catalog record

CREATORMax Defez, Filippo Quarenghi, Mathieu Vrac, Stephan Mandt, Tom Beucler
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51090
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Scale-Adaptive Framework for Joint Spatiotemporal Super-Resolution with Diffusion Models Max Defez, Filippo Quarenghi, Mathieu Vrac, Stephan Mandt, Tom Beucler Abstract Deep-learning video super-resolution has progressed rapidly, but climate applications typically super-resolve (increase resolution) either space or time, and joint spatiotemporal models are often designed for a single pair of super-resolution (SR) factors (upscaling spatial and temporal ratio between the low-resolution sequence and the high-resolution sequence), limiting transfer across spatial resolutions and temporal cadences (frame rates). We present a scale-adaptive framework that reuses the same architecture across factors by decomposing spatiotemporal SR into a deterministic prediction of the conditional mean, with attention, and a residual conditional diffusion model, with an optional mass-conservation (same precipitation amount in inputs and outputs) transform to preserve aggregated totals. Assuming that larger SR factors primarily increase underdetermination (hence required context and residual uncertainty) rather than changing the conditional-mean structure, scale adaptivity is achieved by retuning three factor-dependent hyperparameters before retraining: the diffusion noise schedule amplitude β \beta (larger for larger factors to increase diversity), the temporal context length L L (set to maintain comparable attention horizons across cadences) and optionally a third, the mass-conservation function f f (tapered to limit the amplification of extremes for large factors). Demonstrated on reanalysis precipitation over France (Coméphore), the same architecture spans super-resolution factors from 1 to 25 in space and 1 to 6 in time, yielding a reusable architecture and tuning recipe for joint spatiotemporal super-resolution across scales. K eywords super-resolution \cdot scale-adaptative \cdot attention \cdot denoising diffusion probabilistic modeling \cdot precipitation 1 Impact statement We propose a ready-to-use diffusion-based video super-resolution architecture that reuses the same model design across spatial and temporal refinement settings, requiring only 3 retuned hyperparameters before retraining. This enables reuse across environmental datasets without redesigning the architecture. Demonstrated on precipitation, a challenging zero-inflated, heavy-tailed variable, the model generates realistic ensembles across scales, supporting climate-impact applications such as downscaling. 2 Introduction: Spatiotemporal Super-Resolution as Conditional Generative Modeling Video Super-Resolution (VSR) aims to recover high-fidelity, high-resolution (HR) frame sequences from their low-resolution (LR) counterparts by exploiting spatiotemporal correlations to resolve details. While data-driven methods, namely deep learning algorithms ranging from deterministic regression models (e.g., transformers, recurrent neural networks) to generative models (e.g., diffusion models), have dominated the state-of-the-art VSR in computer vision, the adoption of data-driven VSR for climate science remains rare ( 11 , 26 , 4 ). A central barrier is scale adaptivity. Most deep-learning VSR models are designed and trained for a single super-resolution setting (e.g., \times 4 ) and a fixed combination of spatial grid and temporal cadence. This rigidity is particularly limiting in atmospheric science, where products from satellites, radar mosaics, gauges, and reanalyses come with heterogeneous resolutions and sampling frequencies. As a result, practitioners often fully redesign model architectures for each new dataset or target resolution, making pipelines problem-specific and computationally costly. We address this gap in architecture design by focusing on the spatiotemporal super-resolution of precipitation data. Precipitation is a particularly challenging testbed due to its high intermittency, non-Gaussian statistics, and complex multiscale interactions ( 19 ). Accurate spatiotemporal VSR in this context is not merely about aesthetic enhancement; it is essential for resolving localized extreme events and temporal interactions for impact models (e.g., flash flood forecasting) as demonstrated by 15 and 2 . Research in precipitation nowcasting provided foundational VSR architectures to model spatiotemporal atmospheric dynamics. Early deep learning approaches, such as the ConvLSTM network by 20 , demonstrated that neural networks modeling sequences could effectively capture the temporal evolution of rain fields. However, these early deterministic methods share a fundamental limitation: by optimizing errors at the pixel level (e.g., Mean Squared Error), they smooth out high-frequency details and lead to a systematic underestimation of extremes. To address this spectral bias, precipitation nowcasting shifted toward generative models capable of modeling the stochasticity inherent in fine-scale weather systems. 9 introduced a foundational recurrent conditional Generative Adversarial Network (cGAN) specifically designed to super-resolve time-evolving atmospheric fields. Building on this, 3 recently demonstrated the utility of cGANs for generating consistently evolving global rain fields. However, despite their ability to recover high-frequency textures, GANs remain prone to training instabilities and mode collapse, often failing to capture the full probability distribution of the data 23 . Recent progress therefore increasingly relies on probabilistic diffusion models. This transition is again paralleled in the nowcasting literature, where 25 introduced DiffCast to separate deterministic motion trends from stochastic residuals. This decomposition, separating bulk advection from fine-scale texture, is equally critical for spatiotemporal super-resolution, as demonstrated by the success of the SpatioTemporal Video Diffusion (STVD) architecture from 22 . Despite these advances, diffusion-based VSR architectures, including STVD, are typically tuned for fixed grids and cadences, limiting transfer across SR factors (S,T) . Our contribution is a modeling principle for architectural reuse: we decompose joint spatiotemporal SR into (i) a deterministic predictor of the conditional mean and (ii) a diffusion model of stochastic residuals. We assume that increasing SR factors mainly increases underdetermination (and thus the required temporal context and residual uncertainty) without fundamentally changing the conditional-mean structure. Accordingly, for each (S,T) we train a separate instance with shared architecture and training procedure (but not shared weights), and retune only two factor-dependent hyperparameters (attention context length and diffusion noise schedule) to calibrate context and residual variance, rather than redesigning the architecture. This yields a model that is approximately scale-adaptive, although each pair still requires independent training and separate weights. 3 Methodology 3.1 Problem Definition and Notation To explore the design space of spatiotemporal SR factors, we consider pairs (S,T)\in{\mathbb{N}^{*}}^{2} , where S and T denote the spatial and temporal SR factors, respectively, and use this sweep to guide hyperparameter choices. Let H and W denote the HR frame height and width, in our case H=W=100 . An HR observation is a nonnegative frame in \mathbb{R}_{+}^{H\times W} . We index frames by time t\in\mathcal{T} and tile l\in\mathcal{L} , and denote the corresponding HR frame by y^{t,l}\in\mathbb{R}_{+}^{H\times W} . In our experiments, we adopt a “perfect-model setting” in which LR frames are a deterministic coarsening of HR frames, computed by averaging over non-overlapping T temporal blocks and S\times S spatial blocks (see Appendix 8 for the formal averaging operator). In practice, this setup is rarely observed, although it can be approximated using preprocessing techniques such as bias correction. The resulting LR frame is denoted x^{t,l}\in\mathbb{R}_{+}^{(H/S)\times(W/S)} . Leveraging temporal autocorrelation, the input is a LR sequence of length L ending at t , i.e., x^{t,l}(L)\equiv\big(x^{t-(L-1)T,l},\,\ldots,\,x^{t,l}\big)\in\big(\mathbb{R}_{+}^{(H/S)\times(W/S)}\big)^{L} . We also condition on a static auxiliary field M_{l} (tile topography), it is the only additional input used in the model. Given an LR sequence ending at time t , our model predicts the corresponding HR block without future information: \big(\hat{y}^{t,l},\ldots,\hat{y}^{t+T-1,l}\big)=\Phi_{S,T}\!\left[x^{t,l}(L),M_{l}\right] , where \Phi_{S,T} is the SR mapping for factors (S,T) ; our goal is to develop a versatile architecture that can learn \Phi_{S,T} for any (S,T) . The overall architecture is identical for all (S,T) , except the 2-3 hyperparameters described in Section 3.3 . However, each (S,T) has its own model instance with specific weights learned from data at the corresponding resolution. 3.2 A Residual Diffusion Architecture for Scale-Adaptive Super-Resolution To motivate a generative approach, consider (S,T)=(25,6) : the model must reconstruct

Read the complete text