STLDM: Spatio-Temporal Latent Diffusion Model for Precipitation Nowcasting

Shi Quan Foo [email protected], Chi-Ho Wong [email protected], Zhihan Gao [email protected], Dit-Yan Yeung [email protected], Ka-Hing Wong [email protected], Wai-Kin Wong [email protected] · 2025

Read STLDM: Spatio-Temporal Latent Diffusion Model for Precipitation Nowcasting by Shi Quan Foo [email protected], Chi-Ho Wong [email protected], Zhihan Gao [email protected], Dit-Yan Yeung [email protected], Ka-Hing Wong [email protected], Wai-Kin Wong [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21118.txt.

Introduction

Shi Quan Foo, Dit-Yan Yeung, and four co-authors split rain forecasting into the part that is predictable and the part that is not. Deterministic nowcasters blur because the atmosphere's randomness gets averaged away; purely generative models draw sharp storms in the wrong places. STLDM assigns each failure to a different component. A conditioning network makes the deterministic forecast. A latent diffusion model, trained end to end with its autoencoder and the conditioning network, enhances that forecast with realistic fine structure.

On multiple radar datasets the hybrid beats the state of the art while also improving inference speed. Minutes-ahead rainfall maps drive flood warnings and emergency response, so sharpness and accuracy arriving together, rather than traded against each other, has direct public-safety value.

Catalog record

CREATORShi Quan Foo [email protected], Chi-Ho Wong [email protected], Zhihan Gao [email protected], Dit-Yan Yeung [email protected], Ka-Hing Wong [email protected], Wai-Kin Wong [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21118

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STLDM: Spatio-Temporal Latent Diffusion Model for Precipitation Nowcasting Shi Quan Foo [email protected], Chi-Ho Wong [email protected], Zhihan Gao [email protected], Dit-Yan Yeung [email protected], Ka-Hing Wong [email protected], Wai-Kin Wong [email protected] Abstract Precipitation nowcasting is a critical spatio-temporal prediction task for society to prevent severe damage owing to extreme weather events. Despite the advances in this field, the complex and stochastic nature of this task still poses challenges to existing approaches. Specifically, deterministic models tend to produce blurry predictions while generative models often struggle with poor accuracy. In this paper, we present a simple yet effective model architecture termed STLDM , a diffusion-based model that learns the latent representation from end to end alongside both the Variational Autoencoder and the conditioning network. STLDM decomposes this task into two stages: a deterministic forecasting stage handled by the conditioning network, and an enhancement stage performed by the latent diffusion model. Experimental results on multiple radar datasets demonstrate that STLDM achieves superior performance compared to the state of the art, while also improving inference efficiency. The code is available in https://github.com/sqfoo/stldm_official . 1 Introduction Precipitation nowcasting is a short-term prediction task for precipitation events over a specific region, based on weather data such as radar and satellite observations. Accurate and timely nowcasting is crucial to society, as it enables us to take preventive actions for mitigating potential economic loss and other adverse impacts due to extreme weather. Traditionally, meteorologists utilized algorithmic methods such as optical-flow methods ( 21 ) and the guidance from numerical weather prediction (NWP) models on this nowcasting task. With the emergence of deep learning, data-driven models have been extensively explored for modeling the spatio-temporal patterns of precipitation events. Despite their lack of interpretability, these deep learning approaches often outperform traditional methods in terms of accuracy and efficiency. Broadly, these approaches can be categorized into two main research categories: video prediction ( 27 ; 28 ; 10 ) , which models the 4D spatio-temporal trend with a ground truth observation for performance evaluation; and video generation ( 48 ; 18 ; 9 ) , which adopts generative models to synthesize the target data distributions with less consideration to the alignment to the ground truth and more emphasis on the visual fidelity. Figure 1 : This demonstrates that deterministic models result in blurry predictions while generative models suffer from the issue of inaccurate predictions. Our proposed STLDM is capable of forecasting accurate predictions while maintaining a nice appearance. Recent works highlight the challenges posed by the stochastic nature of precipitation nowcasting due to the inherent unpredictability of open systems. Deterministic models, in particular, tend to capture the global motion trend well with the missing of details at the micro-level, resulting in blurry predictions over longer lead times. This leads to the difficulty in practical forecasting operations ( 23 ) . On the other hand, generative models are capable of modeling micro-level weather phenomena through simulating the data distribution, which tolerates the nature of stochasticity, thereby producing realistic and sharp forecasts ( 48 ; 18 ; 9 ) . However, they often suffer from low accuracy in predicting large-scale weather events. In summary, both these approaches could only achieve either high accuracy (deterministic models) or high visual quality (generative models) as shown in Figure 1 . In this paper, we first re-formulate the precipitation nowcasting task into two sequential subtasks: Forecasting and Enhancement based on the observations above. First, a Translator with the same architecture as deterministic models is implemented to perform the forecasting task, i.e., roughly forecast the upcoming precipitation events, \overline{Y} . The objective here is to capture the approximate global motion trend of the future. Then, a diffusion model is used to refine the first estimation by the Translator, \overline{Y} , by introducing it as the conditional variable such that the global motion trend of generated samples is constrained well. To accelerate the inference process, we employ this sampling process in the latent space. Therefore, we propose and present a novel and simple Spatio-Temporal Latent Diffusion Model – STLDM for these re-formulated objectives, effectively handling the stochasticity in precipitation nowcasting. STLDM consists of three modules: a Variational AutoEncoder, a Translator (aka Conditioning Network), and a Latent Denoising Network. Furthermore, we train STLDM in a manner from end to end to encourage cooperation among all modules. The experimental results show that STLDM is capable of achieving the state-of-the-art performance on both pixel accuracy and visual fidelity, outperforming most diffusion-based models. The contributions of this work are summarized as follows: • We re-formulate the precipitation nowcasting task into: Forecasting and Enhancement in sequence. Based on this, we propose STLDM, a simple yet efficient model utilizing the Latent Diffusion Model. • To the best of our knowledge, this is the first work that trains a Latent Denoising Network alongside a Variational AutoEncoder and a Conditioning Network for the precipitation nowcasting task. • Our STLDM achieves state-of-the-art performance on multiple real-life radar echo datasets across most evaluation metrics while offering faster sampling speeds compared to other diffusion-based models. 2 Related Works 2.1 Precipitation Nowcasting as a Spatio-Temporal Task Precipitation nowcasting is commonly interpreted as a spatio-temporal predictive task to predict the next N output frames, \hat{Y}_{1:N} , given M input frames, X_{1:M} . It is formulated as: \argmax_{\hat{Y}_{1},...,\hat{Y}_{N}}p(\hat{Y}_{1},...,\hat{Y}_{N}\mid X_{1},...,X_{M}) (1) Based on this formulation, various deep learning models that leverage spatio-temporal features have been proposed. ConvLSTM ( 27 ) , which integrates convolution layers into LSTM cells in an encoder-forecaster architecture, is the first method proposed for this task. Later, PredRNN ( 41 ) introduced a novel structure, the ST-LSTM unit, to extract spatio-temporal features and model future frames in a zigzag memory flow. Building on this foundation, several modifications have been proposed, including Memory In Memory (MIM) ( 43 ) , the gradient highway ( 40 ) , and reversed scheduled sampling ( 42 ) . Following the success of Transformers and their attention mechanisms in natural language processing (NLP) tasks ( 36 ) and vision processing tasks ( 6 ) , several Transformer-based models have been deployed to capture the long-term spatio-temporal features of this nowcasting task. For instance, Cuboid attention in Earthformer ( 10 ) and Feature Extraction Balance Module in Rainformer ( 1 ) are proposed to extract and model both the global and local rainfall features. In parallel, CNN-based models have also been explored for spatio-temporal modeling, like SimVP ( 8 ) and TAU ( 31 ) . Both works adopt a U-Net-like structure, i.e., Encoder-Translator-Decoder with Spatial Encoder and Decoder. SimVP and TAU introduced Inception modules and Temporal Attention Unit as translators, respectively, to learn the temporal evolution. These works are primarily composed of convolution operations, achieving efficiency and remarkable performance. However, neither of these works can produce sharp predictions for long lead times due to the stochastic nature of this task. To address this issue, several loss functions were proposed as alternatives to the conventional L2 loss, such as SSL ( 3 ) and FACL ( 44 ) . Meanwhile, probabilistic models have been employed to capture the spatio-temporal features by estimating the conditional distribution of the future frames, thereby enabling ensemble predictions with high visual fidelity, such as GANs ( 23 ; 2 ; 48 ) and variational autoencoders

Read the complete text