Attention-Enhanced Convolutional Autoencoder and Structured Delay Embeddings for Weather Prediction

Amirpasha Hedayat, Karthik Duraisamy · 2025

Read Attention-Enhanced Convolutional Autoencoder and Structured Delay Embeddings for Weather Prediction by Amirpasha Hedayat, Karthik Duraisamy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12682.txt.

Introduction

Amirpasha Hedayat and Karthik Duraisamy build a lean framework for short-range weather prediction. Weather is a chaotic, high-dimensional system, and recent AI forecasters need heavy computing to handle it. Their approach first compresses weather fields with a convolutional autoencoder strengthened by attention blocks. It then learns a simple linear operator in a time-delayed version of that compressed space. The dynamics are advanced by cheap linear steps rather than by a large neural network at every moment.

On the ERA5 reanalysis dataset, the system predicted weather patterns inside its training period well. It struggled more when pushed into future periods beyond the training window. Their analysis points to projection error, the loss from compressing the weather state, as the main bottleneck rather than the step forward itself. They suggest pairing such models with larger AI systems where budgets are tight.

Catalog record

CREATORAmirpasha Hedayat, Karthik Duraisamy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12682

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention-Enhanced Convolutional Autoencoder and Structured Delay Embeddings for Weather Prediction Amirpasha Hedayat, Karthik Duraisamy Abstract Weather prediction is a quintessential problem involving the forecasting of a complex, nonlinear, and chaotic high-dimensional dynamical system. This work introduces an efficient reduced-order modeling (ROM) framework for short-range weather prediction and investigates fundamental questions in dimensionality reduction and reduced order modeling of such systems. Unlike recent AI-driven models, which require extensive computational resources, our framework prioritizes efficiency while achieving reasonable accuracy. Specifically, a ResNet-based convolutional autoencoder augmented by block attention modules is developed to reduce the dimensionality of high-dimensional weather data. Subsequently, a linear operator is learned in the time-delayed embedding of the latent space to efficiently capture the dynamics. Using the ERA5 reanalysis dataset, we demonstrate that this framework performs well in-distribution as evidenced by effectively predicting weather patterns within training data periods. We also identify important limitations in generalizing to future states, particularly in maintaining prediction accuracy beyond the training window. Our analysis reveals that weather systems exhibit strong temporal correlations that can be effectively captured through linear operations in an appropriately constructed embedding space, and that projection error rather than inference error is the main bottleneck. These findings shed light on some key challenges in reduced-order modeling of chaotic systems and point toward opportunities for hybrid approaches that combine efficient reduced-order models as baselines with more sophisticated AI architectures, particularly for applications in long-term climate modeling where computational efficiency is paramount. \coltauthor and Department of Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA. † † year: 2025 † † proceedings: : Preprint. keywords Model Reduction, Autoencoders, Attention Mechanisms, Weather Prediction 1 Introduction Weather and climate prediction has long been recognized as a grand challenge in science and engineering. Atmospheric dynamics is governed by complex, chaotic, nonlinear partial differential equations (PDEs) involving coupled processes across vast spatial and temporal scales. Accurate forecasting necessitates high-resolution models that capture fine-scale phenomena, leading to significant computational costs in traditional numerical weather prediction (NWP) methods ( Bauer2015 ) . Recently, AI models for weather prediction, such as FourCastNet ( pathak2022fourcastnet ) , GraphCast ( lam2022graphcast ) , and ClimaX ( nguyen2023climax ) , have demonstrated good predictive capabilities by leveraging deep neural networks within architectures such as transformers and neural operators to learn spatiotemporal patterns from large datasets. However, these models require extensive computational resources for development, often involving thousands of GPU hours and large-scale parallel computing infrastructures. Reduced Order Modeling (ROM) techniques aim to simplify high-dimensional systems by projecting them onto a lower-dimensional manifold while preserving essential dynamics. By combining attention-enhanced Convolutional Autoencoders (CAEs) for dimensionality reduction with simple time delayed linear models for evolution, we develop a framework that captures the general patterns and dynamics of weather systems with significantly reduced computational costs. Motivated by the need for computationally efficient yet reasonably accurate forecasting methods, this paper takes a different approach, ROMs, to investigate fundamental questions about weather prediction and dimensionality reduction. While previous studies have focused primarily on improving prediction accuracy, we aim to understand: a) What are the key limiting factors in compressed representations of atmospheric states? b) How effectively can temporal correlations in weather patterns be captured through linear operations in appropriate embedding spaces? c) What role does the quality of dimensional reduction play versus the sophistication of the prediction mechanism? d) How well do traditional error metrics align with meteorologically significant features? Our approach is also motivated by the broader need to develop computationally efficient baseline models that can complement existing state-of-the-art AI systems. By establishing and analyzing such baselines, we can better understand the potential for hybrid approaches that combine the efficiency of reduced-order models with the accuracy of more complex AI architectures. This understanding becomes particularly crucial when considering applications to long-term climate modeling, where computational efficiency remains a critical constraint. The remainder of the paper is organized as follows: Section 2 reviews related work in AI-based weather prediction and reduced-order modeling. Section 3 describes the proposed methodology, including detailed formulations of CBAM and the time-delayed linear model. Section presents experimental results and analyses, discussing the limitations and key insights. Section concludes the paper with the main takeaways and future research directions. 2 Related Work The integration of machine learning (ML) and artificial intelligence (AI) into weather prediction has led to significant advancements in forecasting accuracy and computational efficiency ( reichstein2019deep ) . One of the notable AI-based weather models is FourCastNet, which employs a Fourier-based neural network architecture to efficiently capture global weather patterns by leveraging spectral convolutions adept at handling the periodic nature of atmospheric data ( pathak2022fourcastnet ) . GraphCast utilizes graph neural networks (GNNs) to model spatial dependencies inherent in weather systems, effectively capturing localized interactions and long-range dependencies ( lam2022graphcast ) . ClimaX and its newer version, Aurora, adopt a Transformer-based architecture to build a weather foundation model, excelling in modeling long-term dependencies and complex spatiotemporal relationships within weather data ( nguyen2023climax ; aurora ) . Despite their impressive performance, these AI-driven models often require substantial computational resources. This high computational demand poses challenges for scalability and real-time application, especially in resource-constrained environments. Reduced-order modeling (ROM) aims to simplify high-dimensional dynamical systems by projecting them onto a lower-dimensional manifold while preserving essential dynamical characteristics. Traditional intrusive, projection-based ROM techniques, such as Galerkin and Petrov-Galerkin methods, have been widely used in various fields, including fluid dynamics and structural engineering ( Carlberg2011 ; Benner2015 ; HUANG2022110742 ) . Data-driven approaches including Operator Inference ( Peherstorfer2016 ; pan2018data ; BENNER2020113433 ; GEELEN2023115717 ; Kramer2024 ) have gained prominence for their ability to build efficient models in the absence of full-order models. Convolutional Autoencoders (CAEs) have been employed for dimensionality reduction in high-dimensional datasets ( hinton2006reducing ; masci2011stacked ; Mao2016 ; Theis2017 ; xu2020multi ; pan2020physics ) . By leveraging convolutional layers, CAEs can effectively capture spatial hierarchies and local dependencies, making them suitable for applications involving spatially structured data such as weather fields. These structures can be further improved by adding attention layers in the network. Attention mechanisms have revolutionized the field of deep learning by enabling models to focus on the most relevant parts of the input data, improving performance in various tasks. The concept was popularized in natural language processing (NLP) with the introduction of the Transformer architecture ( vaswani2017attention ) , which relies entirely on self-attention mechanisms to capture dependencies between input elements. In computer vision, attention mechanisms have been incorporated to enhance feature representation in convolutional neural networks (CNNs). The Squeeze-and-Excitation (SE) network ( hu2018squeeze ) introduced channel-wise attention by adaptively recalibrating channel-wise feature responses. Building upon this, the Convolutional Block Attention Module (CBAM) ( woo2018cbam ) sequentially applies both channel and spatial attention, allowing the network to focus on what and where to emphasize or suppress. Our work builds upon these foundations by developing a ResNet-based ( he2016deep ) CAE enhanced with CBAM to achieve effective dimensionality reduction of high-dimensional weather data. By inferring a time-delay-enhanced linear model from the encoded space, we aim to evaluate the model’s ability to learn the dynamics within the training data and assess its generalization capabilities to unseen future states. 3 Methodology This section outlines the proposed ROM framework for weather prediction. The framework

Read the complete text