Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems

Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla · 2026

Read Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems by Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01486.txt.

Introduction

Xuesong Wang and co-authors propose multi-scale wavelet transformers for dynamical operator learning. Data-driven surrogates run faster than solvers, yet neural operators show spectral bias, damping high-frequency parts carrying small-scale structure, which can destabilize long-horizon weather forecasting. Transformers learn dynamics in a tokenized wavelet domain splitting low- and high-frequency content across scales. Wavelet-preserving downsampling keeps high-frequency features, while wavelet attention tracks dependencies across scales. Tests on chaotic systems show error cuts and better spectral fidelity, and ERA5 tests cut climate bias.

Catalog record

CREATORXuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61947
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01486

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla Abstract Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long-horizon spectral fidelity. On the ERA5 climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world forecasting setting. 1 Introduction High-fidelity numerical solvers remain the primary tools for modeling dynamical systems governed by Partial Differential Equations (PDEs), but can be prohibitively expensive in many-query workloads (e.g., design loops and uncertainty quantification), motivating data-driven surrogates that learn simulator input-output maps and enable orders-of-magnitude faster inference once trained ( 23 ; 20 ) . Among these, neural operators learn a map between function spaces that can be discretization-invariant, which allows training on one discretization and evaluation on others, as exemplified by DeepONet and the Fourier Neural Operator ( 29 ; 23 ) . However, these advantages do not come without limitations. A key failure mode in neural operators is spectral bias, sometimes referred as oversmoothing, due to the tendency of neural networks to learn low-frequency components faster and more reliably than high-frequency components ( 34 ; 4 ; 41 ) . In the context of neural operators, spectral bias manifests as systematic under/over-estimation of high-wavenumber content, which is particularly harmful for multi-scale and chaotic systems where small-scale errors can rapidly degrade large-scale predictions ( 5 ) . The cause of spectral bias can be mainly attributed to two aspects: architectural bottlenecks that limit high-frequency feature representation such as Fourier-mode truncation ( 19 ) ; and standard training objectives such as MSE-type losses that can overweight low-frequency energy and encourage smooth solutions ( 38 ) . Existing work addresses spectral bias in neural operators through three primary directions. (i) Multi-scale architectures ( 35 ; 27 ; 36 ) that aim to propagate information across resolutions and recover fine-scale structures. While effective in many regimes, multi-scale feature extraction alone does not guarantee that high-frequency content is preserved when downsampling is aggressive. (ii) Frequency-aware representations ( 16 ; 31 ; 9 ) . Some approaches explicitly separate low/high-frequency components via either heuristics or more principled Wavelet transforms. However, they often provide only coarse frequency separation or resort to less expressive operators such as convolutional networks ( 39 ) . Recent transformer-based advances ( 45 ) exploit attention for modelling wavelet feature dependencies but are directly applied on the physical grid which can be computationally expensive for high-resolutions. (iii) Objective functions and rollout regularization ( 27 ; 15 ; 26 ; 24 ; 13 ) . Several works alter training objectives to emphasize small-scale errors or improve long-term stability. Nonetheless, these approaches are largely complementary to architectural solutions, and diffusion-based methods can incur additional inference-time costs due to multi-step sampling. In this work, we focus on proposing an architecture that enhances the frequency-aware presentation in a multi-scale manner, trained with the standard relative \ell_{2} (MSE-type) training objective, to isolate the role of representation and inductive bias. The resulting method, Multi-Scale Wavelet Transformer (MSWT) operates in a wavelet space underlying a physical dynamical system. The design has three components: (i) a lightweight patch tokenizer mapping the physical field to tokens, reducing the effective resolution and improving efficiency; (ii) a multi-scale wavelet down/up-sampling scheme in token space that improves efficiency and explicitly preserves low-frequency and high-frequency sub-bands across scales; and (iii) a wavelet-based attention that models dependencies across wavelet sub-bands. Our main contributions are: 1. Mitigating spectral bias via wavelet attention. We introduce a wavelet-based transformer architecture that models multiband wavelet content across scales, resulting in improved spectral fidelity. 2. Improved efficiency and long-horizon stability. By combining tokenization with wavelet-based down/up sampling and an efficient wavelet attention mechanism, we reduce attention cost significantly while improving stability and accuracy in long rollouts of chaotic dynamical systems. 3. Long-horizon evaluation on ERA5 with climatology metrics. We validate on the ERA5 atmospheric reanalysis of the global climate ( 14 ) and demonstrate reduced climatological bias in free-running forecasts, indicating improved long-term physical consistency. 2 Related Work Neural operators. Neural operators have emerged as a general paradigm for surrogate modeling of PDEs and dynamical systems ( 29 ; 20 ) . A widely used backbone is the Fourier Neural Operator (FNO), which performs global mixing of Fourier features through truncated Fourier representations ( 23 ) . More recently, transformer-style token mixing has been adapted to operator learning, including attention-based and tokenized formulations ( 3 ; 12 ; 42 ; 21 ; 11 ) . Continuous or geometry-aware variants further extend attention-based operators beyond regular grids ( 40 ; 42 ) . These developments motivate architectures that combine the expressiveness of attention with inductive biases suited for multi-scale physics. Our MSWT follows this direction by moving attention into a wavelet space representation. Understanding and mitigating spectral bias. Recent works have investigated why operator learners misrepresent fine scales and how this impacts long-horizon rollouts. Learning-dynamics analyses based on the frequency principle argue that gradient-based optimization tends to fit coarse structures first, slowing convergence of fine-scale components and inducing oversmoothing in multi-scale PDEs ( 43 ) . Related perspectives such as eigenvector bias show that feature parameterizations can preferentially align with certain spectral components, making multi-scale PDEs challenging even with Fourier features ( 41 ) . For neural operators, spectral studies of FNO-style models report systematic difficulty in learning non-dominant (often higher-wavenumber) modes ( 33 ) , and mean-field expressivity results identify Fourier-mode truncation as an explicit bandwidth bottleneck that limits representable fine-scale content ( 19 ) . Mitigation strategies include multi-scale architectures that improve cross-resolution information flow ( 35 ; 27 ; 36 ) and frequency-aware reweighting or scaling heuristics that amplify high-frequency error signals ( 16 ) . A complementary direction learns dynamics in compressed or latent representations, where reduced-order structure can alleviate error accumulation and improve long-horizon stability and spectral behavior ( 18 ; 22 ) . This motivates our use of a lightweight patch tokenizer to embed high-dimensional fields into a token space for attention. Wavelet transforms for mitigating spectral bias. Wavelet transforms provide a principled representation that extracts global frequency signals while retaining localized spatial information ( 31 ) . This is particularly attractive for operator learning because high-frequency components are often localized (sharp gradients), and a wavelet basis can preserve these structures more naturally than global Fourier bases. Wavelet Neural Operators incorporate wavelet transforms into operator layers and demonstrate improved multi-scale representation compared to purely Fourier-based mixing ( 39 ) . But the expressiveness can be limited by architectures

Read the complete text