Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems
Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla · 2026
Read Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems by Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01486.txt.
Introduction
Xuesong Wang and co-authors propose multi-scale wavelet transformers for dynamical operator learning. Data-driven surrogates run faster than solvers, yet neural operators show spectral bias, damping high-frequency parts carrying small-scale structure, which can destabilize long-horizon weather forecasting. Transformers learn dynamics in a tokenized wavelet domain splitting low- and high-frequency content across scales. Wavelet-preserving downsampling keeps high-frequency features, while wavelet attention tracks dependencies across scales. Tests on chaotic systems show error cuts and better spectral fidelity, and ERA5 tests cut climate bias.
Catalog record
| CREATOR | Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61947 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.01486
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O’Kane, Edwin V. Bonilla Abstract Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long-horizon spectral fidelity. On the ERA5 climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world forecasting setting. 1 Introduction High-fidelity numerical solvers remain the primary tools for modeling dynamical systems governed by Partial Differential Equations (PDEs), but can be prohibitively expensive in many-query workloads (e.g., design loops and uncertainty quantification), motivating data-driven surrogates that learn simulator input-output maps and enable orders-of-magnitude faster inference once trained ( 23 ; 20 ) . Among these, neural operators learn a map between function spaces that can be discretization-invariant, which allows training on one discretization and evaluation on others, as exemplified by DeepONet and the Fourier Neural Operator ( 29 ; 23 ) . However, these advantages do not come without limitations. A key failure mode in neural operators is spectral bias, sometimes referred as oversmoothing, due to the tendency of neural networks to learn low-frequency components faster and more reliably than high-frequency components ( 34 ; 4 ; 41 ) . In the context of neural operators, spectral bias manifests as systematic under/over-estimation of high-wavenumber content, which is particularly harmful for multi-scale and chaotic systems where small-scale errors can rapidly degrade large-scale predictions ( 5 ) . The cause of spectral bias can be mainly attributed to two aspects: architectural bottlenecks that limit high-frequency feature representation such as Fourier-mode truncation ( 19 ) ; and standard training objectives such as MSE-type losses that can overweight low-frequency energy and encourage smooth solutions ( 38 ) . Existing work addresses spectral bias in neural operators through three primary directions. (i) Multi-scale architectures ( 35 ; 27 ; 36 ) that aim to propagate information across resolutions and recover fine-scale structures. While effective in many regimes, multi-scale feature extraction alone does not guarantee that high-frequency content is preserved when downsampling is aggressive. (ii) Frequency-aware representations ( 16 ; 31 ; 9 ) . Some approaches explicitly separate low/high-frequency components via either heuristics or more principled Wavelet transforms. However, they often provide only coarse frequency separation or resort to less expressive operators such as convolutional networks ( 39 ) . Recent transformer-based advances ( 45 ) exploit attention for modelling wavelet feature dependencies but are directly applied on the physical grid which can be computationally expensive for high-resolutions. (iii) Objective functions and rollout regularization ( 27 ; 15 ; 26 ; 24 ; 13 ) . Several works alter training objectives to emphasize small-scale errors or improve long-term stability. Nonetheless, these approaches are largely complementary to architectural solutions, and diffusion-based methods can incur additional inference-time costs due to multi-step sampling. In this work, we focus on proposing an architecture that enhances the frequency-aware presentation in a multi-scale manner, trained with the standard relative \ell_{2} (MSE-type) training objective, to isolate the role of representation and inductive bias. The resulting method, Multi-Scale Wavelet Transformer (MSWT) operates in a wavelet space underlying a physical dynamical system. The design has three components: (i) a lightweight patch tokenizer mapping the physical field to tokens, reducing the effective resolution and improving efficiency; (ii) a multi-scale wavelet down/up-sampling scheme in token space that improves efficiency and explicitly preserves low-frequency and high-frequency sub-bands across scales; and (iii) a wavelet-based attention that models dependencies across wavelet sub-bands. Our main contributions are: 1. Mitigating spectral bias via wavelet attention. We introduce a wavelet-based transformer architecture that models multiband wavelet content across scales, resulting in improved spectral fidelity. 2. Improved efficiency and long-horizon stability. By combining tokenization with wavelet-based down/up sampling and an efficient wavelet attention mechanism, we reduce attention cost significantly while improving stability and accuracy in long rollouts of chaotic dynamical systems. 3. Long-horizon evaluation on ERA5 with climatology metrics. We validate on the ERA5 atmospheric reanalysis of the global climate ( 14 ) and demonstrate reduced climatological bias in free-running forecasts, indicating improved long-term physical consistency. 2 Related Work Neural operators. Neural operators have emerged as a general paradigm for surrogate modeling of PDEs and dynamical systems ( 29 ; 20 ) . A widely used backbone is the Fourier Neural Operator (FNO), which performs global mixing of Fourier features through truncated Fourier representations ( 23 ) . More recently, transformer-style token mixing has been adapted to operator learning, including attention-based and tokenized formulations ( 3 ; 12 ; 42 ; 21 ; 11 ) . Continuous or geometry-aware variants further extend attention-based operators beyond regular grids ( 40 ; 42 ) . These developments motivate architectures that combine the expressiveness of attention with inductive biases suited for multi-scale physics. Our MSWT follows this direction by moving attention into a wavelet space representation. Understanding and mitigating spectral bias. Recent works have investigated why operator learners misrepresent fine scales and how this impacts long-horizon rollouts. Learning-dynamics analyses based on the frequency principle argue that gradient-based optimization tends to fit coarse structures first, slowing convergence of fine-scale components and inducing oversmoothing in multi-scale PDEs ( 43 ) . Related perspectives such as eigenvector bias show that feature parameterizations can preferentially align with certain spectral components, making multi-scale PDEs challenging even with Fourier features ( 41 ) . For neural operators, spectral studies of FNO-style models report systematic difficulty in learning non-dominant (often higher-wavenumber) modes ( 33 ) , and mean-field expressivity results identify Fourier-mode truncation as an explicit bandwidth bottleneck that limits representable fine-scale content ( 19 ) . Mitigation strategies include multi-scale architectures that improve cross-resolution information flow ( 35 ; 27 ; 36 ) and frequency-aware reweighting or scaling heuristics that amplify high-frequency error signals ( 16 ) . A complementary direction learns dynamics in compressed or latent representations, where reduced-order structure can alleviate error accumulation and improve long-horizon stability and spectral behavior ( 18 ; 22 ) . This motivates our use of a lightweight patch tokenizer to embed high-dimensional fields into a token space for attention. Wavelet transforms for mitigating spectral bias. Wavelet transforms provide a principled representation that extracts global frequency signals while retaining localized spatial information ( 31 ) . This is particularly attractive for operator learning because high-frequency components are often localized (sharp gradients), and a wavelet basis can preserve these structures more naturally than global Fourier bases. Wavelet Neural Operators incorporate wavelet transforms into operator layers and demonstrate improved multi-scale representation compared to purely Fourier-based mixing ( 39 ) . But the expressiveness can be limited by architectures