FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers
Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Zhenzhe Zhang, Tianchen Zhu, Shanghang Zhang, Jianxin Li · 2025
Read FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers by Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Zhenzhe Zhang, Tianchen Zhu, Shanghang Zhang, Jianxin Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06858.txt.
Introduction
Tianyu Chen and seven co-authors fix a weakness of Fourier neural operators, the workhorse neural solvers for partial differential equations. These models learn in the frequency domain, but high-frequency signals are sparse and usually truncated by a fixed pattern. The lost detail hurts high-resolution inputs and long-term forecasts exactly where accuracy matters.
FreqMoE first learns the low-frequency weights, exploiting the dependence of high frequencies on low ones. A sparse upward-cycling strategy then grows a mixture of experts that extends those weights into high-frequency regions. Across regular and irregular grids it gained up to about seventeen percent accuracy while using a fiftieth of the parameters of a dense model. Long-term prediction stayed stable, and the scheme ports to other operator variants.
Catalog record
| CREATOR | Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Zhenzhe Zhang, Tianchen Zhu, Shanghang Zhang, Jianxin Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Zhenzhe Zhang, Tianchen Zhu, Shanghang Zhang, Jianxin Li Abstract Fourier Neural Operators (FNO) have emerged as promising solutions for efficiently solving partial differential equations (PDEs) by learning infinite-dimensional function mappings through frequency domain transformations. However, the sparsity of high-frequency signals limits computational efficiency for high-dimensional inputs, and fixed-pattern truncation often causes high-frequency signal loss, reducing performance in scenarios such as high-resolution inputs or long-term predictions. To address these challenges, we propose FreqMoE, an efficient and progressive training framework that exploits the dependency of high-frequency signals on low-frequency components. The model first learns low-frequency weights and then applies a sparse upward-cycling strategy to construct a mixture of experts (MoE) in the frequency domain, effectively extending the learned weights to high-frequency regions. Experiments on both regular and irregular grid PDEs demonstrate that FreqMoE achieves up to 16.6% accuracy improvement while using merely 2.1% parameters (47.32 × \times reduction) compared to dense FNO. Furthermore, the approach demonstrates remarkable stability in long-term predictions and generalizes seamlessly to various FNO variants and grid structures, establishing a new “ L ow frequency P retraining, H igh frequency F ine-tuning” paradigm for solving PDEs. 1 1 footnotetext: The corresponding author is Haoyi Zhou ([email protected]). Figure 1: Motivation of FreqMoE. Traditional FNO directly truncates high-frequency components (left), while FreqMoE(ours) efficiently preserves them through sparse dynamic experts (right). This design enables high-frequency modeling with negligible computational overhead. 1 Introduction Efficient solutions to large-scale partial differential equations (PDEs) play a crucial role in numerous scientific computing applications, ranging from weather forecasting through Navier-Stokes equations to quantum simulations in physics Bi et al. 2022 ; Pathak et al. 2022 ; Li et al. 2023b ; Childs et al. 2021 . As spatial resolution and temporal steps increase, traditional numerical solvers face prohibitive computational costs, spurring the development of neural approaches that promise to balance accuracy with efficiency. Among these approaches, Fourier neural operators (FNO Li et al. 2021 and its variants (Geo-FNO Li et al. 2023a , FFNO Tran et al. 2023 , TFNO Kossaifi et al. 2023 ) have emerged as particularly promising, leveraging the inherent sparsity of physical fields in frequency domain. By operating on a compact window of low-frequency signals while truncating higher frequencies, these methods achieve scale-free processing across arbitrary resolutions with reduced computational complexity. However, this frequency truncation presents a fundamental trade-off: while enabling computational efficiency, the loss of high-frequency information can significantly degrade performance in high-resolution scenarios and, as a result, accumulate errors in long-term predictions Lippe et al. 2023 ; Cao et al. 2024 . Recent efforts to overcome high-frequency limitations have explored post-training refinement strategies. These methods aim to recover truncated frequency information through various approaches, such as diffusion-based iterative refinement Lippe et al. 2023 and numerical solver guidance Cao et al. 2024 . However, existing solutions often incur substantial computational overhead or are restricted to specific scenarios, highlighting the need for a more computationally efficient approach. To address these limitations, we propose FreqMoE, a light-weight post-training framework inspired by upcycled MoE Models He et al. 2024 ; Zhang et al. 2024 . This framework enables FNO trained on low-frequency domains to adapt to high-frequency signals dynamically. Our method leverages a pre-trained FNO as a base expert for low-frequency components while initializing specialized high-frequency experts(Fig. 1 ). To handle the inherent sparsity of high-frequency components, we incorporate a gating mechanism to selectively activate the most relevant high-frequency Experts during prediction. Our approach is motivated by a fundamental observation in physical systems: high-frequency signals typically exhibit strong dependencies on low-frequency components, as exemplified by the energy cascade phenomenon in fluid mechanics McKeown et al. 2023 . Capitalizing on this physical insight, we initialize high-frequency experts re-using the base expert’s weights through a LoRA-like strategy Hu et al. 2022 . Specifically, we decompose the high-frequency expert’s weights into two components: a shared base weight R_{\text{base}} and a low-rank delta weight \Delta R . For the i -th high-frequency Expert, its weights are constructed as R_{i}=R+\Delta R_{i} , where the shared base weights R could be intialized by the pre-trained dense FNO. This architecture offers significant computational efficiency improvement through two key design choices: (1) during post-training, low-rank delta weights are parameter-efficient, and (2) during inference, only the Top-K selected experts participate in prediction. The extremely low-rank nature of \Delta R ensures minimal computational overhead in both stages, making our method particularly practical for real-world applications. Through extensive evaluation on both regular and irregular grid PDEs, we demonstrate the effectiveness of FreqMoE in high-resolution and long-term prediction scenarios. Our experiments reveal compelling advantages: in high-resolution tasks (512×512), FreqMoE achieves up to 16.6% accuracy improvement while using merely 2.1% parameters (47.32 \times reduction) compared to conventional FNO. This efficiency extends to unstructured meshes, where FreqMoE maintains superior performance with 27.37 \times parameter reduction. Furthermore, long-term rollout experiments showcase FreqMoE’s stability in mitigating error accumulation, particularly in challenging high-resolution scenarios. The key contributions of this work are threefold: 1. We propose FreqMoE, a lightweight post-training framework that dynamically enhances high-frequency processing capabilities in neural PDE solvers. Our approach generalizes seamlessly across the FNO family on both structured and unstructured grids, establishing an efficient ”low-frequency pretraining, high-frequency fine-tuning” paradigm. 2. Inspired by physical principles of frequency dependencies in PDEs, we develop a LoRA-based expert initialization scheme that efficiently reuses low-frequency weights. This design achieves remarkable parameter efficiency (47.32 \times reduction) while maintaining competitive performance through sparse dynamic computation. 3. Through comprehensive evaluation on diverse PDE systems, we demonstrate that FreqMoE significantly outperforms conventional FNO variants, achieving up to 16.6% accuracy improvement in high-resolution tasks (512×512) and superior stability in long-term predictions, all while maintaining minimal computational overhead. 2 Related Works Figure 2: Methods overview of FreqMoE. (a) The standard Fourier Neural Operator (FNO) architecture consisting of input lifting (P), a sequence of Fourier layers, and output projection (Q). (b) Our modified Fourier layer design with a mixture-of-experts mechanism , where the gating networker dynamically assigns frequency components to specialized experts after FFT decomposition. High-frequency components (lighter shades) are processed by high-frequency experts, while low-frequency components are handled by the base expert. (c) Our expert initialization strategy , where pre-trained weights R are used as a shared base component R_{\text{base}} and expert-specific delta weights \Delta R are initialized with LoRA trick, enabling efficient parameter sharing and specialized frequency processing. 2.1 Fourier Neural Operators Fourier Neural Operators (FNO) Li et al. 2021 have revolutionized PDE solving by introducing FFT-based spectral convolution layers to learn mappings between infinite-dimensional function spaces. This foundational work has sparked numerous architectural innovations: Geo-FNO Li et al. 2023a and SFNO Bonev et al. 2023 extended the framework to handle irregular grids and spherical geometries, while F-FNO Tran et al. 2023 enhanced scalability through separable spectral convolutions and advanced training strategies. T-FNO Kossaifi et al. 2023 further improved parameter efficiency and generalization by implementing global tensor decomposition. Despite these advancements, the issue of high-frequency truncation—a critical limitation in FNO—remains largely unaddressed. Our work directly tackles this gap by enhancing high-frequency signal processing capabilities, offering a complementary approach that integrates seamlessly with existing FNO architectures to further improve performance. 2.2 Sparse Upcycling Techniques