Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators

Shanda Li, Shinjae Yoo, Yiming Yang · 2025

Read Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators by Shanda Li, Shinjae Yoo, Yiming Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19396.txt.

Introduction

Fourier Neural Operators solve hard equations by working in frequency space, but growing them means adding Fourier modes, and retuning each big model costs a fortune. Li, Yoo, and Yang adapt a scaling scheme called maximal update parametrization to these operators. The math shows how the best tuning settings found on a small model carry over to much larger ones with more modes, with no fresh search.

Tests across several equations back the derivation: settings tuned small still performed at the billion-parameter scale, matching or beating hand-tuned accuracy. The authors say the transfer cuts the tuning bill for large equation solvers while keeping quality up.

Catalog record

CREATORShanda Li, Shinjae Yoo, Yiming Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators Shanda Li, Shinjae Yoo, Yiming Yang Abstract Fourier Neural Operators (FNOs) offer a principled approach for solving complex partial differential equations (PDEs). However, scaling them to handle more complex PDEs requires increasing the number of Fourier modes, which significantly expands the number of model parameters and makes hyperparameter tuning computationally impractical. To address this, we introduce μ \mu Transfer-FNO , a zero-shot hyperparameter transfer technique that enables optimal configurations, tuned on smaller FNOs, to be directly applied to billion-parameter FNOs without additional tuning. Building on the Maximal Update Parametrization ( μ \mu P) framework, we mathematically derive a parametrization scheme that facilitates the transfer of optimal hyperparameters across models with different numbers of Fourier modes in FNOs, which is validated through extensive experiments on various PDEs. Our empirical study shows that μ \mu Transfer-FNO reduces computational cost for tuning hyperparameters on large FNOs while maintaining or improving accuracy. 1 Introduction With the explosive success of deep learning, Fourier Neural Operators (FNOs) ( Li et al. 2021 ) have emerged as a promising paradigm for solving complex PDEs by learning mappings between function spaces. The kernel integral operator is the most crucial component in FNO. The operator first transforms the input function into the spectral domain and then updates the features for the low-frequency components while zeros out the high-frequency ones. The capacity of FNOs is directly tied to the number of Fourier modes K used in the network. It is desirable to use a large K for modeling complex PDE dynamics and capturing fine-grained spatio-temporal structures. However, it can be expensive to scale up the number of Fourier modes K in FNO. For a d -dimensional PDE, the parameter count of the kernel integral operator scales as \mathcal{O}(K^{d}) , leading to substantial computational requirements. For instance, even for a 4-layer FNO with d=3 , increasing K from 3 to 24 expands the model from 1.7M to 906M parameters, with the Fourier integral operator accounting for over 98% of the total parameters. 1 1 1 Detailed model configurations can be found in Section 4.1 . As the parameter count of FNOs grows, hyperparameter tuning—already a computationally expensive process—becomes infeasible at such scales. Standard approaches involve tuning hyperparameters directly on large models, which demands extensive computational resources and is often prohibitive in practice. This bottleneck motivates our central research question: Can we efficiently tune hyperparameters for FNOs with large numbers of Fourier modes? Figure 1 : Loss values of FNO-3D on the incompressible Navier-Stokes Equation , with varying numbers of Fourier modes K and different learning rates. The left and right panels correspond to models trained with standard techniques and \mu Transfer-FNO, respectively. The star on each curve marks the optimal learning rate which leads to the lowest loss value. Under standard parametrization, the optimal learning rates shifts as model size increases; while \mu Transfer ensures a stable optimal learning rate when the model size scales from 1.7M to near 1B. One straightforward idea is search for the optimal set of hyperparameters on small FNOs and apply it to large ones. However, naively transferring the hyperparameter across scales can lead to sub-optimal performance. On the left panel of Figure 1 , we show that the optimal hyperparameter changes as model scales under the standard training recipe. In this work, we address this challenge by developing a zero-shot hyperparameter transfer technique tailored for FNOs, \mu Transfer-FNO . The key insight is to properly scale the kernel integral operator parameters as we increase the number of Fourier modes K , so that the optimal hyperparameter configuration remain relatively stable across model sizes (e.g., the right panel of Figure 1 ). This insight suggests that we can identify good hyperparameters using a small proxy model and transfer them to much larger FNOs without additional tuning. Our approach builds on the Maximal Update Parametrization ( \mu P) framework ( Yang & Hu 2021 ) . Informally, parametrization refers to the scaling of initialization variances and learning rates of model parameters when the model size increases. The Maximal Update Parametrization ( \mu P) is a unique parametrization with many desirable theoretical and empirical properties which can be rigorously defined and mathematically derived. In particular, prior works show that \mu P enables hyperparameter transfer under model width/depth scaling for MLP and Transformers ( Yang et al. 2022 ; Yang et al. 2024 ) . However, whether \mu P for neural operators exists and whether it can empirically enable hyperparameter transfer remain unknown. We derive the first \mu P formulation for FNO under the scaling of the Fourier mode size K . This extension is technically non-trivial due to the unique nature of FNOs: (1) Unlike MLPs and Transformers that process finite-dimensional vectors/sequences, FNOs operate on continuous functions and is agnostic to the discretization; (2) The working mechanism of the kernel integral operator is significantly different from and more complicated than the common deep learning modules such as linear layers or embedding weights analyzed in prior works. Our analysis leads to the following result: Theorem 1.1 (Informal version of Theorem 3.5 ) . The \mu P for FNO scales the initialization variances of the kernel integral parameters by \Theta\left(\frac{1}{d\log K}\right) and their learning rates by \Theta\left(\frac{1}{\sqrt{d\log K}}\right) , where d and K denotes the PDE dimensionality and number of Fourier modes, respectively. As can be seen in the above, the scaling rate is drastically different from all the existing results (e.g., \Theta(m^{-1}) for width scaling and \Theta(L^{-1/2}) for depth scaling), highlighting the uniqueness of the kernel integral operator and enriching the \mu P framework. Technically, this is because existing \mu P derivations generally boil down to analyzing the behavior of averages of random variables, while in our deviation, the maximum of K^{d} (sub-)Gaussian random variables emerges, which naturally leads to a \sqrt{d\log K} term ( Vershynin 2018 ) . This theoretical result leads to \mu Transfer-FNO ( Algorithm 1 ), an efficient hyperparameter tuning strategy for large-scale FNOs. We conduct extensive experiments to validate our theoretical results and demonstrate the effectiveness of \mu Transfer. We study three important PDE problems: Burgers’ Equation, Darcy Flow, and Navier-Stokes Equation. We empirically show that \mu Transfer preserves the optimal learning rate, training batch size, and optimizer configuration across model scales, and can be applied to both the vanilla FNO training and the advanced Physics Informed Neural Operators ( Li et al. 2024 ) . Specifically, on Navier-Stokes Equations, \mu Transfer-FNO successfully scales to FNOs with nearly 1B parameters and achieves lower test error than the baseline using only 0.30\times training compute. In summary, our contributions in this work are threefold: • We are the first to derive the Maximal Update Parametrization ( \mu P) theory for FNOs by employing new techniques for analyzing FNOs. • Based on our derived scaling rates in \mu P, we introduce the \mu Transfer-FNO algorithm , for zero-shot hyper-parameter transfer in FNO. • We experimentally validate our theoretical claims on various PDEs, different training algorithms, and different hyper-parameters, showing the robustness of the theory. We also demonstrate that \mu Transfer-FNO can significantly reduce computational costs while maintaining or improving accuracy in practice. 2 Preliminaries In this

Read the complete text