Multi-scale DeepOnet (Mscale-DeepOnet) for Mitigating Spectral Bias in Learning High Frequency Operators of Oscillatory Functions
B. Wang, Lizuo Liu, Wei Cai · 2025
Read Multi-scale DeepOnet (Mscale-DeepOnet) for Mitigating Spectral Bias in Learning High Frequency Operators of Oscillatory Functions by B. Wang, Lizuo Liu, Wei Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10932.txt.
Introduction
Bo Wang, Lizuo Liu, and Wei Cai improve operator learning for wave problems in this two thousand twenty-five paper. DeepOnet is a neural architecture that learns mappings between functions, such as from a wave equation's coefficient to its solution. Standard networks suffer spectral bias: they learn low frequencies easily and high frequencies poorly. For highly oscillatory functions, that bias cripples accuracy.
The proposed multi-scale DeepOnet inserts multiscale neural networks into both the branch and trunk parts of the original design. That change lets the model capture varied high-frequency components of both the mapping and its output. The test application is the nonlinear map from the Helmholtz equation's coefficient to its solution, central to wave scattering. Numerical results show substantial improvement over ordinary DeepOnet in the high-frequency regime, at a similar number of network parameters.
Catalog record
| CREATOR | B. Wang, Lizuo Liu, Wei Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10932
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-scale DeepOnet (Mscale-DeepOnet) for Mitigating Spectral Bias in Learning High Frequency Operators of Oscillatory Functions B. Wang, Lizuo Liu, Wei Cai Abstract In this paper, a multi-scale DeepOnet (Mscale-DeepOnet) is proposed to reduce the spectral bias of the DeepOnet in learning high-frequency mapping between highly oscillatory functions, with an application to the nonlinear mapping between the coefficient of the Helmholtz equation and its solution. The Mscale-DeepOnet introduces the multiscale neural network in the branch and trunk networks of the original DeepOnet, the resulting Mscale-DeepOnet is shown to be able to capture various high-frequency components of the mapping itself and its image. Numerical results demonstrate the substantial improvement of the Mscale-DeepOnet for the problem of wave scattering in the high-frequency regime over the normal DeepOnet with a similar number of network parameters. 1 Introduction The DeepONet has shown its ability to learn not only explicit mathematical operators like integration and fractional derivatives, but also PDE operators [ 1 , 3 , 4 , 7 , 9 ] . However, like neural networks such as PINN in solving PDEs, neural operators including DeepOnet and FNO demonstrates the behaviour of spectral bias where learning is preferred for low frequency component of the approximation [ 11 , 12 , 14 ] . The high frequency aspect of a mapping between functions reflects the dependence of the output function on the value of the input function such as in the simple nonlinear mapping of u(x)=\sin(ma(x)) for large value of m . This kind of functional relationship appears in a similar manner in the relation between the material dielectric properties of a scatterer and the scattering field. Meanwhile, for high frequency wave scattering, the scattering field u(x) by itself could be an oscillatory function. However, we will need to distinguish the high frequency behavior of the scattering field u(x) and that of the nonlinear mapping as the latter comes from the form of the Green’s function of the Helmholtz equations for the scattering problem. It will be seen that the large value m related to the size of the scattering and a larger value corresponds to scattering problems in high frequency regime for a given wavelength of the incidence wave. In this paper, we proposed an enhanced variant of the DeepOnet by introducing the technique of multiscale deep neural network proposed in [ 8 ] into the trunk-net of the DeepOnet structure to reduce the spectral bias of DeepOnet in learning high frequency mapping between functions, especially oscillatory functions. A previous work on mutiscale FNO [ 13 ] has been shown to reduce much the spectral bias of high-frequency mapping. The rest of the paper is organized as follows. In Section 2, we will review the DeepOnet and the multiscale DeepOnet will be introduced in Section 3. Numerical results for the Mscale-DeepOnet will be given in Section 4. Finally, a conclusion and future work is discussed in Section 5. 2 DeepOnet The universal approximation theory of neural networks is one of the mathematical basis for the broad applications of neural network, which justifies approximating functions or operators by weighted compositions of some functions, whose inputs are also weighted. The parameters, i.e., the weights and bias, could be obtained by minimizing a loss function with optimization algorithms such as stochastic gradient descent and its variants. Thus, neural network learning turns an approximation problem to one of optimization with respect to the parameters. In this section, we focus on the approximation of operators. The work of [ 2 ] gives a constructive procedure for approximating nonlinear operator \mathcal{G} between continuous functions \mathcal{G}\left(f\right)\left(x\right) in a compact subset of C(\mathcal{X}) with \mathcal{X}\subseteq\mathbb{R}^{d} and continuous functions f\left(x\right) in a compact subset of C\left(\mathcal{F}\right) with \mathcal{F}\subseteq\mathbb{R}^{d} \mathcal{G}:f(x)\in C(\mathcal{F})\rightarrow\mathcal{G}(f)(x)\in C(\mathcal{X}), (1) where C\left(\mathcal{X}\right) and C\left(\mathcal{F}\right) are the continuous function spaces over \mathcal{X} and \mathcal{F} , respectively, and \mathcal{X} and \mathcal{F} are compact subsets of \mathbb{R}^{d} , the Euclidean space of dimension d . The universal approximations with respect to operators are based on the two following results: 1. Universal Approximation of Functions [ 2 ] : Given any \varepsilon_{1}>0, there exists a positive integer N , \left\{\bm{w}_{k}\right\}_{k=1}^{N}\in\mathbb{R}^{d},\left\{{b}_{k}\right\}_{k=1}^{N}\in\mathbb{R} , such that functions \mathcal{G}(f)(x) selected from a compact subset \mathcal{U} of C(\mathcal{X}) could be uniformly approximated by a one-hidden-layer neural network with any Tauber-Wiener \left(\text{TW}\right) activation function \sigma_{t} \left|\mathcal{G}(f)(x)-\sum_{k=1}^{N}{\color[rgb]{0,0,0}\mathcal{J}_{k}\left(\mathcal{G}(f)\right)}\sigma_{t}\left(\bm{w}_{k}\cdot x+b_{k}\right)\right|<\varepsilon_{1},\quad\forall\text{ }x\in\mathcal{X}, (2) where \color[rgb]{0,0,0}\mathcal{J}_{k}\left(\mathcal{G}(f)\right) is a linear continuous functional defined on \mathcal{V} (a compact subset of C(\mathcal{F}) ), and all \bm{w}_{k},b_{k} are independent of x and f\left(x\right) . A Tauber-Wiener activation function is defined as follows. Definition 1 . Assume \mathbb{R} is the set of real numbers. \sigma:\mathbb{R}\rightarrow\mathbb{R} is called a Tauber-Wiener \left(\text{TW}\right) function if all the linear combinations g(x)=\sum\limits_{i=1}^{I}c_{i}\sigma\left({w}_{i}x+b_{i}\right) are dense in every C\left[a,b\right] , where \left\{{w}_{i}\right\}^{I}_{i=1},\left\{b_{i}\right\}^{I}_{i=1},\left\{c_{i}\right\}^{I}_{i=1}\in\mathbb{R} are real constants. 2. Universal Approximation of Functionals [ 2 ] : Given any \varepsilon_{2}>0 , there exists a positive integer M , m points \left\{x_{j}\right\}_{j=1}^{m}\in\mathcal{F} with real constants c_{i}^{k},{W}_{ij}^{k},B_{i}^{k}\in\mathbb{R},i=1,\ldots,M,j=1,\ldots,m, such that a functional \color[rgb]{0,0,0}\mathcal{J}_{k}\left(\mathcal{G}(f)\right) could be approximated by a one-hidden-layer neural network with any TW activation function \sigma_{b} \left|{\color[rgb]{0,0,0}\mathcal{J}_{k}\left(\mathcal{G}(f)\right)}-\sum_{i=1}^{M}c_{i}^{k}\sigma_{b}\left(\sum_{j=1}^{m}{W}_{ij}^{k}f\left(x_{j}\right)+B_{i}^{k}\right)\right|<\varepsilon_{2},\quad\forall f\in\mathcal{V}, (3) where the coefficients c_{i}^{k},{W}_{ij}^{k},B_{i}^{k} and nodes \{x_{j}\}_{j=1}^{m} and m,M are all independent of f\left(x\right) . Combining these two universal approximations, the authors of [ 2 ] propose the universal approximations of nonlinear operators by neural networks when restricted to the compact subset \mathcal{V} of the continuous function space C\left(\mathcal{F}\right) defined on a compact domain \mathcal{F} in \mathbb{R}^{d} . Namely, given any \varepsilon>0, there exists positive integers M,N , m points \left\{x_{j}\right\}_{j=1}^{m}\in\mathcal{F}\subseteq\mathbb{R}^{d} with real constants c_{i}^{k},W_{ij}^{k},B_{i}^{k}\in\mathbb{R},i=1,\ldots,M,j=1,\ldots,m, \left\{\bm{w}_{k}\right\}_{k=1}^{N}\in\mathbb{R}^{d},\left\{b_{k}\right\}_{k=1}^{N}\in\mathbb{R} that are all independent of continuous functions f\in\mathcal{V}\subseteq C(\mathcal{F}) and x\in\mathcal{X}\subseteq\mathbb{R}^{d} such that \left|\mathcal{G}(f)(x)-\sum_{k=1}^{N}\sum_{i=1}^{M}c_{i}^{k}\sigma_{b}\left(\sum_{j=1}^{m}W_{ij}^{k}f\left(x_{j}\right)+B_{i}^{k}\right)\sigma_{t}\left(\bm{w}_{k}\cdot x+b_{k}\right)\right|<\varepsilon (4) 3 Multiscale DeepOnet In this section, we introduce a multiscale technique into the DeepOnet framework to enhance its performance in learning operators whose outputs are highly oscillatory functions. The high oscillation could come from the oscillation of the input function or the high frequency nature of the objective operator. 3.1 MscaleDNN - multi-scale deep neural network The multi-scale DNN(MscaleDNN) [ 8 ] uses a simple parallel architectural framework to approximate functions f:\Omega\to\mathbb{R} with a wide range of spectral content for a general domains \Omega\subset\mathbb{R}^{d} . The MscaleDNN is designed to reduce the spectral bias of DNNs [ 12 , 11 ] , which characterizes the preference toward lower frequencies in the learning dynamics of DNNs. The main idea of MscaleDNN can be understood from the following approximation problem. Consider a band-limited function f with a limited frequency range, \hbox{supp}\hat{f}(\bm{k})\subset\{\bm{k}\in\mathbb{R}^{d},\;|\bm{k}|\leq k_{\mathrm{max}}\}. (5) First, we partition the frequency domain of f(\bm{x}) A_{i}=\{\bm{k}\in\mathbb{R}^{d},\;K_{i-1}\leq|\bm{k}|\leq K_{i}\},\quad i=1,2,\cdots,M, (6) where 0=K_{0}<K_{1}<\cdots<K_{M}=k_{\mathrm{max}} and \hbox{supp}\hat{f}(\bm{k})\subset\bigcup_{i=1}^{M}A_{i} . Then, f can be represented as a sum of functions with non-overlapping frequency information f(\bm{x})=\sum_{i=1}^{M}f_{i}(\bm{x}),\quad f_{i}(\bm{x})=\int_{A_{i}}\hat{f}(\bm{k})e^{j\bm{k}\cdot\bm{x}}\,d\bm{k}. (7) Next, one performs a radial scaling on in the frequency domain \hat{f}^{(scale)}_{i}(\bm{k})=\hat{f}_{i}(\alpha_{i}\bm{k}), (8) which corresponds to scaling in the physical domain f_{i}(\bm{x})=\alpha_{i}^{n}f^{(scale)}_{i}(\alpha_{i}\bm{x}). If the scaling factor is sufficiently large, the scaled function becomes a low-frequency function with a Fourier transform compactly supported at lower frequencies, \hbox{supp}\hat{f}_{i}^{(scale)}(\bm{k})\subset\Big\{\bm{k}\in\mathbb{R}^{d},\;\frac{K_{i-1}}{\alpha_{i}}\leq|\bm{k}|\leq\frac{K_{i}}{\alpha_{i}}\Big\},\quad i=1,2,\cdots,M. (9) Due to the spectral bias of the DNNs toward low frequencies, f^{(scale)}_{i}(\bm{x}) can be quickly learned by a DNN f_{\theta_{i}}(x) parameterized by \theta_{i} , f_{i}(\bm{x})=\alpha_{i}^{n}f^{(scale)}_{i}(\alpha_{i}\bm{x})\sim\alpha_{i}^{n}f_{\theta_{i}}(\alpha_{i}\bm{x}). (10) Since f(\bm{x})=\sum_{i=1}^{M}f_{i}\left(\bm{x}\right), the MscaleDNN approximation can be found by f(\bm{x})\sim\sum_{i=1}^{M}\alpha_{i}^{n}f_{\theta_{i}}(\alpha_{i}\bm{x}). (11) The MscaleDNN uses a series of sub-networks with different scaled inputs to approximate