Some Best Practices in Operator Learning

2024

Read Some Best Practices in Operator Learning (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06686.txt.

Introduction

Dustin Enyeart and Guang Lin share practical lessons on training choices for operator learning, in work from 2024. Tuning hyperparameters by search is costly, so robust defaults matter. Tests span three designs, DeepONets, Fourier neural operators and Koopman autoencoders, across several differential equations, hunting for trends that hold broadly. Choices studied include activation functions, dropout and stochastic weight averaging.

Catalog record

YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Some Best Practices in Operator Learning Abstract Hyperparameters searches are computationally expensive. This paper studies some general choices of hyperparameters and training methods specifically for operator learning. It considers the architectures DeepONets, Fourier neural operators and Koopman autoencoders for several differential equations to find robust trends. Some options considered are activation functions, dropout and stochastic weight averaging. Keywords: Operator learning, Best practices, Hyperparameters, Training methods, DeepONet, Fourier neural operator, Koopman autoencoder, Activation function, Dropout, Stochastic weight averaging, Learning rate finder 25 November 2024 Dustin Enyeart Department of Mathematics Purdue University [email protected] Guang Lin * * * Corresponding author Department of Mathematics School of Mechanical Engineering Purdue University [email protected] 1 Introduction A neural operator is a neural network that is intended to approximate an operator between function spaces [ 1 , 2 , 3 ] . An example of an output function for a neural operator is a solution to a differential equation. Examples of input functions for a neural operator are the initial conditions or the boundary conditions for the differential equation. The study of neural operators is called operator learning . Various choices of hyperparameters and training strategies are made when training a neural network. Limiting such choices to known robust choices can significantly improve the speed of hyperparameters searches. This paper studies the effect of some of these choices in operator learning, that is, it tries to find some best practices in operator learning. This is not a comparison between different architectures. In order, this paper studies the choice of activation function, the use of dropout, the use of stochastic weight averaging and the use of a learning rate finder. 2 Architectures This section introduces the architectures used in this paper. These are DeepONets, Fourier neural operators and Koopman autoencoders. 2.1 DeepONets Deep neural operators , which are abbreviated as DeepONets , are a neural operator architecture [ 4 , 5 , 6 , 7 , 8 , 9 ] . They consist of two parts. The first part encodes information about the differential equation into a latent space. It is called the branch network . The second part encodes a position into a latent space. It is called the trunk network . An example input for the branch network is the initial condition for a dynamic differential equation, and an example for the trunk network is the space-time point to evaluate the solution to the differential equation. In practice, there is usually only one forward pass for the branch network, but many forward passes for the trunk network to encode all the desired positions. The dimensions of the latent spaces of the branch network and the trunk network are the same. And, the output of such a model is the dot product between both latent spaces. Intuitively, the branch network encodes the solution to the differential equation as a set of basis functions, and the trunk network encodes the position as the coefficients of these basis functions. {\lx@inpgf@ignorespaces u} {\lx@inpgf@ignorespaces E_{u}} {\lx@inpgf@ignorespaces y} {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces E_{x}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} \scriptstyle{\lx@inpgf@ignorespaces\bigotimes} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} \scriptstyle{\lx@inpgf@ignorespaces\bigotimes} Figure 1: The DeepONet architecture: The input u is the input function, and the input x is the point where the output function is evaluated. Their encodings are denoted by E_{u} and E_{x} , respectively. The output is denoted by y . 2.2 Fourier Operator Networks Fourier neural operators , which are abbreviated as FNO , are a neural operator architecture [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 9 ] . They are based on spectral convolution layers. In a spectral convolution layer , first the Fourier transform is applied. Then, the higher modes are dropped. Then, a linear function is applied. Then, the inverse Fourier transform is applied [ 17 ] . Similarly to residuals for a normal convolution layer, a linear function is applied to the original input, and the output is added elementwise to the output of the inverse Fourier transform. Finally, an activation function is applied. While a normal convolution layer is local, a spectral convolution layer is global. Furthermore, for a spectral convolution layer to be computationally reasonable, the fast Fourier transform needs to be used [ 18 ] . Thus, the data needs to be structured as a grid. First, in a Fourier neural operator, a feed-forward neural network is applied to increase the channel dimension. Then, a sequence of spectral convolution layers are applied. Then, a feed-forward neural network is applied to decrease the channel dimension to the output size. The original channels include the input function, such as the initial conditions. For time-independent differential equation, the Fourier neural network can be applied for each time step, or the desired time can be included in the channels. In this paper, the time is included in the channels, that is, for each desired time, the Fourier neural network is applied to the initial conditions together with the desired time. When using grids of various resolutions, the spatial step size is also included. And, for inhomogeneous equations, the spatial position is also included. {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces\cdots} {\lx@inpgf@ignorespaces y} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{spectral}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{spectral}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} Figure 2: A Fourier neural operator: The input is denoted by x . First, a feed-forward neural network is used to increase the channel dimension. Then, a sequence of spectral convolution layers are applied. Then, a feed-forward neural network is used to decrease the channel dimension. The output is denoted by y . {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces y} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{FFT}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{linear}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{filter}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{linear}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{IFFT}} \scriptstyle{\lx@inpgf@ignorespaces\bigoplus} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{act}} \scriptstyle{\lx@inpgf@ignorespaces\bigoplus} Figure 3: A spectral convolution layer: The input is denoted by x . On the bottom, a linear layer is applied to each channel. On the top, first the fast Fourier transform is applied. Then, the higher modes are dropped. Then, a linear layer is applied to each channel. Then, the inverse fast Fourier transform is applied. Then, the top and bottom are added elementwise, and an activation function is applied. The output is denoted by y . 2.3 Koopman Autoencoders Koopman autoencoders are a neural operator architecture that are used for time-dependent differential equations [ 19 , 20 , 21 , 22 , 23 ] . They are popular for dynamic mode decomposition [ 24 , 25 , 26 , 27 , 28 ] and control [ 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 ] . The Koopman formulation of classical mechanics is an alternative framework of classical mechanics [ 37 , 38 , 39 ] . It was inspired by the Hamiltonian formulation of quantum mechanics [ 40 , 41 , 42 ] . In this theory, the physical state at a given time is represented as a state in an infinite-dimensional latent space, and an infinite-dimensional operator governs the time evolution in this latent space. Intuitively, the Koopman formulation can be thought of as removing nonlinearity by infinitely increasing the dimension. The Koopman formulation of classical mechanics can be discretized to provide a numerical scheme by approximating the time-evolution operator by a finite-dimensional matrix K . In this scheme, a physical state s_{0} can be evolved into a later physical state s_{n} by encoding it into the latent space, applying the matrix K repetitively and

Read the complete text