Adversarial Autoencoders in Operator Learning
2024
Read Adversarial Autoencoders in Operator Learning (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07811.txt.
Introduction
Dustin Enyeart and Guang Lin study what happens when a rival training step is added to two popular neural operator designs. Their paper appeared in 2024. Neural operators learn how one function maps to another, and two common designs are DeepONets and Koopman autoencoders. Both designs compress and rebuild their input. Adding a rival, or adversarial, step has improved such compress-and-rebuild models in other areas.
The authors add that rival step to both designs and test how each one performs. The paper is brief and reports the comparison in plain terms. Readers get a first look at whether the trick that helps image models also helps models that learn operators. The work points to a cheap change that other groups can test on their own problems.
Catalog record
| YEAR | 2024 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07811
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Autoencoders in Operator Learning Abstract DeepONets and Koopman autoencoders are two prevalent neural operator architectures. These architectures are autoencoders. An adversarial addition to an autoencoder have improved performance of autoencoders in various areas of machine learning. In this paper, the use an adversarial addition for these two neural operator architectures is studied. Keywords: Operator learning, Adversarial autoencoder, DeepONet, Koopman autoencoder, Differential equation, Neural network 25 November 2024 Dustin Enyeart Department of Mathematics Purdue University [email protected] Guang Lin * * * Corresponding author Department of Mathematics School of Mechanical Engineering Purdue University [email protected] 1 Introduction A neural operator is a neural network that is intended to approximate an operator between function spaces [ 1 , 2 , 3 ] . An example of an output function for a neural operator is a solution to a differential equation. Examples of input functions for a neural operator are the initial conditions or boundary conditions for the differential equation. DeepONets are Koopman autoencoders are both neural operator architectures. Furthermore, they are both autoencoders, that is, they consist of an encoder to a latent space and a decoder from this latent space. An adversarial addition to autoencoders is a common extension for neural networks. Intuitively, an adversarial addition to an autoencoder is to encourage the encoder to use the entire latent space in a continuous way, that is, near inputs are near in the latent space. First, in this paper, autoencoders and adversarial autoencoders are explained. Then, the neural operator architectures DeepONets and Koopman autoencoders are introduced. Then, the differential equations that are used in the numerical experiments are presented. Then, results from numerical experiments are presented. Finally, the paper ends with a conclusion of the numerical experiments. 2 Adversarial Autoencoders Autoencoders are a neural network architecture that consists of the composition of an encoder and a decoder [ 4 , 5 ] . The encoder is a neural network that maps the input into a latent space , and the decoder is a neural network from the latent space to the original space. The input and output are in the same space, and the loss of an autoencoder is a measure of the difference between the input and output. Normally, for neural networks, the latent space is smaller than the input space, and the autoencoder tries to learn a lower dimensional representation, that is, it compresses the input. Adversarial Autoencoders are an extension of autoencoders to include a discriminator [ 6 , 7 ] . The discriminator is a neural network from the latent space to the real numbers between 0 and 1 . The discriminator learns to distinguish whether a point in the latent space is in the image of the encoder. Furthermore, the encoder also tries to learn to fool the discriminator. The idea is that this encourages the encoder to use the entire latent space. Data for points in the range of the encoder is from encoding points from the input space, and data that is not in the range of the encoder is from picking random points in the latent space. Such random points are picked from a probability distribution that is similar to the distribution of the true encodings, such as the normal distribution with the same mean and standard deviation as the true encodings. {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces y} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{encoder}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{decoder}} {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces y} {\lx@inpgf@ignorespaces z} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{encoder}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{decoder}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{discriminator}} Figure 1: An autoencoder and an adversarial autoencoder: The left is an autoencoder, where an encoder maps the input x into a latent space, and a decoder maps this encoding to the output y . The right is an adversarial autoencoder, which extends the autoencoder to include a discriminator that maps an encoding to an output z . The discriminator is trained separately from the encoder-decoder pair so that they oppose each other instead of help each other. In practice, they are trained on alternative epochs. Thus, the encoder-decoder pair improves at fooling the discriminator, then the discriminator improves at distinguishing what came from the input, and the process repeats. Intuitively, the output of the discriminator is a probability. This is normally done by using the sigmoid function as the activation function. The sigmoid function is the function that maps an input x to \frac{1}{1+\exp(-x)}. Furthermore, the loss for the discriminator is computed with the binary cross entropy loss. The binary cross entropy loss between a prediction a and a target b is b\cdot\log(a)+(1-b)\cdot\log(1-a). It is popular for binary classification tasks [ 8 ] . When the encoder-decoder pair is being trained, the binary cross entropy loss is used with the opposite targets. For the experiments in this chapter, this loss term is weighted by the accuracy loss so that this term does not dominate the training. Figure 2: The sigmoid function Adversarial autoencoders are particularly popular for generative models [ 9 ] . The idea is that once the encoder is trained to fool the discriminator, a decoding of a random point in the latent space will share characteristics similar to the input data. For example, if the autoencoder learns to compress images of a certain style, then the decoding of a random point in the latent space will hopefully be an image of that style. In an adversarial autoencoder, it is common to add noise to the output of the encoder before passing it to the discriminator. This helps the model be more robust instead of memorizing the encodings of the input data and encourages continuity. This can be thought of as an aid to having only a finite amount of data. 3 Architectures This section introduces the architectures used in this paper. These are DeepONets and Koopman autoencoders. 3.1 DeepONets Deep neural operators , which are abbreviated as DeepONets , are a neural operator architecture [ 10 , 11 , 12 , 13 , 14 , 15 ] . They consist of two parts. The first part encodes information about the differential equation into a latent space. It is called the branch network . The second part encodes a position into a latent space. It is called the trunk network . An example input for the branch network is the initial condition for a dynamic differential equation, and an example for the trunk network is the space-time point to evaluate the solution to the differential equation. In practice, there is usually only one forward pass for the branch network, but many forward passes for the trunk network to encode all the desired positions. The dimensions of the latent spaces of the branch network and the trunk network are the same. And, the output of such a model is the dot product between both latent spaces. Intuitively, the branch network encodes the solution to the differential equation as a set of basis functions, and the trunk network encodes the position as the coefficients of these basis functions. {\lx@inpgf@ignorespaces u} {\lx@inpgf@ignorespaces E_{u}} {\lx@inpgf@ignorespaces y} {\lx@inpgf@ignorespaces x} {\lx@inpgf@ignorespaces E_{x}} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} \scriptstyle{\lx@inpgf@ignorespaces\bigotimes} \scriptstyle{\lx@inpgf@ignorespaces\mathrm{net}} \scriptstyle{\lx@inpgf@ignorespaces\bigotimes} Figure 3: The DeepONet architecture: The input u is the input function, and the input x is the point where the output function is evaluated. Their encodings are denoted by E_{u} and E_{x} , respectively. The output is denoted by y . 3.2 Koopman Autoencoders