Improved Physics-informed neural networks loss function regularization with a variance-based term

John M. Hanna, Hugues Talbot, Irene E. Vignon-Clementel · 2024

Read Improved Physics-informed neural networks loss function regularization with a variance-based term by John M. Hanna, Hugues Talbot, Irene E. Vignon-Clementel (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13993.txt.

Introduction

Neural networks that embed physical laws can solve hard equations, but their training often leaves pockets of large error where the solution bends sharply. Hanna and co-authors propose a better training target in this 2024 paper. Standard training cuts the average error alone, which lets local outliers survive; their loss adds the spread of the error as a second term to minimize.

Squeezing both the mean and the spread pushed errors to spread more evenly and cut the worst local errors. They tested the idea on four problems of rising difficulty, from a one-dimensional Poisson equation to steady fluid flow in two dimensions. Solution quality rose and peak error fell, at almost no extra computing cost. The change is simple to adopt in existing code.

Catalog record

CREATORJohn M. Hanna, Hugues Talbot, Irene E. Vignon-Clementel
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13993

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improved Physics-informed neural networks loss function regularization with a variance-based term John M. Hanna, Hugues Talbot, Irene E. Vignon-Clementel Abstract In machine learning and statistical modeling, the mean square or absolute error is commonly adopted as an error metric, also called a "loss function". While effective in reducing the average error, this approach may fail to address localized outliers, leading to significant inaccuracies in regions with sharp gradients or discontinuities. This issue is particularly evident in physics-informed neural networks (PINNs), where such localized errors are expected and affect the overall solution. To overcome this limitation, we propose a novel loss function that combines the mean and the standard deviation of the chosen error metric. By minimizing this combined loss function, the method ensures a more uniform error distribution and reduces the impact of localized high-error regions. The proposed loss function is easy to implement and tested on problems of varying complexity: 1D Poisson equation, unsteady Burger’s equation, 2D linear elastic solid mechanics, and 2D steady Navier Stokes equations. Results demonstrate improved solution quality and lower maximum error compared to the standard mean-based loss, with minimal impact on computational time. Journal: Engineering Applications of Artificial Intelligence 1 Introduction Deep learning and its applications have been growing rapidly in the past decade or so. This is mainly due to the availability of large datasets and the groundbreaking technology of graphics processing units (GPUs) which have made it possible to train large neural network models with huge datasets. Recent neural network architectures have improved the accuracy of some deep learning tasks. For example, graph neural networks [ 1 ] have made it easier to deal with data that can be organized in graphs, which has many applications such as in molecular biology [ 2 ] or social networks [ 3 ] . Transformer-based architectures [ 4 ] with the novel self attention mechanism have significantly contributed to progress in natural language processing [ 5 ] and computer vision [ 6 ] tasks. In the recent years, neural networks were presented to solve partial differential equations (PDE), introducing physics-informed neural networks (PINN) [ 7 ] . PINN reduce the need for big datasets and regularize outputs since the physics, governed by PDEs, are naturally regularizing the predictions. The main idea is to approximate the solution field of PDEs thanks to the strong approximation capabilities of neural networks [ 8 ] . The solution is obtained by minimizing a combined loss function which represents the data mismatch (initial/ boundary conditions enforcement and/or sparse data in the domain) and the residual of the PDE over points sampled in the spatiotemporal domain called collocation points. The residual is usually obtained by automatic differentiation [ 9 ] . Since, the introduction of PINN, they have been applied in several fields including solid mechanics [ 10 , 11 ] , fluid mechanics [ 12 , 13 , 14 , 15 ] , flow in porous media [ 16 , 17 , 18 ] , topology optimization [ 19 , 20 ] , composites manufacturing [ 21 , 22 ] , fracture mechanics [ 23 , 24 ] , and many others. One of the most crucial elements in deep learning, whether for supervised learning tasks or physics-informed neural networks tasks, is the choice of loss function. A pertinent loss function can lead to significant improvements in the optimization process, thus good accuracy can be reached more rapidly. In general, the loss function represents the mean error over the training points. In classical machine learning, many loss functions have been proposed, often associated with a specific task. The most common loss used in regression, the Mean Square Error (MSE) assumes that the error distribution is Gaussian with a uniform variance, which is often not realistic. In practice, the MSE works well enough even when the error distribution is only symmetric. However, when the variance of the error is not uniform (the heteroscedastic case), it is useful to use a loss that takes the local variance into account: \text{Loss}_{\text{H}}=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{(y_{i}-\hat{y_{i}})^{2}}{2\hat{\sigma_{i}}^{2}}+\frac{1}{2}\log\hat{\sigma_{i}}^{2}\right), where y_{i} is the true target value, \hat{y_{i}} the predicted mean, \hat{\sigma_{i}}^{2} the predicted variance. This model will predict both mean and variance around each training point, which is helpful when considering uncertainty. Such an approach was first proposed in risk-averse portfolio optimization in finance [ 25 ] . In practice this loss is difficult to optimize [ 26 ] . Gaussian processes [ 27 ] , in particular krieging [ 28 ] are also able to estimate localized mean and variance throughout the whole domain, but are not immediately applicable to neural networks, although there has been progress in this area [ 29 ] . In these works, the main motivation is to estimate a local variance and adapt the MSE locally. However, dealing with non-uniform variance can also be seen as a form of robust regression, i.e. regression which is less sensitive to outliers. Loss functions like the mean absolute error or its differentiable variant, the Huber loss [ 30 ] for regression tasks have been proposed in the literature to better deal with outliers. In practice, these losses are better suited for finding sparse solutions to regression problem [ 31 ] . With PINN architectures, the situation is different. Classical outliers that represent unexpected out-of-distribution events can occur, but much more common are regions with high gradients or discontinuities that can lead to high localized residuals as compared to other regions, which in turn possibly deteriorates the predictions. These must be dealt with in a different fashion than in classical learning. Several recent works attempt to improve the accuracy of PINN solution by adding extra regularization terms to the loss function. An example is the gradient-enhanced PINNs introduced by [ 32 ] where a higher order derivative term is added to the loss function. This term represents the residual gradient so as to ensure not just the residual of the PDE to be small, but also its derivative, which improves the solution accuracy. However it comes at a high computational cost, typically more than (d+1)\times the original PINN computational time, where d is the dimension of the problem. Another work improved the training of PINN using a measure of the variance, or uncertainty of the output [ 33 ] . The main idea is to have two different outputs from the neural network, one representing the mean of the solution and the other represents the variance. In this sense it is similar to the heteroscedastic learning methods in classical learning mentioned above. Another term is added in the loss function that represents the log of the variance, the second output of the neural network. The idea is fundamentally different from our proposal in the sense that the variance of the solution is predicted by the neural network rather than being calculated. The idea can be rather seen as an automated point-wise scaling of the collocation points. In this work, we present a new loss function that targets this issue. The core idea is based on an addition to the loss function. This addition includes the standard deviation of the chosen error to be minimized. By this addition, we are ensuring that minimizing the loss function can reduce both the

Read the complete text