A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks

Kyle R. Chickering · 2024

Read A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks by Kyle R. Chickering (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09752.txt.

Introduction

Kyle Chickering, writing alone, speeds up hard derivative math for neural networks in a 2024 paper. Networks used to solve differential equations need derivatives of their own outputs, sometimes many layers deep. Standard autodifferentiation slows at an exponential rate as the order grows, which blocks that use. Chickering extends a classic method called TangentProp, which pushes derivatives through a network layer by layer.

The new method, n-TangentProp, computes the exact nth derivative in near-linear time instead. It suits dense feed-forward networks with smooth activation rules. Tests across depths, widths, and derivative counts backed the claim. In physics-informed networks, which bake equations into training, the method cut training time sharply. Gains grew with model size and with the number of derivatives the loss required. Code is released with the paper.

Catalog record

CREATORKyle R. Chickering
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09752

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks Kyle R. Chickering Abstract The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose n n - TangentProp , the natural extension of the TangentProp formalism [ 34 ] to arbitrarily many derivatives. n n - TangentProp computes the exact derivative d n / d ​ x n ​ f ​ ( x ) d^{n}/dx^{n}f(x) in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network f f with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where n n - TangentProp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n_tangentprop . I Introduction Physics-informed neural networks (PINNs) were introduced as a numerical method to solve forward and inverse problems involving differential equations using neural networks instead of traditional numerical solvers [ 29 ] . Their use has recently come under scrutiny for several reasons, including a lack of high-accuracy results, poor run-times compared to standard numerical methods, and complicated training dynamics [ 22 ] . Due to methodological issues in the aforementioned studies, including the failure to use the well-established strategies outlined in [ 37 ] , we believe that the pursuit of PINNs should not be abandoned and to that end, propose an algorithm which partially addresses the valid concerns over PINN training times. A primary reason to use PINNs over standard numerical methods is because by approximating the solution to an ODE or PDE by a neural network we obtain a C^{\infty} approximation to the model solution which can be evaluated at arbitrary points in the domain, as well as allowing for the study of high-order derivatives (see Figures 7 and 10 ). This strength is also a weakness, since during training we must repeatedly take derivatives of the neural network with respect to the inputs. This is done using autodifferentiation [ 3 , 29 ] , which suffers from unfavorable scaling when taking multiple derivatives. In particular, taking n derivatives of a neural network f with M parameters gives the exponential runtime \mathcal{O}(M^{n}) . For training PINNs we often need to take two or more derivatives, and in many practical applications this exponential runtime already becomes prohibitive. Furthermore this difficulty cannot simply be overcome by horizontally scaling compute, since repeated applications of autodifferentiation cannot be parallelized on a GPU due to the recursive nature of computing high-order derivatives. In this paper we introduce n - TangentProp , which addresses these issues by computing \frac{d^{n}}{dx^{n}}f(x) in quasilinear \mathcal{O}(e^{\sqrt{n}}M) time instead of the exponential time \mathcal{O}(\frac{e^{\sqrt{n}}}{n}M^{n}) . n - TangentProp is an exact method, and thus there is no accuracy degradation when using this method. n - TangentProp is the natural extension of the TangentProp formalism [ 34 ] to n derivatives. TangentProp was introduced in the context of MNIST digit classification as a way to enforce a smoothness condition on the first derivative of a neural network based classifier. The motivation for n - TangentProp starts from the observation that for PINN training we only need higher-order derivatives with respect to the network inputs, not with respect to the network weights. In practice the dimensionality of the input data d is much smaller than the number of parameters M , and therefore it is unnecessary to compute a fully filled out computational graph for all higher order derivatives. Instead of building the full computational graph, we directly differentiate the network during the forward pass and can thus compute all n derivatives with respect to the network inputs during a single forward pass. Our contributions are three-fold 1. We derive the n - TangentProp formalism and give an algorithm for it’s implementation. 2. We show that for simple network architectures consisting of stacked linear, densely connected layers with the \tanh activation function, our method empirically follows the theoretical scaling laws for a variety of widths, depths, and batch sizes. 3. We show that in the context of PINN training our method can significantly reduce training time and memory requirements when compared to the standard PINN implementation. II PINN Training PINNs are neural networks trained to approximate the solution to a given ODE or PDE [ 29 ] . Because neural networks with smooth activation functions are C^{\infty} function approximators, and C^{\infty} functions are dense in most function space which are used in practice (like the L^{2} based Sobolev family of function spaces), we can train a neural network using gradient descent to approximate the solution to a given differential equation. It is this simple observation that led to the introduction of PINNs in the paper [ 29 ] . We give a brief overview of the methodology behind PINNs, but refer the reader to the recent surveys [ 1 , 14 , 36 ] and the references therein, as the literature abounds with introductory material on PINNs. Let u_{\theta}(\bm{x}) be a feed-forward, densely connected, neural network with parameters \theta . Our goal is to optimize these parameters in such a way that u_{\theta} is an approximate C^{\infty} solution to the differential equation F(\partial^{\alpha}u;\bm{x})=0 for some multi-index \alpha [ 13 ] . We train the neural network on the discrete domain \Omega=\{\bm{x}_{1},\cdots\bm{x}_{N}\} using the loss target \displaystyle L(u_{\theta})=\frac{1}{N}\sum_{k=1}^{N}|F(\partial^{\alpha}u_{\theta};\bm{x}_{k})|^{2}+\text{BC}, (1) which is the mean-squared error (MSE) of the differential equation residual with BC being appropriately enforced boundary conditions. Since u_{\theta} is |\alpha| -times continuously differentiable we can use autodifferentiation to exactly compute the derivatives appearing above, and thus by the approximation theorem [ 9 , 17 ] , if the solution to F(\partial^{\alpha}u;\bm{x})=0 lies in a space in which C^{\infty} functions are dense, we can theoretically train a neural network to approximate the true solution of the PDE to arbitrary precision. In practice this is not so easy: PINN training is complicated by needing to enforce boundary conditions, which introduces problems inherent to multi-target machine learning [ 5 , 38 ] . There is also the problem of effectively choosing collocation points from the domain, as well as the problem of choosing a good network architecture [ 37 , 42 ] . PINN training appears naturally unstable, likely due to a poorly conditioned Hessian, and thus training these networks can be difficult [ 19 , 30 , 39 ] . This instability has further been related to the conditioning of a specific differential operator related to the underlying differential operator [ 11 ] . PINNs also appear to struggle fitting high-frequency components of the target solution [ 21 ] as a consequence of spectral bias (f-principle) [ 28 , 41 ] . Additionally, convergence under the loss function ( 1 ) is often slow. In practice it is usually better to use ”Sobolev training” [ 10 , 20 , 35 ] which replaces ( 1 ) with the Sobolev-norm [ 13 ]

Read the complete text