Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization

Andrés Guzmán-Cordero, Felix Dangel, Gil Goldshlager, Marius Zeinhofer · 2025

Read Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization by Andrés Guzmán-Cordero, Felix Dangel, Gil Goldshlager, Marius Zeinhofer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12149.txt.

Introduction

Physics-guided neural nets learn fast under natural gradient steps, but each step costs heavy compute. Andrés Guzmán-Cordero, Felix Dangel, Gil Goldshlager, and Marius Zeinhofer cut that cost in a 2025 paper. Three tricks stack in their method.

A matrix identity shrinks the core solve by a wide margin. A borrowed subsampling scheme adds momentum-like speed. Random sketches trim work further when batches run large. In tests, the stack reached the same error up to 75 times faster. Random tricks helped most early on in small problems. The team maps where each trick pays off.

Catalog record

CREATORAndrés Guzmán-Cordero, Felix Dangel, Gil Goldshlager, Marius Zeinhofer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization Andrés Guzmán-Cordero, Felix Dangel, Gil Goldshlager, Marius Zeinhofer Abstract Natural gradient methods significantly accelerate the training of Physics-Informed Neural Networks (PINNs), but are often prohibitively costly. We introduce a suite of techniques to improve the accuracy and efficiency of energy natural gradient descent (ENGD) for PINNs. First, we leverage the Woodbury formula to dramatically reduce the computational complexity of ENGD. Second, we adapt the Subsampled Projected-Increment Natural Gradient Descent algorithm from the variational Monte Carlo literature to accelerate the convergence. Third, we explore the use of randomized algorithms to further reduce the computational cost in the case of large batch sizes. We find that randomization accelerates progress in the early stages of training for low-dimensional problems, and we identify key barriers to attaining acceleration in other scenarios. Our numerical experiments demonstrate that our methods outperform previous approaches, achieving the same L 2 L^{2} error as the original ENGD up to 75 × 75\times faster. 1 Introduction Using neural networks to solve partial differential equations (PDEs) is a promising and highly active research direction at the intersection of machine learning and scientific computing. While the idea is not new [ 7 , 20 ] , it has gained traction with the advent of Physics Informed Neural Networks (PINNs) [ 33 ] . Compared to traditional methods, neural networks avoid the need for sophisticated, problem-dependent discretization schemes and promise to scale better to high-dimensional problems [ 33 ] . However, it is very challenging to attain the accuracy levels of traditional solvers with neural networks. This is in large part due to the loss landscape’s non-convexity and ill-conditioning [ 44 ] . To address this problem, second-order methods such as energy natural gradient descent [ 27 , ENGD,] and Kronecker-factored approximate curvature [ 24 , 6 , KFAC,] have been proposed. These methods greatly improve upon the accuracies attainable by first-order optimizers like SGD or Adam [ 6 ] , but they suffer from either a high per-iteration cost or a high implementation complexity. As a result, there remains a significant need for more accurate and efficient methods to train PINNs. In parallel to the PINN community, neural networks have also been leveraged in conjunction with the variational Monte Carlo method [ 9 , VMC,] to simulate quantum many-body systems [ 3 , 14 , 21 , 32 ] . Second-order optimizers are very common within this field and the most popular one, known as stochastic reconfiguration [ 42 , 1 , SR,] , shares a similar computational structure to ENGD, owing to a similar mathematical derivation as a projected functional algorithm [ 28 ] . Within the neural network VMC community, the quest to model complex quantum systems to as high of an accuracy as possible has led to significant advances in algorithmic implementations of the SR optimizer, including the use of the Woodbury matrix identity to reduce the computational cost of each optimization step [ 4 , MinSR,] and the development of momentum schemes that can accelerate the convergence of natural gradient methods [ 11 , SPRING,] . Given the similarities between VMC and PINNs it is natural to wonder if PINNs can benefit from the techniques developed in the more mature optimization literature for VMC. In this paper, we answer this question in the affirmative (See Figure 1 ). We first show that both MinSR and SPRING can be adapted to drastically improve ENGD for PINNs. Going beyond transfer, we explore the use of randomized methods to further reduce the cost of both ENGD and SPRING iterations. Concretely, our contributions are as follows: 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 VMC+SR ∗ [ 9 , 42 ] NNs in VMC [ 3 ] K-FAC [ 32 ] MinSR † [ 4 ] SPRING † [ 11 ] PINNs [ 33 ] ENGD ∗ [ 27 ] K-FAC [ 6 ] Us † Nyström Method [ 48 ] Sketch and Solve † [ 38 ] Sketch and Precond. [ 37 ] Low-rank Approx. † [ 13 ] Nyström PCG [ 10 ] Figure 1 : Timeline of VMC , PINN and RLA methods . * : Stochastic reconfiguration (SR) and energy natural gradient descent (ENGD) both precondition their stochastic gradients with the inverse of an appropriate curvature matrix. \dagger : Inspired by MinSR, we apply Woodbury’s matrix identity to use the kernel matrix instead of ENGD’s Gauss-Newton matrix, thus reducing the cost to O(N^{2}P) instead of O(P^{3}) , where N denotes the batch size and P the number of parameters; we further introduce SPRING for PINNs to transport curvature information across optimization steps; last, we use a GPU-efficient Nyström approximation to reduce the iteration cost for large batch sizes. • We transfer ideas from the VMC community to PINNs , showing that the Woodbury matrix identity drastically improves the computational bottleneck of ENGD for PINNs. Concretely (details in Section 3 ), we compute in sample space, as its dimensionality is typically much smaller than the parameter space. This enables an exact ENGD for PINNs with up to 10^{6} trainable weights on standard consumer hardware with drastically improved computation time. Unlike K-FAC, the implementation is architecture-agnostic and thus highly flexible. We then show that the SPRING algorithm introduced in the VMC community can further improve the training process of PINNs, yielding consistently faster convergence and removing the need for the expensive line search from ENGD. The resulting method, summarized in algorithm 1 , is straightforward to implement and achieves state-of-the-art results, and we thus recommend it as the new optimizer of choice for training PINNs. • Going beyond transfer, we explore the use of randomized algorithms based on a variant of Nyström’s method [ 10 ] to further accelerate both ENGD and SPRING in the case of large sample sizes. As standard formulations of Nyström’s method require SVDs, which are extremely slow on GPU in contemporary ML libraries, we develop a GPU-efficient variant of Nyström that only uses Cholesky solves on square matrices of sketch dimension. We highlight this GPU-efficient Nyström approximation as an independent contribution which can be useful in a wide array of machine learning applications. For PINNs, we find that it can accelerate the early phases of training for low-dimensional problems, and we identify key barriers to extending this acceleration to the later phases and higher-dimensional settings. • We present thorough empirical evidence for our claims , demonstrating the utility of the Woodbury identity and the SPRING algorithm as well as the power and the limitations of randomization. We explore both low- and high-dimensional problems and we meticulously tune the hyperparameters of each method to attain optimal performance, with final results supported by more than 4500 training runs in total. Related work The Woodbury identity has also been used to train neural networks for traditional machine learning problems [ 5 , 49 , 35 ] and other applications, such as model sparsification [ 41 ] , involving the Fisher information matrix in traditional deep learning. Descriptions of the ill-conditioned landscape of PINNs and some solutions have been explored Rathore

Read the complete text