Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks
Kang An, Chenhao Si, Shiqian Ma, Ming Yan · 2026
Read Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks by Kang An, Chenhao Si, Shiqian Ma, Ming Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15392.txt.
Introduction
An and three colleagues speed up physics-informed neural networks, which often train slowly and shakily on hard partial differential equations. Their light curvature-aware scheme adds an adaptive fix to standard first-order optimizers, using back-to-back gradient gaps as a cheap gauge of changing loss shape. Tests across PDE benchmarks show faster convergence, steadier training, and better accuracy, including on high-dimensional heat flow.
Catalog record
| CREATOR | Kang An, Chenhao Si, Shiqian Ma, Ming Yan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69356 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15392
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks Kang An, Chenhao Si, Shiqian Ma, Ming Yan Abstract Physics-Informed Neural Networks (PINNs) often suffer from slow convergence, training instability, and reduced accuracy on challenging partial differential equations due to the anisotropic and rapidly varying geometry of their loss landscapes. We propose a lightweight curvature-aware optimization framework that augments existing first-order optimizers with an adaptive predictive correction based on secant information. Consecutive gradient differences are used as a cheap proxy for local geometric change, together with a step-normalized secant curvature indicator to control the correction strength. The framework is plug-and-play, computationally efficient, and broadly compatible with existing optimizers, without explicitly forming second-order matrices. Experiments on diverse PDE benchmarks show consistent improvements in convergence speed, training stability, and solution accuracy over standard optimizers and strong baselines, including on the high-dimensional heat equation, Gray–Scott system, Belousov–Zhabotinsky system, and 2D Kuramoto–Sivashinsky system. 1 Introduction Physics-Informed Neural Networks (PINNs) have emerged as a versatile framework for integrating physical laws into neural networks [ 28 , 13 ] . By incorporating governing Partial Differential Equations (PDEs) into the training objective via automatic differentiation, PINNs enable the solution of both forward and inverse problems without requiring dense observational data. This paradigm combines two complementary strengths: the expressive power of neural networks in approximating complex solution spaces, and the ability to enforce physical constraints through PDE residuals together with boundary and initial conditions. Owing to these advantages, PINNs have demonstrated effectiveness across a broad range of applications, including heat transfer [ 42 , 3 , 31 , 21 ] , solid mechanics [ 10 , 8 ] , stochastic systems [ 45 , 4 ] , and uncertainty quantification [ 44 , 46 , 43 ] . Despite their strong empirical promise, PINNs remain notoriously difficult to train efficiently and reliably across many practically important regimes. For stiff PDEs and problems with sharp gradients or boundary/interior layers, optimization often becomes slow and unstable, frequently leading to degraded accuracy [ 22 , 38 ] . Multi-scale and high-frequency settings, such as Helmholtz equations, introduce additional challenges due to spectral bias and optimization stiffness, causing PINNs to capture low-frequency components more readily than oscillatory structures [ 38 , 39 , 16 , 19 ] . Moreover, because the PINN objective couples PDE residuals with boundary and initial constraints, different loss components may evolve on highly disparate scales. This can lead to gradient imbalance, incomplete enforcement of physical constraints, and pronounced sensitivity to optimizer choice, sampling strategy, and hyperparameter tuning [ 38 , 40 ] . To mitigate these issues, prior work has explored adaptive loss balancing, residual-based sampling, curriculum strategies, and improved initialization [ 38 , 40 , 22 , 2 , 34 , 32 ] . Despite meaningful progress, robust and consistently efficient PINN training remains an open challenge. Recent studies increasingly suggest that these training difficulties are not merely empirical phenomena, but are closely tied to the geometry of the underlying optimization landscape. More broadly in deep learning, loss-landscape analyses have shown that local curvature, sharpness, and directional anisotropy play a central role in shaping optimization behavior and solution quality [ 18 , 47 ] . For PINNs, this connection is even more pronounced: the coupling of PDE residuals, boundary and initial constraints, and higher-order differential operators often gives rise to highly ill-conditioned and anisotropic objectives, with optimization trajectories that traverse regions of markedly different local geometry [ 30 ] . Such landscapes can contain narrow or stiff directions together with relatively flat valleys, making standard first-order updates poorly matched to the local structure and contributing to slow convergence, instability, and sensitivity to training settings. These observations indicate that improving PINN training requires not only better balancing of losses or samples, but also optimization mechanisms that can respond more effectively to the evolving local geometry of the loss landscape. Recent studies have therefore revisited PINN training from the perspective of optimization algorithm design. Empirical and theoretical results show that optimizer choice strongly affects convergence speed, training stability, and final accuracy in PINNs [ 15 , 35 ] . While second-order and quasi-Newton perspectives can better capture the structure of PINN objectives [ 30 , 37 , 35 , 11 ] , existing approaches often introduce substantial computational overhead, require problem-specific tuning, or remain difficult to integrate into standard training pipelines. These limitations motivate a lightweight and broadly compatible framework that can exploit local curvature information more effectively in PINN optimization. In this work, we propose a curvature-aware optimization framework for PINNs that augments existing first-order optimizers with local geometric information. The framework is lightweight, plug-and-play, and broadly compatible, enabling improved adaptation to the anisotropic and rapidly varying landscapes that arise in PINN training. As a result, it improves convergence speed, training stability, and solution accuracy across diverse problems. The main contributions are summarized as follows: • We highlight rapidly varying local curvature as a key optimization challenge in PINN training, and revisit PINN optimization from the perspective of matching update dynamics to the anisotropic and ill-conditioned geometry of PINN objectives. • We propose a lightweight and plug-and-play curvature-aware optimization framework that explicitly incorporates local geometric information into the update process, enabling more effective optimization for PINNs. • The proposed framework is broadly compatible with existing first-order optimizers and is designed to improve geometric adaptation without explicit construction of full second-order matrices, making it practical for integration into modern PINN training pipelines. • We validate the proposed framework on diverse PDE benchmarks and demonstrate consistent improvements in convergence speed, training stability, and solution accuracy over standard optimizers and strong optimizer baselines, highlighting the benefit of explicit curvature-aware adaptation in PINNs. The remainder of this paper is organized as follows. Section 2 briefly introduces PINNs. Section 3 - 3.1 analyze the PINN loss landscape and its associated curvature characteristics. Section 3.2 then introduces the proposed curvature-aware optimization framework. Section 4 provides a rigorous convergence analysis of the framework based on Adam. Finally, Section 5 reports numerical results on several PDE benchmarks. 2 Physics-Informed Neural Network Denote the spatial domain as \Omega\subset\mathbb{R}^{n} with boundary \partial\Omega , and let T represent the time domain. The spatial-temporal variable is given by (\mathbf{x},t)\in\Omega\times T . A time-dependent partial differential equation (PDE) over this domain is defined as follows: \displaystyle\mathcal{F}[u](\mathbf{x},t) \displaystyle=0,\quad(\mathbf{x},t)\in\Omega\times T, (1) \displaystyle\mathcal{B}[u](\mathbf{x},t) \displaystyle=0,\quad(\mathbf{x},t)\in\partial\Omega\times T,\quad\text{(boundary condition)} (2) \displaystyle\mathcal{I}[u](\mathbf{x},0) \displaystyle=0,\quad\mathbf{x}\in\Omega,\hskip 51.0pt\text{(initial condition)} (3) where \mathcal{F} , \mathcal{B} , and \mathcal{I} are differential operators, and u(\mathbf{x},t) is the solution to the PDE, subject to boundary and initial conditions. A PINN parameterized by \theta approximates the solution u(\mathbf{x},t) . The input to the neural network is (\mathbf{x},t) , and the approximation is denoted by \hat{u}(\theta)(\mathbf{x},t) . The PINN minimizes the following objective function: \displaystyle\mathcal{L}(\theta)=\lambda_{F}\mathcal{L}_{F}(\theta)+\lambda_{B}\mathcal{L}_{B}(\theta)+\lambda_{I}\mathcal{L}_{I}(\theta), (4) where \displaystyle\mathcal{L}_{F}(\theta) \displaystyle=\frac{1}{N_{f}}\sum_{(\mathbf{x},t)\in\Omega_{F}}\big|\mathcal{F}[\hat{u}(\theta)](\mathbf{x},t)\big|^{2}, (5) \displaystyle\mathcal{L}_{B}(\theta) \displaystyle=\frac{1}{N_{b}}\sum_{(\mathbf{x},t)\in\Omega_{B}}\big|\mathcal{B}[\hat{u}(\theta)](\mathbf{x},t)\big|^{2}, (6) \displaystyle\mathcal{L}_{I}(\theta) \displaystyle=\frac{1}{N_{0}}\sum_{(\mathbf{x},0)\in\Omega_{I}}\big|\mathcal{I}[\hat{u}(\theta)](\mathbf{x},0)\big|^{2}. (7) Here, \Omega_{F} , \Omega_{B} , and \Omega_{I} are the training sets for the PDE residual, boundary condition, and initial condition, respectively, with cardinality N_{f} , N_{b} , and N_{0} . The weights \lambda_{F} , \lambda_{B} , and \lambda_{I} are hyperparameters tuning the contributions of each loss component. Notably, \Omega_{F} may include points on the boundary or at the initial time, allowing \Omega_{F}\cap\Omega_{B} and \Omega_{F}\cap\Omega_{I} to be non-empty. 3 Why the instantaneous gradient is