Convergence Guarantees for Gradient-Based Training of Neural PDE Solvers: From Linear to Nonlinear PDEs

Wei Zhao, Tao Luo · 2025

Read Convergence Guarantees for Gradient-Based Training of Neural PDE Solvers: From Linear to Nonlinear PDEs by Wei Zhao, Tao Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14002.txt.

Introduction

Wei Zhao and Tao Luo prove when training of neural solvers for physics equations will settle well, in a 2025 paper. Such solvers learn to meet an equation and its edge rules. Users need proof that training will not drift or stall in odd cases.

For straight equations, they grow a past frame to cover a broad class of cases. For curved equations, they prove steps move toward key points with no need for huge networks. Their proof also shows a built in brake that keeps weights from flying off. Test runs on computers back the claims for both solver types.

Catalog record

CREATORWei Zhao, Tao Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergence Guarantees for Gradient-Based Training of Neural PDE Solvers: From Linear to Nonlinear PDEs Wei Zhao, Tao Luo Abstract We present a unified convergence theory for gradient-based training of neural network methods for partial differential equations (PDEs), covering both physics-informed neural networks (PINNs) and the Deep Ritz method. For linear PDEs, we extend the neural tangent kernel (NTK) framework for PINNs to establish global convergence guarantees for a broad class of linear operators. For nonlinear PDEs, we prove convergence to critical points via the Łojasiewicz inequality under the random feature model, eliminating the need for strong over-parameterization and encompassing both gradient flow and implicit gradient descent dynamics. Our results further reveal that the random feature model exhibits an implicit regularization effect, preventing parameter divergence to infinity. Theoretical findings are corroborated by numerical experiments, providing new insights into the training dynamics and robustness of neural network PDE solvers. 1 Introduction Partial differential equations (PDEs) form the mathematical foundation for modeling phenomena across physics, engineering, and applied sciences. While linear PDEs are relatively well-understood, nonlinear PDEs, ubiquitous in modeling complex systems, pose significant analytical and computational challenges due to their lack of superposition principles and potential for solution singularities [ 1 , 2 ] . Recent advances in machine learning have introduced neural PDE solvers, such as physics-informed neural networks [ 3 ] and the Deep Ritz method [ 4 ] , as flexible alternatives to traditional numerical methods. These approaches have demonstrated empirical success in high-dimensional and nonlinear settings [ 5 , 6 , 7 , 8 ] , but their theoretical convergence guarantees remain limited, especially for nonlinear PDEs. Most existing convergence analyses for physics-informed neural networks are developed within the neural tangent kernel framework [ 9 , 10 ] , which primarily provides guarantees for second-order linear PDEs using over-parameterized networks [ 11 , 12 , 13 ] . While it is commonly believed that NTK-based results could be extended to broader classes of linear PDEs, rigorous proofs beyond the second-order setting are still lacking. For the Deep Ritz method, convergence analyses typically rely on coercivity of the bilinear form and Rademacher complexity estimates [ 14 , 15 , 16 ] ; however, these approaches are mostly confined to linear elliptic equations with convex energy functionals, and the extension to general variational problems remains underexplored. Crucially, neither framework currently offers provable convergence guarantees for solving nonlinear PDEs. In particular, when PINNs are used to solve equations with nonlinear differential operators, the associated NTK matrix evolves dynamically during training and, as shown in [ 17 ] , fails to converge to a deterministic kernel in the infinite-width limit. For the Deep Ritz method, the non-convexity inherent in nonlinear PDEs further complicates the analysis. This theoretical gap poses a major challenge to our understanding of neural PDE solvers in nonlinear regimes. In this work, we overcome the aforementioned theoretical limitations by establishing a systematic convergence theory for neural PDE solvers across both linear and nonlinear regimes. For linear PDEs, we extend the NTK framework to establish convergence guarantees for PINNs solving a broad class of linear operators, surpassing results limited to second-order cases. For nonlinear PDEs, we introduce a new approach using the Łojasiewicz inequality [ 18 ] to rigorously characterize optimization dynamics and guarantee convergence to critical points for important nonlinear cases. This provides the first convergence theory unifying PINNs and Deep Ritz methods in nonlinear settings. More precisely, the main contributions of this paper are as follows: (i) We establish convergence to global minima for over-parameterized PINNs in solving a broad class of linear PDEs, thereby significantly extending existing NTK-based results that are limited to second-order cases ( Theorem 1 ). (ii) We provide a convergence framework for PINN and Deep Ritz solvers under both gradient flow and implicit gradient descent dynamics, assuming coercivity of the loss function ( Proposition 1 ). (iii) Under the random feature model, we prove convergence to critical points for both PINN and Deep Ritz solvers when applied to a wide range of PDEs, including all evolutionary equations and several fundamental classes of nonlinear PDEs ( Theorems 3 and 4 ). Moreover, our analysis reveals an intrinsic regularization effect induced by the random feature model. This paper is organized as follows. In Section 2 , we review related works on machine learning-based PDE solvers and existing convergence analyses. Section 3 introduces the problem setting, and establishes general convergence results. Section 4 presents our main convergence results for solving nonlinear PDEs under different cases. Section 5 provides experimental evidence supporting our theoretical findings. Finally, Section 6 concludes the paper and discusses potential directions for future research. Technical proofs and supplementary materials are included in the appendix. 2 Related works Machine learning PDE solvers. There are various machine learning-based solvers for PDEs, among which physics-informed neural networks [ 3 ] and Deep Ritz method [ 4 ] are the most widely used. PINNs incorporate the PDE structure directly into the loss function, while Deep Ritz leverages the variational form of certain problems. Both methods have demonstrated remarkable empirical performance in solving a wide variety of nonlinear PDEs including, for example, the Allen–Cahn equation [ 19 ] and Schrödinger equation [ 20 ] across numerous applications [ 21 , 22 , 23 ] . Despite their success, theoretical understanding of their convergence properties, particularly for nonlinear PDEs, remains limited and is an active area of ongoing research. Existing convergence analysis using NTK framework. The neural tangent kernel (NTK) framework, which approximates over-parameterized neural networks as linear models with an almost constant Gram matrix during training, underpins much of the existing convergence analysis [ 9 , 10 ] . NTK was initially applied to study gradient descent in supervised learning settings [ 24 , 25 , 26 ] , and has been extended to analyze the convergence of PINNs for second-order linear PDEs [ 11 , 12 , 13 ] . These results typically show that, for highly over-parameterized NTK-scaled neural networks, the training loss converges to zero with gradient-based optimization methods. Convergence analysis using Łojasiewicz inequality. The Łojasiewicz inequality [ 18 ] is a fundamental analytical tool in the field of optimization, especially for studying the convergence properties of gradient-based algorithms [ 27 , 28 ] . Traditionally, it has been widely used to analyze the convergence in various non-convex and nonsmooth optimization problems [ 29 , 30 , 31 ] . In recent years, the Łojasiewicz inequality has also been increasingly applied in the context of supervised learning [ 32 , 33 ] . Researchers have leveraged this inequality to study the convergence behavior of machine learning algorithms, providing theoretical guarantees for global or local convergence under mild assumptions [ 34 , 35 ] . 3 Mathematical Setup and General Supporting Results In this section, we present the basic mathematical setup and introduce both PINNs and Deep Ritz solvers. In Section 3.2 , we establish two main results: first, a rigorous global convergence guarantee for PINNs in solving a broad class of linear PDEs based on the NTK approach; second, a more general convergence result to critical points, which serves as the foundation for our subsequent analysis of nonlinear PDEs. 3.1 Problem setting We consider a

Read the complete text