Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs
Markus Bachmayr 1, Wolfgang Dahmen 1,2, Chenguang Duan 1, Mathias Oster 1 · 2026
Read Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs by Markus Bachmayr 1, Wolfgang Dahmen 1,2, Chenguang Duan 1, Mathias Oster 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23185.txt.
Introduction
Markus Bachmayr and three co-authors study how pre-scaling affects training neural networks for parametric equations. These networks estimate solutions of partial differential equations that depend on parameters. Training can be slow or unstable when the underlying operators are badly scaled.
Using well-conditioned frame forms of the operators, they show clear gains for standard training methods. They also analyze numeric stability, not just speed. Reliable training matters as neural solvers move into engineering and science workflows. Frame forms are the pre-scaling vehicle, re-expressing operators so standard optimizers see a well-scaled problem. Real improvement in standard training performance is the reported effect, not a marginal gain.
Catalog record
| CREATOR | Markus Bachmayr 1, Wolfgang Dahmen 1,2, Chenguang Duan 1, Mathias Oster 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23185
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preconditioning and Numerical Stability in Neural Network Training for Parametric PDEs Thanks: The authors acknowledge funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 442047500/SFB 1481 Sparsity and Singular Structures . M.B. acknowledges funding by the European Union (ERC, COCOA, 101170147). W.D. acknowledges funding by National Science Foundation grants FRG DMS-2245097, DMS-2513112, and RTG DMS-2038080. Markus Bachmayr 1, Wolfgang Dahmen 1,2, Chenguang Duan 1, Mathias Oster 1 Abstract . In the context of training neural network-based approximations of solutions of parameter-dependent PDEs, we investigate the effect of preconditioning via well-conditioned frame representations of operators and demonstrate a significant improvement on the performance of standard training methods. We also observe that standard representations of preconditioned matrices are insufficient for obtaining numerical stability and propose a generally applicable form of stable representations that enables computations with single- and half-precision floating point numbers without loss of precision. Keywords. parameter-dependent partial differential equations, preconditioning, finite element frames, neural networks, numerical stability Mathematics Subject Classification. 41A30, 65D40, 65N55 1. Introduction 1.1. Problem formulation The need for computationally efficient approximations of solutions of parameter-dependent PDEs arises in many different applications. Abstractly, given a family of differential operators \mathcal{R}_{y} in residual form, parameterized by y\in Y , solutions u(x,y) of \mathcal{R}_{y}(u(\cdot,y))=0 can be viewed as functions of spatio-temporal variables x\in\Omega\subset\mathbb{R}^{d} as well as of the parametric variables y ranging over a given parameter domain Y . Here Y can be a subset of a finite-dimensional Euclidean space, Y\subset\mathbb{R}^{p} with some p\in\mathbb{N} , or more generally of an infinite-dimensional sequence or function space. In which sense u (or a derived quantity of interest) is to be approximated over \Omega\times Y , is determined by approximating the mapping y\mapsto u(y)=u(\cdot,y) in appropriate norms for functions on Y and \Omega . This task, sometimes referred to as “operator learning,” can arise both in a context of model reduction (as for the accelerated solution of inverse or optimization problems) and in uncertainty quantification. Especially for challenging problems that are not amenable to linear model reduction methods such as POD or reduced basis methods, approximations for u based on neural networks are of interest. While in certain approaches such as physics-informed neural networks (PINNs), values of u are approximated by a neural network with input parameters x and y , it can be beneficial to use neural network representations only in the (potentially high-dimensional) y -variable, while the dependence on x is approximated by well-understood classical discretization methods. Assuming \{\varphi_{i}\}_{i\in\mathcal{I}} with \#\mathcal{I}<\infty to be a suitable set of fixed basis functions on \Omega , this leads to hybrid approximations of the form (1.1) u(x,y)\approx\tilde{u}(x,y;\theta)=\sum_{i\in\mathcal{I}}\mathbf{u}_{i}(y;\theta)\varphi_{i}(x), where the approximation coefficients \mathbf{u}=(\mathbf{u}_{i})_{i\in\mathcal{I}} are jointly represented as neural networks and depend on y as well as on a vector \theta of trainable network parameters. Note that the basis functions \varphi_{i} could be standard finite element basis functions or elements of a problem-adapted reduced basis. Representations of the form ( 1.1 ) have been used, for example, for problems with finite-dimensional Y in [ 19 , 25 , 21 , 1 , 9 ] , but are also common in the setting of operator learning with infinite-dimensional Y [ 3 , 24 ] . For any method that yields a quantification of the achieved accuracy of approximations with respect to a suitable norm, it is important to decide in which (weak) sense the equation \mathcal{R}_{y}(u(\cdot,y))=0 is required to hold. For a proper weak formulation, one should have \mathcal{R}_{y}\colon U_{y}\to V_{y}^{\prime} with Hilbert spaces U_{y} , V_{y} for each y , postponig for a moment further comments on the choice of the trial- and test-spaces U_{y},V_{y} . A commonly used way to obtain the approximation \tilde{u} to y\mapsto u(y)\in U_{y} is then to employ empirical risk minimization (1.2) \min_{\theta}\sum_{k=1}^{K}\lVert u(y_{k})-\tilde{u}(y_{k};\theta)\rVert_{U_{y_{k}}}^{2}, see, e.g. [ 19 , 25 ] , where for the sake of computational convenience, the norms \|\cdot\|_{U_{y}} are sometimes replaced by L_{2} -norms. Viewing the parameters as random variables with probability distribution \mu and assuming the y_{k} , k=1,\ldots,K , to be i.i.d. draws from \mu , this corresponds to approximately minimizing a Monte Carlo approximation of the expectation \mathbb{E}_{y\sim\mu}\big[\|u(y)-\tilde{u}(y_{k};\theta)\|^{2}_{U_{y_{k}}}\big] . If the norms \|\cdot\|_{U_{y}} are uniformly equivalent over Y , the expectation is a standard Bochner norm. If not, under mild conditions on the scalar products, inducing \|\cdot\|_{U_{y}} , the above expectation is still well-defined as a so-called direct integral, see [ 1 ] . Alternatively, to mitigate the computational cost of a large number of high-fidelity training samples u(y_{k}) , it can also be advantageous to instead use residual losses . To infer from a residual the accuracy of the approximation it is essential in which norm the residual is measured. As shown in [ 1 ] , under the above assumptions on the mapping properties of \mathcal{R}_{y} , a proper choice is \lVert\mathcal{R}_{y}(w)\rVert_{V_{y}^{\prime}} (the dual test norm), because if the underlying weak formulations are stable one has \lVert\mathcal{R}_{y}(w)\rVert_{V_{y}^{\prime}}\eqsim\lVert w\rVert_{U_{y}} , a property that we refer to as variational correctness . Hence the size of the loss is a reliable and efficient a posteriori error bound. Then ( 1.2 ) can be replaced by (1.3) \min_{\theta}\sum_{k=1}^{K}\bigl\lVert\mathcal{R}_{y_{k}}\bigl(\tilde{u}(y_{k};\theta)\bigr)\bigr\rVert_{V_{y_{k}}^{\prime}}^{2}. The issue that we address in this paper concerns both types of optimization problems ( 1.2 ) and ( 2.5 ). The quality of corresponding numerical results depends crucially on the performance of optimization schemes used for the loss minimization. While the nonlinear structure of neural networks poses an unavoidable obstacle, the separation of spatial and parametric variables, facilitated by the hybrid format ( 1.1 ), nevertheless offers substantial advantages. A central point in what follows is that each summand in the objective is a quadratic form that offers room for formulating the objective such that it becomes favorable for descent strategies. An intrinsic obstruction lies in the fact that the norms on U_{y} and V_{y} are typically induced by differential operators, for example by the Laplacian when U_{y}=H^{1}_{0}(\Omega) . We explain later below in more detail that both types of loss ( 1.2 ) and ( 2.5 ) exhibit similar features with regard to the dependence on differential operators, opening the door to exploiting preconditioning techniques for removing PDE-related ill-conditioning from the optimization problems. 1.2. Choice of basis, preconditioning, and performance of optimization schemes The most basic and well-understood scenario is \mathcal{R}_{y}v=A_{y}v-f , where A_{y}\colon U_{y}\to V_{y}^{\prime} is a linear operator and f\in V_{y}^{\prime} . This corresponds to a weak formulation \langle A_{y}u(y),v\rangle=f(v),\quad\forall\,v\in V_{y},\;y\in Y. Variational correctness in the above sense is then equivalent to the well-posedness of this weak formulation (that is, the conditions in the Babuška-Nečas theorem hold), which in turn means that A_{y}:U_{y}\to V^{\prime}_{y} is a norm isomorphism. For simplicity of exposition, we assume that the spaces U_{y} , V_{y} agree for all y with equivalent norms, so that we can suppress dependence on y in corresponding notations. Note that then the problems (1.4) \min_{w\in U}\|A_{y}w-f\|^{2}_{V^{\prime}}\quad\mbox{or}\quad\min_{w\in U}\|u(y)-w\|^{2}_{U} are well-conditioned when posed in V^{\prime} , U , respectively, so that descent methods in U or V^{\prime} would converge rapidly. However, as soon as these problems are posed over some finite dimensional trial space U_{\mathcal{I}}={\rm span}\,\Phi,\quad\Phi=\{\varphi_{i}\colon i\in\mathcal{I}\}, the objective becomes a quadratic form over \mathbb{R}^{\#\mathcal{I}} whose numerical conditioning now depends on the choice of the