Robust Matrix-Free Newton-Krylov Solvers via Automatic Differentiation Journal: Journal of Computational Physics

Marco Pasquale, Stefano Markidis · 2026

Read Robust Matrix-Free Newton-Krylov Solvers via Automatic Differentiation Journal: Journal of Computational Physics by Marco Pasquale, Stefano Markidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13378.txt.

Introduction

Marco Pasquale and Stefano Markidis test whether forward-mode automatic differentiation can replace finite differences inside Jacobian-free Newton-Krylov solvers. These solvers tackle stiff nonlinear equations without forming the full Jacobian matrix. Finite differences, however, make their linearisation fragile in limited precision. Holding every other solver choice fixed, the authors find automatic differentiation speeds computation by two to three orders of magnitude on processors and graphics cards alike. It also lifts the share of runs that finish from forty-two percent to at least ninety-five percent.

Catalog record

CREATORMarco Pasquale, Stefano Markidis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53896
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Matrix-Free Newton-Krylov Solvers via Automatic Differentiation Journal: Journal of Computational Physics Marco Pasquale, Stefano Markidis Abstract Jacobian-Free Newton-Krylov (JFNK) methods avoid forming the full Jacobian, but still require Jacobian-vector products, i.e., Gateaux derivatives of the nonlinear residual along Krylov directions. In standard Finite Differences (FD) formulations, these products are obtained by perturbing the Newton state and differencing residuals, making the linearization sensitive to round-off error and floating-point precision. This work evaluates the global impact of forward-mode Automatic Differentiation (AD) as a replacement for FD Jacobian-vector product in finite-precision JFNK solvers. The comparison keeps the discretization, Newton iteration, line search, Krylov methods, tolerances, and CPU/GPU backend fixed, only varying linearization strategy. Benchmarks include Burgers dynamics, Su-Olson radiation diffusion, reaction-diffusion, and nonlinear time-harmonic Maxwell equations, each evaluated in different nonlinear regimes. By preventing degradation of the Krylov operator, AD accelerates computation by 2–3 orders of magnitude across both CPU and GPU architectures. More importantly, it drastically improves global solver robustness, achieving a minimum completion rate of 95%, compared to just 42% for FD. Ultimately, accurate Gateaux derivatives unify performance and accuracy in JFNK methods, making AD the optimal choice for stiff nonlinear and reduced-precision environments. 1 Introduction The solution of nonlinear problems is central to numerous scientific and engineering applications, making robust nonlinear solvers indispensable in computational physics. In particular, Jacobian-Free Newton-Krylov (JFNK) algorithms are extensively employed for large-scale systems governed by Partial Differential Equations (PDEs). These methods are advantageous because they couple the rapid convergence characteristics of Newton’s method with the memory efficiency of matrix-free Krylov subspace linear solvers [ 16 , 15 ] . Additionally, JFNK provides a highly efficient computational framework capable of addressing both implicit time-dependent initial boundary value problems (IBVPs) and boundary value problems (BVPs). Instead of forming and storing the full Jacobian matrix, the Newton correction is computed using Krylov subspace methods such as GMRES [ 26 ] , Conjugate Gradient (CG) [ 12 ] , or BiCGSTAB [ 29 ] . These methods require only the action of the Jacobian on vectors. This matrix-free formulation avoids explicit Jacobian assembly, but it does not avoid linearization. At each Newton iteration, the Krylov solver requires repeated Jacobian-vector products, which correspond to Gateaux derivatives of the nonlinear residual along Krylov directions. The quality of these products directly affects the Krylov subspace, the computed Newton correction, and therefore the convergence of the full nonlinear solve. For this reason, the Jacobian-vector product is a critical numerical component of the JFNK method which links accuracy to performance. In standard JFNK implementations, Jacobian-Vector Products (JVPs) are commonly approximated by Finite Differences (FD) of the nonlinear residual [ 16 , 24 ] . This requires perturbing the current Newton state along a Krylov direction and differencing two residual evaluations. The perturbation size must be selected by the user or by a heuristic rule. However, its choice might have large implications. If the perturbation is too large, the derivative is biased by truncation error. If it is too small, cancellation and round-off might dominate. This sensitivity becomes especially important when using reduced precision, e.g. single-precision (FP32) instead of double-precision (FP64). In these cases, the admissible range of perturbation sizes narrows and inaccurate JVP can lead to Krylov stagnation, poor Newton corrections, and solver failure. The broader question of when derivatives should be trusted over finite differences has also been studied in the context of noisy computational functions [ 21 ] . The increased use of reduced-precision arithmetic, spurred by Machine Learning (ML) workloads and accelerators with FP32 and half-precision (FP16 units), makes this issue increasingly important. In several areas of computational science, including computational fluid dynamics, weather and climate modelling, and accelerator-based simulation, single- and mixed-precision are attractive because they reduce memory traffic and can improve throughput on modern hardware [ 17 , 11 , 7 , 14 , 28 ] . However, the benefit of reduced precision is not automatic for nonlinear solvers. The inner Krylov iteration can be affected by round-off error, loss of orthogonality in the Arnoldi or Lanczos process, and inaccurate FD Jacobian-vector products [ 8 ] . Consequently, the linearization strategy can determine whether a JFNK solver actually benefits from reduced precision. Automatic Differentiation (AD) [ 27 ] provides an alternative to FD for computing derivatives of numerical algorithms [ 5 , 9 , 10 ] . AD has been used to compute sparse Jacobians and derivative matrices in scientific computing [ 4 , 31 ] , and modern tools now support increasingly complex software environments and parallel programming models [ 22 ] . AD has also been incorporated into scientific computing frameworks and multiphysics environments, including PETSc-based Newton–Krylov workflows, MOOSE, and Trilinos toolchains [ 13 , 18 , 20 ] . More recently, differentiable programming has renewed interest in differentiating numerical solvers for differential equations, especially in inverse problems, data assimilation, and physics-informed learning [ 25 , 3 , 19 ] . Although AD offers advantages, JFNK solvers have been traditionally implemented with FD to avoid coding complexity and performance overheads associated with it. In this work, we use AD as a numerical device within a matrix-free Newton–Krylov method. Forward-mode AD computes the Jacobian-vector product required by the Krylov solver by directly differentiating the implemented discrete residual. Replacing FD JVPs with AD alters the linearized operator presented to the Krylov solver. JFNK tightly couples multiple iterative layers, including the outer continuation loop, Newton iterations, Krylov subspace construction, and line search. Because of this nested integration, errors in the JVP can easily cascade. Therefore, the impact of AD must be evaluated across the complete nonlinear solver, not just as an isolated derivative. The objective of this paper is to provide a complete evaluation of forward-mode AD as a replacement for FD JVP in JFNK solvers for nonlinear PDEs. We measure how this choice affects Krylov convergence, Newton convergence, execution time, and solver failure rates, at the same time comparing any possible additional overhead introduced by AD. In this work, robustness refers to the ability of the full nonlinear solver to converge reliably across problem classes, parameter regimes, floating-point precisions, and Krylov methods without stagnation or problem-specific tuning. Our benchmark suite covers both time-dependent IBVPs and frequency-domain BVPs. It includes viscous Burgers dynamics, Su-Olson radiation diffusion, reaction-diffusion, and nonlinear time-harmonic Maxwell equations in Kerr media. These problems span nonsymmetric and symmetric systems, in parameter regimes with different degrees of stiffness. We evaluate FP64 and FP32 arithmetic, CPU and GPU execution, and multiple Krylov solvers. The goal is to identify how Jacobian-vector product construction affects the reliability and performance of practical matrix-free nonlinear PDE solvers. We demonstrate that AD improves robustness primarily by preventing finite-precision degradation of the Krylov operator. The largest differences appear in FP32, where FD JVP can cause severe Krylov stagnation. In a representative FP32 Burgers case, FD approximations inflated the iteration count to near the GMRES cap, requiring hundreds of thousands of total Krylov iterations to reach convergence. By comparison, AD maintained a tightly bounded Krylov subspace, reducing the iterations by 491 \times and accelerating the full nonlinear solve by 169 \times , recovering the same converged solution. 2 Preliminaries We consider a continuous nonlinear PDE problem generally written as \mathbf{G}\left(x_{1},\dots,x_{n},u,\frac{\partial u}{\partial x_{1}},\dots,\frac{\partial u}{\partial x_{n}},\frac{\partial^{2}u}{\partial x_{i}\partial

Read the complete text