Towards Universal Solvers: Using PGD Attack in Active Learning to Increase Generalizability of Neural Operators as Knowledge Distillation from Numerical PDE Solvers
Yifei Sun · 2025
Read Towards Universal Solvers: Using PGD Attack in Active Learning to Increase Generalizability of Neural Operators as Knowledge Distillation from Numerical PDE Solvers by Yifei Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18989.txt.
Introduction
Yifei Sun studies neural operators, fast stand-ins for solvers of partial differential equations. Sun asks two things. How well does a learned operator generalize? And how well does it resist attack? The view is tied to the solver. Both outputs tend to shift as one when inputs are nudged.
Sun defines generalization through a model-solver error map at fixed inputs. Robustness is defined through bounded rises in attack loss. This gives operator learning its own safety terms. They differ from the terms used for image tasks. Users can judge when a fast stand-in is safe to trust.
Catalog record
| CREATOR | Yifei Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Universal Solvers: Using PGD Attack in Active Learning to Increase Generalizability of Neural Operators as Knowledge Distillation from Numerical PDE Solvers Yifei Sun Abstract Nonlinear partial differential equations (PDEs) typically require very fine spatial and temporal discretizations and local linear/Taylor-type approximations of nonlinear terms; this leads to large memory footprints and long step-by-step solver runtimes. [ 39 ] Neural operators extend neural networks’ universal function approximation perspective to mappings between function spaces and learn direct function-to-function surrogates that can be queried at arbitrary spatio-temporal points, exemplified by Fourier Neural Operators (FNOs) and DeepONets. [ 39 , 46 ] Compared with iterative time-marching, Fourier-based operator architectures can truncate high-frequency components and perform fast single-shot inference while maintaining high accuracy on in-distribution data, indicating possible redundancy in traditional solvers’ spatio-temporal discretization. [ 39 , 41 ] A major limitation of data-driven neural operators is poor out-of-distribution (OOD) generalization: when input functions have ranges or structural features that depart from the training distribution, neural-operator predictions can differ substantially from solver ground truth, preventing their use as universal solvers. To address this, we propose a teacher–student distillation framework in which a differentiable numerical solver (teacher) supervises a compact neural-operator student. Because solver-generated labels are expensive, we embed an active/adversarial sample selection loop in a minimax (PGD-style) formulation: with the student fixed we search (via projected gradient maximization) for input perturbations that maximize the student–teacher discrepancy (subject to physically motivated smoothness/energy constraints), add those worst-case examples to the training pool, and update the student. [ 47 ] Leveraging a differentiable spectral solver such as Exponax permits backpropagation of solver gradients into the adversarial search, stabilizing and strengthening sample mining under periodic/spectral settings. [ 32 ] We validate our method using FNO students and a differentiable spectral solver on canonical benchmarks (1D Burgers and 2D Navier–Stokes). Experiments show that adversarially guided distillation substantially improves OOD robustness while retaining the low parameter count and fast single-shot inference of neural operators, expanding their practical utility as surrogate PDE solvers. Introduction Nonlinear partial differential equations (PDEs) arise across the physical and engineering sciences and often govern complex, multi-scale dynamics such as shock formation, turbulence, pattern formation and other strongly nonlinear phenomena. Classical numerical approaches for such problems—finite differences, finite volumes/elements, spectral methods and their time-integrators—rely on discretizing the spatial and temporal domains into fine meshes and advancing the solution through many small time steps. This strategy approximates nonlinear terms locally via linearization or low-order Taylor expansions and enforces stability through small time steps (e.g., CFL-like constraints) and refined spatial resolution. While well-understood and broadly successful, this approach can lead to very large numbers of spatio-temporal degrees of freedom, heavy memory (and GPU) requirements, and long wall-clock forward-solve times for long-time or high-resolution simulations. Moreover, classical discretizations remain susceptible to numerical instabilities when resolution or timestep choices are suboptimal, exhibiting blow-up, spurious oscillations, and Gibbs-like artifacts near discontinuities or sharp gradients (for example, shock waves in Burgers-type problems). An alternative paradigm that has gained rapid traction is to learn operators —maps between infinite-dimensional function spaces—directly from data. Neural operators extend the idea of neural networks as universal function approximators to universal operator approximators, i.e., models that take an input function (e.g., an initial condition or forcing field) and output a target function (e.g., the solution field at a later time) without performing explicit step-by-step time marching. Representative architectures include DeepONet, which combines branch and trunk networks motivated by operator approximation theory [ 46 ] , and Fourier Neural Operators (FNOs), which parametrize integral kernels in Fourier space and have demonstrated fast, accurate, single-shot predictions for canonical PDE benchmarks such as Burgers and Navier–Stokes [ 39 ] . These neural-operator methods can be queried at arbitrary spatio-temporal coordinates, and when trained and evaluated in-distribution they often produce highly accurate outputs while dramatically reducing runtime compared to conventional solvers. In particular, Fourier-based methods exploit frequency truncation—retaining only the dominant low-frequency modes—so a compact set of parameters can capture the essential large-scale dynamics and produce rapid, single-pass inference [ 39 ] . The success of neural operators in many settings highlights two complementary insights. First, for many practical prediction tasks there can be redundancy in the very fine spatio-temporal discretizations used by classical solvers: the information required to obtain an accurate map from inputs to outputs can be compressed into a smaller set of spectral coefficients or learned features. Second, neural operators’ interpolation and regression characteristics often make them numerically robust: unlike an explicit time integrator whose stability can catastrophically fail when timesteps are too large or the spatial grid too coarse, learned surrogates trained on representative solutions tend to produce outputs that remain near the training-output manifold and therefore avoid the uncontrolled numeric blow-up typical in poorly resolved solvers. Taken together, this explains why, in many empirical tests, FNO-like surrogates can provide accurate predictions with coarser temporal granularity and much lower computational cost [ 39 ] . Despite these advantages, a fundamental limitation of current neural-operator approaches is their data-driven nature: they typically generalize well only when test inputs are drawn from the same distribution as the training data. When confronted with out-of-distribution (OOD) inputs—especially those whose local cell-wise ranges, spectral content, or global statistics differ substantially from the training set—neural-operator predictions can deviate severely from the ground-truth solutions produced by trusted numerical solvers. In practice this manifests as large predictive error or qualitatively wrong dynamics when the model is asked to extrapolate beyond the training manifold. This lack of robust OOD generalization means that neural operators are not yet reliable universal solvers that one can confidently apply to arbitrary inputs without careful validation or retraining. From a theoretical perspective, this shortcoming is unsurprising: compact neural-operator parametrizations (few layers, truncated spectra, relatively small parameter counts) trade expressive generality for speed and parsimony, and the complex, distribution-sensitive evolution laws of nonlinear PDEs mean that dynamics can change qualitatively as input distributions shift. A second, related body of PDE-learning work is formed by physics-informed neural networks (PINNs) and their variants. PINNs treat a neural network as a surrogate for the solution function and enforce PDE residuals, boundary and initial conditions via automatic differentiation during training. Because PINNs operate at the level of pointwise collocation (space–time coordinates), active and adaptive sampling of residual points has been extensively studied to locate failure regions (e.g., near shocks or steep gradients) and to improve training efficiency and accuracy [ 78 ] . Recently, adversarial/adaptive schemes for PINNs—e.g., PGD-style adversarial sampling that moves collocation points toward local maxima of the PDE residual—have been proposed to automatically find and correct failure regions without strong distributional assumptions [ 38 ] . While these sampling strategies mitigate some PINN failure modes, PINNs and neural operators fundamentally differ in input–output granularity: PINNs approximate a single solution function across the domain (pointwise), whereas neural operators learn mappings between whole functions (high-dimensional tensor inputs). Consequently, many PINN-centric active-sampling techniques do not translate directly to operator learning, because operator inputs live in very high-dimensional, often unbounded function spaces and residual-landscape assumptions (e.g., low-dimensional parametric forms) are harder to justify. Motivated by these observations, in this work we explore a hybrid solver–student framework that leverages the strengths of both differentiable numerical solvers and compact neural operators.