Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates
Anthony Zhou, Amir Barati Farimani · 2024
Read Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates by Anthony Zhou, Amir Barati Farimani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13074.txt.
Introduction
Anthony Zhou and Amir Barati Farimani change how neural models forecast moving fluids and fields, in a 2024 paper. Most such models guess the whole next state of a system at each step. That direct habit is simple, but it can lose accuracy and stability when time steps get fine.
Their frame has the model guess the rate of change instead, then lets a standard solver step the system forward. The swap adds little cost and fits many model types and physics problems. Models gained accuracy and held steady at fine time steps, and they were no longer locked to one step size. Training on finer data also became possible. The idea gives physics models a more flexible way to roll a system through time.
Catalog record
| CREATOR | Anthony Zhou, Amir Barati Farimani |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates Anthony Zhou, Amir Barati Farimani Abstract Neural surrogates for partial differential equations (PDEs) have become popular due to their potential to quickly simulate physics. With a few exceptions, neural surrogates generally treat the forward evolution of time-dependent PDEs as a black box by directly predicting the next state. While this is a natural and easy framework for applying neural surrogates, it can be an over-simplified and rigid framework for predicting physics. In this work, we evaluate an alternate framework in which neural solvers predict the temporal derivative and an ODE integrator forwards the solution in time, which has little overhead and is broadly applicable across model architectures and PDEs. We find that by simply changing the training target and introducing numerical integration during inference, neural surrogates can gain accuracy and stability in finely-discretized regimes. Predicting temporal derivatives also allows models to not be constrained to a specific temporal discretization, allowing for flexible time-stepping during inference or training on higher-resolution PDE data. Lastly, we investigate why this framework can be beneficial and in what situations does it work well. keywords machine learning, partial differential equations, neural surrogates, numerical methods 1 Introduction Partial differential equations (PDEs) model a wide variety of physical phenomena, from neuron excitations to fluid flows to climate patterns. Many of the most complex yet practically relevant PDEs are time-dependent and can be expressed in the form: \frac{\partial\mathbf{u}}{\partial t}=F(t,\mathbf{x},\mathbf{u},\frac{\partial\mathbf{u}}{\partial x},\frac{\partial^{2}\mathbf{u}}{\partial x^{2}},\ldots)\qquad t\in[0,T],\mathbf{x}\in\Omega (1) where there is one time dimension t=[0,T] and multiple spatial dimensions \mathbf{x}=[x_{1},x_{2},\ldots,x_{D}]^{T}\in\Omega , and \mathbf{u}:[0,T]\times\Omega\rightarrow\mathbb{R}^{d_{p}} is a quantity that is solved for with a physical dimension d_{p} . Solving equations of this form is of great interest, motivating centuries of research and leading to many modern engineering and scientific advances. Although solutions are analytically intractable, they can be approximated by discretizing the domain, where t_{n}\in\{t_{0},\ldots,t_{N}\} , x_{m}\in\{x_{0},\ldots,x_{M}\} , and the solution \mathbf{u}(t_{n},x_{m}) is discretized in this domain. Within this setup, initial conditions \mathbf{u}(0,x_{m}) and boundary conditions B[\mathbf{u}](t_{n},x_{b}),x_{b}=\{x_{m}:x_{m}\in\partial\Omega\} are usually given based on the practical application of the PDE. Numerical Solvers Within this setup, numerical solvers are the dominant approach to approximating PDE solutions. In general, numerical solvers aim to compute \mathbf{u} by calculating an approximation of \frac{\partial\mathbf{u}}{\partial t}|_{t=t_{n}}\approx F(t_{n},\mathbf{u}(t_{n})) and evolving the current solution forward in time using this temporal derivative. This temporal derivative can be approximated by discretizing the spatial derivatives at the current time, such as using finite-difference or finite-element methods. In the most straightforward case, constructing this approximation converts the PDE into an ODE, which allows the solution to be evolved forward in time with an ODE integrator such as forward Euler or Runge-Kutta, also known as the method of lines ( Schiesser 2012 ) . Complex PDEs may require a more careful treatment of the temporal update, such as splitting the update into an intermediate timestep ( Chorin 1967 ) . As a whole, both the temporal and spatial approximation require a disciplined and informed choice in discretization schemes (explicit/implicit, 1st-order/2nd-order, triangle/quad elements, etc.); importantly, the resulting accuracy, convergence, and solution time are all influenced by these choices. Neural Surrogates As a result, deriving and solving an accurate approximation of \mathbf{u} can require significant technical expertise and computational resources. This has motivated a new class of deep learning methods, or neural surrogates , that can learn this update from data, rather than an analytical derivation, and be quickly queried ( Li et al. 2021 ; Lu et al. 2021 ) . The fast inference speed of deep learning models makes neural surrogates attractive, however, they lack performance guarantees and the rigorous mathematical theory behind numerical analysis. Regardless, data-driven neural surrogates have shown empirically good performance and the field is rapidly growing to address fundamental challenges in model accuracy, speed, robustness, and practicality. Many works have proposed neural surrogates based on different deep learning architectures, as well as proposed a diverse set of modifications and architecture changes to tailor networks to physics applications. These include neural surrogates based on neural operators Kovachki et al. 2024 ; Cao 2021 , graph neural networks (GNNs) Li and Farimani 2022 ; Battaglia et al. 2016 , transformers Li et al. 2023 ; Alkin et al. 2024 ; Hemmasian and Farimani 2024 , convolutional neural networks (CNNs) Thuerey et al. 2020 ; Gupta and Brandstetter 2022 , or diffusion models Zhou et al. 2025 ; Kohl et al. 2024 ; Valencia et al. 2025 , each with their different benefits and drawbacks. For example, CNNs are simple to train and empirically show good performance Kohl et al. 2024 , however they can only be applied to regular-grid problems. On the other hand, transformers can be used on arbitrary geometries and perform well; however, they can be costly due to the quadratic complexity of attention, and specialized algorithms have been developed to mitigate this Li et al. 2023 ; Wu et al. 2024 . In addition to advancements in neural network architectures, many works have aimed to improve different aspects of neural surrogates, such as modifying the training strategy or datasets. A few examples of these works include improving the spectral performance of neural surrogates ( Lippe et al. 2023 ) , adapting to irregularly discretized data ( Brandstetter et al. 2023 ; Li et al. 2023 ) , and scaling models to large parameter sizes and across different datasets Hao et al. 2024 ; Herde et al. 2024 ; Zhou et al. 2024 ; Zhou and Farimani 2024 . The relevant literature is too vast to detail here; however, for a list of representative neural surrogate works and their numerical baselines, we recommend McGreivy and Hakim 2024 and for a survey on neural surrogates and operator learning we recommend Kovachki et al. 2024 . Within this large and continually expanding body of work, we evaluate an alternate training framework, one that can be applied generally to different architectures and can work together with different training modifications and across datasets. (a) State prediction. Models are trained to directly predict \mathbf{u}_{n+1}=F_{\theta}(\mathbf{u}_{n}) . (b) Derivative prediction. Models predict \frac{\partial\mathbf{u}}{\partial t}|_{t=t_{n}}=F_{\theta}(\mathbf{u}_{n}) to solve \hat{\mathbf{u}}_{n+1} . (c) Derivative prediction with higher order integrators can be more accurate. (d) Derivative prediction with variable step sizes can be more flexible. Figure 1 : A comparison of state prediction and derivative prediction, where models are either trained to predict \mathbf{u}_{n+1} or \frac{\partial\mathbf{u}}{\partial t}|_{t=t_{n}} . During inference, models are given an initial solution \mathbf{u}_{n} , and predict future solutions along the dashed trajectory. By predicting the temporal derivatives rather than the future solution, derivative prediction can learn spatial updates while an ODE integrator updates the solution in time, which can improve accuracy. Furthermore, derivative prediction can use higher-order integrators or variable timesteps, which further improves its accuracy and flexibility, while being applicable across model architectures and datasets. Contributions Neural surrogates often treat solution updates as a black box and directly predict the next state, which we call state prediction ( \mathbf{u}(t_{n+1})=F_{\theta}(\mathbf{u}(t_{n})) ). State prediction has become the dominant framework for training and querying neural surrogates, as it is the most obvious way to step forward in time, is easy to implement, and allows neural surrogates to take larger steps \Delta t . However, we believe